Home‎ > ‎Issues‎ > ‎Cloud Control Issues‎ > ‎

Unable to Startup OMS on CRONULLA


Overview

The OMS on CRONULLA fails to start with the following messages:

Oracle Management Server Could Not Be Started

Status

Resolved

Solution

Synchronise the clocks between VICTORIA and GRIDCTRL. This synchronises the clocks betweeen CRONULLA and GRIDCTRL as the former is a VM on and VICTORIA.

References

Investigation

OMS Start Up Fails

The OMS on CRONULLA fails to start with the following messages:

Oracle Enterprise Manager Cloud Control 12c Release 4  
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Could Not Be Started
Check EM Server log file for details: /opt/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
Oracle Management Server is Down

The log (/opt/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out) does not appear to contain any useful information:

oracle.jdbc.thinLogonCapability not set
Successfully read all properties from repos.
securePort = 4903
<ConfigManager.getOMSSecureStatus> isOMSSecure = 1 isOMSSecureLocked = 1
In ConfigManager.loadProperties. Loaded properties from property file </opt/app/oracle/gc_inst/em/EMGC_OMS1/sysman/config/emomslogging.properties>
Plug-in deployment status: SUCCESS
Fetched repository credentials from Credential Store
<Dec 25, 2016 9:10:12 AM> <FINEST> <NodeManager> <Waiting for the process to die: 2638>
<Dec 25, 2016 9:10:12 AM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>
<Dec 25, 2016 9:10:12 AM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>

Install Diagnostic Tools

Following the instructions in EMDIAG Omsvfy 13c,12c Kit - Download and Install (Doc ID 1374450.1), I installed the omsvfy tool as follows:

cd /opt/app/oracle/middleware/oms
mkdir emdiag
unzip -q /tmp/omsvfy12.zip -d /opt/app/oracle/middleware/oms/emdiag

The tool was verified as follows:

export ORACLE_HOME=/opt/app/oracle/middleware/oms
export EMDIAG_HOME=${ORACLE_HOME}/emdiag
cd ${EMDIAG_HOME}/bin
./omsvfy version

The result was:

Component            Version                  
-------------------- -------------------------
OMSVFY                               2016.0627
Linux                       EL6.7 3.8.13 (XEN)
                      x86_64-linux-thread-mult
-------------------- -------------------------
OMS                                 12.1.0.4.0
OMS DST Files                                4
OMS JDK                               1.6.0_43
OMS JRE                               1.6.0_43
OMS JRE DST                        tzdata2012i
OMS JSch                               0.1.44o
OMS OPatch                         11.1.0.10.4
OMS OUI                            11.1.0.12.0
OMS PERL                                5.10.0
OMS RCU                             12.1.2.0.0
OMS Zip                                    3.0
OMS Unzip                                 6.00
-------------------- -------------------------
JRF                                 11.1.1.2.0
JRF OPatch                          11.1.0.9.9
JRF OUI                             11.1.0.9.0
-------------------- -------------------------
BI Publisher                         0.0.0.0.1
BI Publisher OPatch                         na
BI Publisher OUI                            na
-------------------- -------------------------
Webtier                             11.1.1.7.0
Webtier DST Files                            4
Webtier JDK                           1.6.0_35
Webtier JRE                           1.6.0_35
Webtier JRE DST                    tzdata2012c
Webtier OPatch                      11.1.0.9.9
Webtier OUI                         11.1.0.9.0
Webtier Zip                               2.32
Webtier Unzip                             5.52
-------------------- -------------------------
WebLogic Server                       10.3.6.0
WebLogic JDK                          1.6.0_43
WebLogic JRE                          1.6.0_43
WebLogic JRE DST                   tzdata2012i
-------------------- -------------------------

Run Diagnostics Tool

Following the instructions in EMDIAG Omsvfy 13c, 12c Kit - How to Use it (Doc ID 1374945.1),

cd ${EMDIAG_HOME}/bin
./omsvfy verify all -level 2 -details

Four (4) violations were found:

-- --------------------------------------------------------------------- --
-- OMSVFY: 2016.0627         OMS: 12.1.0.4.0        25-Dec-2016 11:41:03 --
-- --------------------------------------------------------------------- --
verifyAdminServer
verifyJAVA
201. Java TZ information not up-to-date: 3
     OMS JAVA TZ information is outdated (tzdata2012i)
     WebTier JAVA TZ information is outdated (tzdata2012c)
     WebLogic JAVA TZ information is outdated (tzdata2012i)
verifyNodeManager
verifyOMS
002. Invalid lok/pid file: 2
     Lock file found for OMS in directory (/opt/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/EMGC_OMS1.lok), but app is not running
     PID file found for OMS in directory (/opt/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/nodemanager/EMGC_OMS1.pid), but app is not running
verifyOracle
verifyOS
011. SELinux is not disabled: 1
     SELinux is not disabled
verifyOUI
200. Patched OPatch version not registered: 1
     OPatch version (11.1.0.10.4) is higher than software component in inventory (11.1.0.10.2)
verifyPERL
verifyWebLogic

4 violation(s) found

Clean Up OMSVFY Violations

Update Software

Used the following commands to update the operating system software:

sudo apt-get update

This did not fix the time-zone problem.

Disable SELINUX

Followed the procedure in How to Disable or set SELinux to Permissive mode (Doc ID 457458.1) (as root):

  1. Edit /etc/selinux/config
    • Change the SELINUX value to "SELINUX=disabled".
    • Reboot the server.

Remove Lock and PID Files

Remove the identified lock and PID files as follows:

rm -f /opt/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/EMGC_OMS1.lok
rm -f /opt/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/nodemanager/EMGC_OMS1.pid

Unregistered OPatch Version

I could not find any hits on My Oracle Support for instructions on how to register OPatch in the Oracle Inventory. I left it as i.

Validate Repository

Make Tablespaces Read-Write

During my attempt to migrate the OMR from GRIDTCTRL to GORDON, I had made tablespaces Read-Only as described in 06 Move OMR. I used the following SQL*Plus commands to find what tablespaces were Read-Only in the REPOS database:

set pages 62 lines 204
select tablespace_name, status from dba_tablespaces;

The results were:

TABLESPACE_NAME            STATUS
-------------------------- ---------
SYSTEM                     ONLINE
SYSAUX                     ONLINE
UNDOTBS1                   ONLINE
TEMP                       ONLINE
USERS                      READ ONLY
MGMT_ECM_DEPOT_TS          READ ONLY
MGMT_TABLESPACE            READ ONLY
MGMT_AD4J_TS               READ ONLY
RMAN_CATALOG               READ ONLY

9 rows selected.

I used the following SQL statements to make these tablespaces Read-Write:

alter tablespace users read write;
alter tablespace MGMT_ECM_DEPOT_TS read write;
alter tablespace MGMT_TABLESPACE read write;
alter tablespace MGMT_AD4J_TS read write;
alter tablespace RMAN_CATALOG read write;

This did not fix the start-up problem.

Install Diagnostic Tools

Followed the instructions in EMDIAG REPVFY Kit for Cloud Control 12c, 13c - Download, Install/De-Install and Upgrade (Doc ID 1426973.1) to install repvfy on GRIDCTRL:

unzip -qo /home/oracle/repvfy12_2016.0909.zip -d /opt/oracle/app/OracleHomes/db11g/emdiag
chmod u+x /opt/oracle/app/OracleHomes/db11g/emdiag/bin/repvfy
cd /opt/oracle/app/OracleHomes/db11g/emdiag/bin
./repvfy install

The following summary is produced:

COMPONENT            INFO
-------------------- --------------------
EMDIAG Version       2016.0909
EMDIAG Edition       2
Repository Version   12.1.0.4.0
Database Version     11.2.0.4.0
Test Version         2016.1225
Repository Type      CENTRAL
Verify Tests         569
Object Tests         228
Deployment           SMALL

9 rows selected.

Verify Repository

I ran the following command to verify the repository:

cd /opt/oracle/app/OracleHomes/db11g/emdiag/bin
./repvfy -level 2 -detail

The following summary is produced:

verifyAGENTS
======================================================================
1005. Active Agents with clock-skew problems
=--------- ---------- ---------- ---------- ---------- ---------- ---------=
= Action:
= - If the skew is only a few minutes, adjust the OS clock on the machine
=   or use a NTP service to act as the 'master of time' for the machines.
= - If the skew is more than an hour, the timezone of the Agent will need
=   to get changed, to have the correct time represented.
=   To change the timezone of an Agent:
=   * Update the property on the Agent side:
=     In case the OS changed, and the OS settings are different:
=     $ emctl config agent updateTZ
=     In case the Agent has a bad setting, which is NOT in sync with the OS:
=     $ emctl setproperty agent -name agentTZRegion -value <new value>
=   * Update the timezone in the repository for this Agent, to sync up the
=    internal parameters:
=     SQL> exec mgmt_target.set_agent_tzrgn('<agent_name>','<correct TZ region>')
=     SQL> commit
=--------- ---------- ---------- ---------- ---------- ---------- ---------=

AGENT_NAME                                         TIMEZONE_REGION                   SECONDS CLOCK_SKEW
-------------------------------------------------- ------------------------------ ---------- -------------
redfern1.yaocm.id.au:3872                          Australia/Sydney                     -154 -00h02m34s
cronulla.yaocm.id.au:3872                          Australia/Sydney                     -147 -00h02m27s
redfern2.yaocm.id.au:3872                          Australia/Sydney                     -146 -00h02m26s

3 rows selected.

verifyREPOSITORY
======================================================================
1. Insufficient RAW partitions for new data
=--------- ---------- ---------- ---------- ---------- ---------- ---------=
= Action:
= - Check the status of the repository DBMS_SCHEDULER jobs, and make sure
=   all tasks are running on the appointed time
=   (In particular the 'EM Daily Maintenance' job)
= - To do one-time manual cleanup, run this routine:
=   SQL> CONNECT sysman
=   SQL> exec gc_interval_partition_mgr.partition_maintenance
=--------- ---------- ---------- ---------- ---------- ---------- ---------=

MAX_PTIME
-------------------
2016/09/11 00:00:00

1 row selected.

I fixed the verifyREPOSITORY as follows:

[oracle@gridctrl bin]$ sqlplus sysman

SQL*Plus: Release 11.2.0.4.0 Production on Sun Dec 25 18:54:42 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Enter password: 

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - Production
With the Partitioning option

SQL> exec gc_interval_partition_mgr.partition_maintenance

PL/SQL procedure successfully completed.

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - Production
With the Partitioning option

This did not fix the start-up problem.

Install REPVFY on CRONULLA

Install Software

Followed the instructions in EMDIAG REPVFY Kit for Cloud Control 12c, 13c - Download, Install/De-Install and Upgrade (Doc ID 1426973.1) to install repvfy on CRONULLA:

unzip -qo /home/oracle/repvfy12_2016.0909.zip -d /opt/app/oracle/middleware/oms/emdiag/

Create TNSNAMES Alias for OMR

Created the following entry in the TNSNAMES.ORA on CRONULLA:

cat >>/opt/app/oracle/middleware/oms/network/admin/tnsnames.ora <<DONE
# -----------------------------------------------------------------------------
# TNS Names entries
# -----------------------------------------------------------------------------

repos = (DESCRIPTION =
            (ADDRESS = (PROTOCOL=TCP)(HOST=gridctrl.yaocm.id.au)(PORT=1521))
            (CONNECT_DATA = (SERVICE_NAME=repos.yaocm.id.au)(SERVER=DEDICATED))
          )
DONE

Update ORATAB

Added the following entry to the /etc/oratab on CRONULLA:

cat >>/etc/oratab <<DONE
oms:/opt/app/oracle/middleware/oms:N
DONE

This allows me to crudely set up the Oracle environment for OMS as follows:

[oracle@cronulla admin]$ . oraenv
ORACLE_SID = [oracle] ? oms
/opt/app/oracle/middleware/oms/bin/orabase: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory
The Oracle base for ORACLE_HOME=/opt/app/oracle/middleware/oms is 

Create REPVFY Configuration File

The REPVFY configuration file was created as follows:

cd /opt/app/oracle/middleware/oms/emdiag/cfg/
cp repvfy.cfg.template repvfy.cfg
vi repvfy.cfg

The following changes were made to repvfy.cfg:

ora_tns=repos

Review Test Results

I ran the following commands on CRONULLA:

cd /opt/app/oracle/middleware/oms/emdiag/bin
./repvfy -level 2 -detail

The test results were:

-- --------------------------------------------------------------------- --
-- REPVFY: 2016.0909     Repository: 12.1.0.4.0     25-Dec-2016 21:20:56 --
---------------------------------------------------------------------------
-- Module: all                                       Test:   0, Level: 2 --
-- --------------------------------------------------------------------- --
-- Visit My Oracle Support for a newer version of repvfy                 --
-- --------------------------------------------------------------------- --

verifyAGENTS
1005. Active Agents with clock-skew problems: 3
verifyASLM
verifyAVAILABILITY
verifyBLACKOUTS
verifyCAT
verifyCORE
verifyECM
verifyEMDIAG
verifyEVENTS
verifyEXADATA
verifyJOBS
verifyJVMD
verifyLOADERS
verifyMETRICS
verifyNOTIFICATIONS
verifyOMS
verifyPLUGINS
verifyREPOSITORY
verifyTARGETS
verifyTEMPLATES
verifyUSERS

Synchronise Clocks between VICTORIA and GRIDCTRL

On GRIDCTRL, I ran the following command:

ssh root@victoria date --utc $(date --utc +"%m%D%H%M%Y.%S")

I then rebooted VICTORIA, and the OMS server was then able to be started on CRONULLA.

Comments