Cloud Control Not Starting Automatically in OVM_EM12R2

Overview

After booting the VM created in Manually Create Cloud Control VM from Appliance, the OMS was not running.

Status

Work Around Available

Work Around

The work around is to start everything manually:

/u01/OracleHomes/agent/agent_inst/bin/emctl start agent

export ORACLE_HOME=/u01/OracleHomes/db11g/product/11.2.0/dbhome_1

export ORACLE_SID=emrepus

/u01/OracleHomes/db11g/product/11.2.0/dbhome_1/bin/lsnrctl start

/u01/OracleHomes/db11g/product/11.2.0/dbhome_1/bin/sqlplus / as sysdba

startup

exit

/u01/OracleHomes/Middleware/oms/bin/emctl start oms

Diagnosis

Agent Not Running

The EM12C agent was not running, so I started it:

[oracle@cloudctl ~]$ /u01/OracleHomes/agent/agent_inst/bin/emctl status agent Oracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent is Not Running [oracle@cloudctl ~]$ /u01/OracleHomes/agent/agent_inst/bin/emctl start agent Oracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. Starting agent ..................... started. [oracle@cloudctl ~]$ /u01/OracleHomes/agent/agent_inst/bin/emctl status agent Oracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent Version : 12.1.0.2.0 OMS Version : (unknown) Protocol Version : 12.1.0.1.0 Agent Home : /u01/OracleHomes/agent/agent_inst Agent Binaries : /u01/OracleHomes/agent/core/12.1.0.2.0 Agent Process ID : 1751 Parent Process ID : 1705 Agent URL : https://cloudctl.yaocm.id.au:3872/emd/main/ Repository URL : https://cloudctl.yaocm.id.au:4900/empbs/upload Started at : 2012-10-27 18:01:51 Started by user : oracle Last Reload : (none) Last successful upload : (none) Last attempted upload : (none) Total Megabytes of XML files uploaded so far : 0 Number of XML files pending upload : 24 Size of XML files pending upload(MB) : 0.02 Available disk space on upload filesystem : 25.38% Collection Status : Collections enabled Heartbeat Status : OMS is unreachable Last attempted heartbeat to OMS : 2012-10-27 18:02:11 Last successful heartbeat to OMS : (none) Next scheduled heartbeat to OMS : 2012-10-27 18:02:41 --------------------------------------------------------------- Agent is Running and Ready

OMS Not Running

The Oracle Management Server (OMS) was not running as well, so I started as well:

[oracle@cloudctl ~]$ /u01/OracleHomes/Middleware/oms/bin/emctl status oms Oracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. WebTier is Down Oracle Management Server is Down [oracle@cloudctl ~]$ /u01/OracleHomes/Middleware/oms/bin/emctl start oms Oracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. Starting Oracle Management Server... Starting WebTier... WebTier Successfully Started Oracle Management Server is not functioning because of the following reason: Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.

Repository Database is Down

Isn't this fun? The repository database is down as well. Didn't anythibng start on reboot?

[oracle@cloudctl ~]$ /u01/OracleHomes/db11g/product/11.2.0/dbhome_1/bin/srvctl status ****Unable to retrieve Oracle Clusterware home. Start Oracle Clusterware stack and try again. [oracle@cloudctl ~]$ find /u01 -name crsctl -type f -print [oracle@cloudctl ~]$ export ORACLE_HOME=/u01/OracleHomes/db11g/product/11.2.0/dbhome_1 [oracle@cloudctl ~]$ ls $ORACLE_HOME/dbs hc_emrepus.dat init.ora lkEMREPUS orapwemrepus spfileemrepus.ora [oracle@cloudctl ~]$ export ORACLE_SID=emrepus [oracle@cloudctl ~]$ /u01/OracleHomes/db11g/product/11.2.0/dbhome_1/bin/sqlplus / as sysdba SQL*Plus: Release 11.2.0.3.0 Production on Sat Oct 27 18:31:06 2012 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to an idle instance. SQL> startup ORACLE instance started. Total System Global Area 2505338880 bytes Fixed Size 2230952 bytes Variable Size 771753304 bytes Database Buffers 1711276032 bytes Redo Buffers 20078592 bytes Database mounted. Database opened. SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options

Check OMS:

Listener is not Running

The listener must be down:

[oracle@cloudctl ~]$ /u01/OracleHomes/db11g/product/11.2.0/dbhome_1/bin/lsnrctl status LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 27-OCT-2012 18:33:32 Copyright (c) 1991, 2011, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521))) TNS-12541: TNS:no listener TNS-12560: TNS:protocol adapter error TNS-00511: No listener Linux Error: 111: Connection refused Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=cloudctl.yaocm.id.au)(PORT=1521))) TNS-12541: TNS:no listener TNS-12560: TNS:protocol adapter error TNS-00511: No listener Linux Error: 111: Connection refused [oracle@cloudctl ~]$ /u01/OracleHomes/db11g/product/11.2.0/dbhome_1/bin/lsnrctl start LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 27-OCT-2012 18:33:39 Copyright (c) 1991, 2011, Oracle. All rights reserved. Starting /u01/OracleHomes/db11g/product/11.2.0/dbhome_1/bin/tnslsnr: please wait... TNSLSNR for Linux: Version 11.2.0.3.0 - Production System parameter file is /u01/OracleHomes/db11g/product/11.2.0/dbhome_1/network/admin/listener.ora Log messages written to /u01/OracleHomes/db11g/diag/tnslsnr/cloudctl/listener/alert/log.xml Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521))) Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cloudctl.yaocm.id.au)(PORT=1521))) Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521))) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for Linux: Version 11.2.0.3.0 - Production Start Date 27-OCT-2012 18:33:41 Uptime 0 days 0 hr. 0 min. 0 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/OracleHomes/db11g/product/11.2.0/dbhome_1/network/admin/listener.ora Listener Log File /u01/OracleHomes/db11g/diag/tnslsnr/cloudctl/listener/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cloudctl.yaocm.id.au)(PORT=1521))) The listener supports no services The command completed successfully

OMS is Running

Got the OMS finally up:

[oracle@cloudctl ~]$ /u01/OracleHomes/Middleware/oms/bin/emctl status oms Oracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. WebTier is Up Oracle Management Server is not functioning because of the following reason: Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up. [oracle@cloudctl ~]$ /u01/OracleHomes/Middleware/oms/bin/emctl stop oms Oracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. Stopping WebTier... WebTier Successfully Stopped Stopping Oracle Management Server... Oracle Management Server Successfully Stopped Oracle Management Server is Down [oracle@cloudctl ~]$ /u01/OracleHomes/Middleware/oms/bin/emctl start oms Oracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. Starting Oracle Management Server... Starting WebTier... WebTier Successfully Started Oracle Management Server Successfully Started Oracle Management Server is Up [oracle@cloudctl ~]$ /u01/OracleHomes/Middleware/oms/bin/emctl status oms Oracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. WebTier is Up Oracle Management Server is Up