Unable to Connect to OMS on CRONULLA

Overview

The agent on CRONULLA is unable to connect to OMS there.

Status

Resolved

References

None.

Solution

Restart of CRONULLA fixed problem.

Investigation

Agent Status

Used the following command to check status of the agent:

/opt/app/oracle/em/agent/agent_inst/bin/emctl status agent

The output is:

Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent Version : 12.1.0.3.0 OMS Version : (unknown) Protocol Version : 12.1.0.1.0 Agent Home : /opt/app/oracle/em/agent/agent_inst Agent Binaries : /opt/app/oracle/em/agent/core/12.1.0.3.0 Agent Process ID : 2909 Parent Process ID : 2820 Agent URL : https://cronulla.yaocm.id.au:3872/emd/main/ Repository URL : https://cronulla.yaocm.id.au:4903/empbs/upload Started at : 2014-04-17 19:28:22 Started by user : oracle Last Reload : (none) Last successful upload : (none) Last attempted upload : (none) Total Megabytes of XML files uploaded so far : 0 Number of XML files pending upload : 4,252 Size of XML files pending upload(MB) : 6.09 Available disk space on upload filesystem : 37.98% Collection Status : Collections enabled Heartbeat Status : OMS is unreachable Last attempted heartbeat to OMS : 2014-04-18 22:09:58 Last successful heartbeat to OMS : (none) Next scheduled heartbeat to OMS : 2014-04-18 22:10:28 --------------------------------------------------------------- Agent is Running and Ready

Ping OMS

I tried to ping the OMS:

/opt/app/oracle/em/agent/agent_inst/bin/emctl pingOMS

The output is:

Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD pingOMS error: Failure connecting to https://cronulla.yaocm.id.au:4903/empbs/upload , err Connection refused

Checked Listening Ports

I checked the listening ports:

netstat --listen --tcp -p

The output is:

Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 localhost:x11-ssh-offset *:* LISTEN 2868/sshd tcp 0 0 localhost:6011 *:* LISTEN 4666/sshd tcp 0 0 localhost:6012 *:* LISTEN 29603/sshd tcp 0 0 *:42409 *:* LISTEN 1729/rpc.statd tcp 0 0 localhost:6700 *:* LISTEN 7827/opmn tcp 0 0 *:sunrpc *:* LISTEN 1711/rpcbind tcp 0 0 *:ssh *:* LISTEN 2030/sshd tcp 0 0 localhost:ipp *:* LISTEN 1785/cupsd tcp 0 0 localhost:smtp *:* LISTEN 2106/master tcp 0 0 localhost:x11-ssh-offset *:* LISTEN 2868/sshd tcp 0 0 *:54618 *:* LISTEN 1729/rpc.statd tcp 0 0 localhost:6011 *:* LISTEN 4666/sshd tcp 0 0 localhost:6012 *:* LISTEN 29603/sshd tcp 0 0 cronulla.yaocm.id.au:7102 *:* LISTEN 2394/java tcp 0 0 *:oem-agent *:* LISTEN 2909/java tcp 0 0 cronulla.yaocm.id.au:aequus *:* LISTEN 2515/java tcp 0 0 cronulla.yaocm.id.au:7301 *:* LISTEN 2515/java tcp 0 0 cronulla.yaocm.id.au:7403 *:* LISTEN 2327/java tcp 0 0 localhost:6700 *:* LISTEN 7827/opmn tcp 0 0 *:kti-icad-srvr *:* LISTEN 7827/opmn tcp 0 0 *:sunrpc *:* LISTEN 1711/rpcbind tcp 0 0 *:ssh *:* LISTEN 2030/sshd tcp 0 0 localhost:ipp *:* LISTEN 1785/cupsd tcp 0 0 localhost:smtp *:* LISTEN 2106/master

There is no process listening on port 4903.

Check Status of OMS

Checked the status of OMS:

/opt/app/oracle/em/middleware/oms/bin/emctl status oms -details

The output is:

Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : Console Server Host : cronulla.yaocm.id.au HTTP Console Port : 7788 HTTPS Console Port : 7802 HTTP Upload Port : 4889 HTTPS Upload Port : 4903 EM Instance Home : /opt/app/oracle/em/gc_inst/em/EMGC_OMS1 OMS Log Directory Location : /opt/app/oracle/em/gc_inst/em/EMGC_OMS1/sysman/log OMS is not configured with SLB or virtual hostname Agent Upload is locked. OMS Console is locked. Active CA ID: 1 Console URL: https://cronulla.yaocm.id.au:7802/em Upload URL: https://cronulla.yaocm.id.au:4903/empbs/upload WLS Domain Information Domain Name : GCDomain Admin Server Host : cronulla.yaocm.id.au Admin Server HTTPS Port: 7102 Admin Server is RUNNING Managed Server Information Managed Server Instance Name: EMGC_OMS1 Managed Server Instance Host: cronulla.yaocm.id.au WebTier is Down Oracle Management Server is Up

For some reason, the web tier is down.

Install Diagnostic Tools

The following diagnostic tools were installed:

• 12 Install RDA and EMDIAG Tools
• 13 Install REPVFY

Service Request

I raised SR 3-8877645241 with Oracle Support over this.

Details are:

Customer Problem Description --------------------------------------------------- Problem Summary --------------------------------------------------- Local Agent on OMS host unable to connect to OMS Problem Description --------------------------------------------------- The status of the agent on the host where OMS runs is: Heartbeat Status : OMS is unreachable pingOMS returns: EMD pingOMS error: Failure connecting to https://cronulla.yaocm.id.au:4903/empbs/upload , err Connection refused Error Codes --------------------------------------------------- err Connection refused Problem Category/Subcategory --------------------------------------------------- Enterprise Manager Base Platform: Administration, Security, Performance/Agent: Startup and Configuration Uploaded Files --------------------------------------------------- File: RDA_omsoutput_cronulla.zip Template Question Responses --------------------------------------------------- 1) ### Impact on Business ### Only affecting myself 2) === Additional Problem Description === Environment: NA Who: NA What: Agent on CRONULLA is unable to connect to OMS on same host When: Since 14 MArch 2014 Changes: OMA was patched with 17872949 Where: NA Error messages received: EMD pingOMS error: Failure connecting to https://cronulla.yaocm.id.au:4903/empbs/upload , err Connection refused