The agent on CRONULLA is unable to connect to OMS there.
Resolved
None.
Restart of CRONULLA fixed problem.
Used the following command to check status of the agent:
/opt/app/oracle/em/agent/agent_inst/bin/emctl status agent
The output is:
Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent Version : 12.1.0.3.0 OMS Version : (unknown) Protocol Version : 12.1.0.1.0 Agent Home : /opt/app/oracle/em/agent/agent_inst Agent Binaries : /opt/app/oracle/em/agent/core/12.1.0.3.0 Agent Process ID : 2909 Parent Process ID : 2820 Agent URL : https://cronulla.yaocm.id.au:3872/emd/main/ Repository URL : https://cronulla.yaocm.id.au:4903/empbs/upload Started at : 2014-04-17 19:28:22 Started by user : oracle Last Reload : (none) Last successful upload : (none) Last attempted upload : (none) Total Megabytes of XML files uploaded so far : 0 Number of XML files pending upload : 4,252 Size of XML files pending upload(MB) : 6.09 Available disk space on upload filesystem : 37.98% Collection Status : Collections enabled Heartbeat Status : OMS is unreachable Last attempted heartbeat to OMS : 2014-04-18 22:09:58 Last successful heartbeat to OMS : (none) Next scheduled heartbeat to OMS : 2014-04-18 22:10:28 --------------------------------------------------------------- Agent is Running and Ready
I tried to ping the OMS:
/opt/app/oracle/em/agent/agent_inst/bin/emctl pingOMS
The output is:
Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD pingOMS error: Failure connecting to https://cronulla.yaocm.id.au:4903/empbs/upload , err Connection refused
I checked the listening ports:
netstat --listen --tcp -p
The output is:
Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 localhost:x11-ssh-offset *:* LISTEN 2868/sshd tcp 0 0 localhost:6011 *:* LISTEN 4666/sshd tcp 0 0 localhost:6012 *:* LISTEN 29603/sshd tcp 0 0 *:42409 *:* LISTEN 1729/rpc.statd tcp 0 0 localhost:6700 *:* LISTEN 7827/opmn tcp 0 0 *:sunrpc *:* LISTEN 1711/rpcbind tcp 0 0 *:ssh *:* LISTEN 2030/sshd tcp 0 0 localhost:ipp *:* LISTEN 1785/cupsd tcp 0 0 localhost:smtp *:* LISTEN 2106/master tcp 0 0 localhost:x11-ssh-offset *:* LISTEN 2868/sshd tcp 0 0 *:54618 *:* LISTEN 1729/rpc.statd tcp 0 0 localhost:6011 *:* LISTEN 4666/sshd tcp 0 0 localhost:6012 *:* LISTEN 29603/sshd tcp 0 0 cronulla.yaocm.id.au:7102 *:* LISTEN 2394/java tcp 0 0 *:oem-agent *:* LISTEN 2909/java tcp 0 0 cronulla.yaocm.id.au:aequus *:* LISTEN 2515/java tcp 0 0 cronulla.yaocm.id.au:7301 *:* LISTEN 2515/java tcp 0 0 cronulla.yaocm.id.au:7403 *:* LISTEN 2327/java tcp 0 0 localhost:6700 *:* LISTEN 7827/opmn tcp 0 0 *:kti-icad-srvr *:* LISTEN 7827/opmn tcp 0 0 *:sunrpc *:* LISTEN 1711/rpcbind tcp 0 0 *:ssh *:* LISTEN 2030/sshd tcp 0 0 localhost:ipp *:* LISTEN 1785/cupsd tcp 0 0 localhost:smtp *:* LISTEN 2106/master
There is no process listening on port 4903.
Checked the status of OMS:
/opt/app/oracle/em/middleware/oms/bin/emctl status oms -details
The output is:
Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : Console Server Host : cronulla.yaocm.id.au HTTP Console Port : 7788 HTTPS Console Port : 7802 HTTP Upload Port : 4889 HTTPS Upload Port : 4903 EM Instance Home : /opt/app/oracle/em/gc_inst/em/EMGC_OMS1 OMS Log Directory Location : /opt/app/oracle/em/gc_inst/em/EMGC_OMS1/sysman/log OMS is not configured with SLB or virtual hostname Agent Upload is locked. OMS Console is locked. Active CA ID: 1 Console URL: https://cronulla.yaocm.id.au:7802/em Upload URL: https://cronulla.yaocm.id.au:4903/empbs/upload WLS Domain Information Domain Name : GCDomain Admin Server Host : cronulla.yaocm.id.au Admin Server HTTPS Port: 7102 Admin Server is RUNNING Managed Server Information Managed Server Instance Name: EMGC_OMS1 Managed Server Instance Host: cronulla.yaocm.id.au WebTier is Down Oracle Management Server is Up
For some reason, the web tier is down.
The following diagnostic tools were installed:
I raised SR 3-8877645241 with Oracle Support over this.
Details are:
Customer Problem Description --------------------------------------------------- Problem Summary --------------------------------------------------- Local Agent on OMS host unable to connect to OMS Problem Description --------------------------------------------------- The status of the agent on the host where OMS runs is: Heartbeat Status : OMS is unreachable pingOMS returns: EMD pingOMS error: Failure connecting to https://cronulla.yaocm.id.au:4903/empbs/upload , err Connection refused Error Codes --------------------------------------------------- err Connection refused Problem Category/Subcategory --------------------------------------------------- Enterprise Manager Base Platform: Administration, Security, Performance/Agent: Startup and Configuration Uploaded Files --------------------------------------------------- File: RDA_omsoutput_cronulla.zip Template Question Responses --------------------------------------------------- 1) ### Impact on Business ### Only affecting myself 2) === Additional Problem Description === Environment: NA Who: NA What: Agent on CRONULLA is unable to connect to OMS on same host When: Since 14 MArch 2014 Changes: OMA was patched with 17872949 Where: NA Error messages received: EMD pingOMS error: Failure connecting to https://cronulla.yaocm.id.au:4903/empbs/upload , err Connection refused