Now that I have an operational database running on GORDON, I need to set up the OEM agent before relocating the OEM repository from GRIDCTRL to GORDON.
When I tried to use the OEM agent to GORDON from the OEM console, I get the following error message:
The log at /opt/app/oracle/gc_inst/em/EMGC_OMS1/sysman/agentpush/2016-05-14_19-38-44-PM/applogs/gordon.yaocm.id.au_deploy.log shows:
2016-05-14_19-40-34:INFO:===VALIDATION===:Checking Password Validity on target nodes 2016-05-14_19-40-34:INFO:NODES=gordon.yaocm.id.au Retry Count 5 2016-05-14_19-40-34:INFO:retry count : 1 2016-05-14_19-40-34:INFO:Running cmd /bin/true on node gordon.yaocm.id.au 2016-05-14_19-40-34:INFO:ssh connect timeout 60000 2016-05-14_19-41-34:INFO:Error Message: PROV-16011: Session.connect: java.net.SocketTimeoutException: Read timed out 2016-05-14_19-41-34:INFO:Error Level : -1 2016-05-14_19-41-34:INFO:Cause : null 2016-05-14_19-41-34:INFO:Full Message: Session.connect: java.net.SocketTimeoutException: Read timed out 2016-05-14_19-41-34:INFO:Message: Session.connect: java.net.SocketTimeoutException: Read timed out 2016-05-14_19-41-34:INFO:AuthFailMsg to check password Auth fail keys Auth cancel 2016-05-14_19-41-34:INFO: SSH Connect Failure for user oracle on node gordon.yaocm.id.au 2016-05-14_19-41-34:INFO: Jsch Validation Failed Problem :SSH connection check failed Recommendation: Ensure that SSH daemon (sshd) on the target host is able to respond to login requests. 2016-05-14_19-41-34:INFO:Breaking since the jsch validation has failed 2016-05-14_19-41-34:INFO:InvocationTargetException Exception 2016-05-14_19-41-34:INFO:Printing Exception :java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at oracle.sysman.core.agentpush.ui.deployer.BaseDeployerOps.executeActions(BaseDeployerOps.java:1924) at oracle.sysman.core.agentpush.ui.deployer.InitializationDeployer.deploy(InitializationDeployer.java:452) at oracle.sysman.core.agentpush.ui.deployfwk.DeploymentWorker.run(DeploymentWorker.java:26) at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:311) Caused by: oracle.sysman.core.agentpush.ui.exception.JschValidationException at oracle.sysman.core.agentpush.ui.deployer.JschValidationOps.doRemotePasswordCheck(JschValidationOps.java:591) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at oracle.sysman.core.agentpush.ui.deployer.InitializationDeployer.doSSHValidations(InitializationDeployer.java:161) ... 8 more 2016-05-14_19-41-34:INFO: Jsch Valdation Failed Problem :SSH connection check failed Recommendation: Ensure that SSH daemon (sshd) on the target host is able to respond to login requests. 2016-05-14_19-41-34:INFO:Updating Action SSHValidationswith Status FAILED and error Message :Connection to the SSH daemon (sshd) on the target host failed with the following error : Session.connect: java.net.SocketTimeoutException: Read timed out and problem SSH connection check failed and recommendation Ensure that SSH daemon (sshd) on the target host is able to respond to login requests. 2016-05-14_19-41-34:INFO:=================action status is not empty FAILED
Since I can SSH to GORDON from my client PCs, the SSHD daemon must be running.
On CRONULLA, I run the following command to get logging messages from SSH:
ssh -v gordon.yaocm.id.au
The output is:
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug1: Connecting to gordon.yaocm.id.au [192.168.1.107] port 22. debug1: fd 3 clearing O_NONBLOCK debug1: Connection established. debug1: identity file /home/oracle/.ssh/identity type -1 debug1: identity file /home/oracle/.ssh/identity-cert type -1 debug1: identity file /home/oracle/.ssh/id_rsa type -1 debug1: identity file /home/oracle/.ssh/id_rsa-cert type -1 debug1: identity file /home/oracle/.ssh/id_dsa type -1 debug1: identity file /home/oracle/.ssh/id_dsa-cert type -1 debug1: identity file /home/oracle/.ssh/id_ecdsa type -1 debug1: identity file /home/oracle/.ssh/id_ecdsa-cert type -1 debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1 debug1: match: OpenSSH_6.6.1 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.3 debug1: SSH2_MSG_KEXINIT sent Read from socket failed: Connection reset by peer
This trace shows that GORDON is listening on port 22 with a SSHD daemon (see the highlighted lines).
I could not find any hits on My Oracle Support.
The best answer was found at ssh connection stop at “debug1: SSH2_MSG_KEXINIT sent”
The key point is that both ends of the SSH connection used Jumbo packets (MTU=9000). This can be confirmed by using ping as follows:
[oracle@cronulla ~]$ ping -c 1 gordon PING gordon.yaocm.id.au (192.168.1.107) 56(84) bytes of data. 64 bytes from gordon.yaocm.id.au (192.168.1.107): icmp_seq=1 ttl=64 time=0.661 ms --- gordon.yaocm.id.au ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 1ms rtt min/avg/max/mdev = 0.661/0.661/0.661/0.000 ms [oracle@cronulla ~]$ ping -c 1 -s 8972 gordon PING gordon.yaocm.id.au (192.168.1.107) 8972(9000) bytes of data. --- gordon.yaocm.id.au ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 10000ms
As can be seen, small packets get through, but the Jumbo packets are lost.
In order to resolve this problem, I ran the following command on GORDON:
sudo ip link set dev enp0s3 mtu 1500
And now SSH works.
To make the change permanent, changes were made to /etc/sysconfig/network-scripts/ifcfg-enp0s3.
And the agent was installed successfully.