Since the GI installation failed again. I uninstalled and restarted.
I started the de-installation on REDFERN2.
Following the procedure in How to Deconfigure/Reconfigure(Rebuild OCR) or Deinstall Grid Infrastructure (Doc ID 1377349.1), I ran the following command on REDFERN2:
sudo /opt/app/grid_infra/12.1.0/grid/perl/bin/perl /opt/app/grid_infra/12.1.0/grid/crs/install/rootcrs.pl -deconfig -force -verbose
The output was:
Using configuration parameter file: /opt/app/grid_infra/12.1.0/grid/crs/install/crsconfig_params Usage: srvctl <command> <object> [<options>] commands: enable|disable|start|stop|status|add|remove|modify|update|getenv|setenv|unsetenv|config|upgrade|downgrade objects: database|service|asm|diskgroup|listener|home|ons For detailed help on each command and object and its options use: srvctl <command> -help [-compatible] or srvctl <command> <object> -help [-compatible] PRKO-2012 : nodeapps object is not supported in Oracle Restart Failure in execution (rc=-1, 256, 2) for command /opt/app/grid_infra/12.1.0/grid/bin/crsctl stop crs -f Failure in execution (rc=-1, 256, 2) for command /etc/init.d/ohasd deinstall
This meant that CRS was not running on REDFERN2.
Since this was a two (2) node cluster, REDFERN1 was the last node with GI installed.
Following the procedure in How to Deconfigure/Reconfigure(Rebuild OCR) or Deinstall Grid Infrastructure (Doc ID 1377349.1), I ran the following command on REDFERN1:
sudo /opt/app/grid_infra/12.1.0/grid/perl/bin/perl /opt/app/grid_infra/12.1.0/grid/crs/install/rootcrs.pl -deconfig -force -verbose -lastnode
The output was:
Using configuration parameter file: /opt/app/grid_infra/12.1.0/grid/crs/install/crsconfig_params PRCR-1070 : Failed to check if resource ora.cvu is registered CRS-0184 : Cannot communicate with the CRS daemon. 2017/11/30 13:09:58 CLSRSC-180: An error occurred while executing the command 'srvctl stop cvu -f' (error code 0) PRCR-1070 : Failed to check if resource ora.cvu is registered CRS-0184 : Cannot communicate with the CRS daemon. 2017/11/30 13:09:58 CLSRSC-180: An error occurred while executing the command 'srvctl remove cvu -f' (error code 0) OC4J failed to stop PRCR-1070 : Failed to check if resource ora.oc4j is registered CRS-0184 : Cannot communicate with the CRS daemon. OC4J could not be removed PRCR-1070 : Failed to check if resource ora.oc4j is registered CRS-0184 : Cannot communicate with the CRS daemon. PRCR-1068 : Failed to query resources CRS-0184 : Cannot communicate with the CRS daemon. PRCR-1068 : Failed to query resources CRS-0184 : Cannot communicate with the CRS daemon. PRCR-1068 : Failed to query resources CRS-0184 : Cannot communicate with the CRS daemon. PRCR-1068 : Failed to query resources CRS-0184 : Cannot communicate with the CRS daemon. PRCR-1070 : Failed to check if resource ora.net1.network is registered CRS-0184 : Cannot communicate with the CRS daemon. PRCR-1070 : Failed to check if resource ora.helper is registered CRS-0184 : Cannot communicate with the CRS daemon. PRCR-1070 : Failed to check if resource ora.ons is registered CRS-0184 : Cannot communicate with the CRS daemon. CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'redfern1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'redfern1' CRS-2673: Attempting to stop 'ora.gipcd' on 'redfern1' CRS-2673: Attempting to stop 'ora.gpnpd' on 'redfern1' CRS-2673: Attempting to stop 'ora.evmd' on 'redfern1' CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'redfern1' CRS-2677: Stop of 'ora.cssdmonitor' on 'redfern1' succeeded CRS-2677: Stop of 'ora.gipcd' on 'redfern1' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'redfern1' succeeded CRS-2677: Stop of 'ora.gpnpd' on 'redfern1' succeeded CRS-2677: Stop of 'ora.evmd' on 'redfern1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'redfern1' has completed CRS-4133: Oracle High Availability Services has been stopped. CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start 'ora.evmd' on 'redfern1' CRS-2672: Attempting to start 'ora.mdnsd' on 'redfern1' CRS-2676: Start of 'ora.evmd' on 'redfern1' succeeded CRS-2676: Start of 'ora.mdnsd' on 'redfern1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'redfern1' CRS-2676: Start of 'ora.gpnpd' on 'redfern1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'redfern1' CRS-2672: Attempting to start 'ora.gipcd' on 'redfern1' CRS-2676: Start of 'ora.cssdmonitor' on 'redfern1' succeeded CRS-2676: Start of 'ora.gipcd' on 'redfern1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'redfern1' CRS-2672: Attempting to start 'ora.diskmon' on 'redfern1' CRS-2676: Start of 'ora.diskmon' on 'redfern1' succeeded CRS-2676: Start of 'ora.cssd' on 'redfern1' succeeded 2017/11/30 13:10:33 CLSRSC-147: Failed to get parameter values from GPnP profile '/opt/app/grid_infra/12.1.0/grid/gpnp/profiles/peer/profile.xml' 2017/11/30 13:10:33 CLSRSC-147: Failed to get parameter values from GPnP profile '/opt/app/grid_infra/12.1.0/grid/gpnp/profiles/peer/profile.xml' ASM de-configuration trace file location: /tmp/asmcadc_clean2017-11-30_01-14-44-PM.log ASM Clean Configuration START ASM Clean Configuration END ASM with SID +ASM1 deleted successfully. Check /tmp/asmcadc_clean2017-11-30_01-14-44-PM.log for details. CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'redfern1' CRS-2673: Attempting to stop 'ora.ctssd' on 'redfern1' CRS-2673: Attempting to stop 'ora.evmd' on 'redfern1' CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'redfern1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'redfern1' CRS-2673: Attempting to stop 'ora.gpnpd' on 'redfern1' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'redfern1' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'redfern1' succeeded CRS-2677: Stop of 'ora.ctssd' on 'redfern1' succeeded CRS-2677: Stop of 'ora.evmd' on 'redfern1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'redfern1' CRS-2677: Stop of 'ora.gpnpd' on 'redfern1' succeeded CRS-2677: Stop of 'ora.cssd' on 'redfern1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'redfern1' CRS-2677: Stop of 'ora.gipcd' on 'redfern1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'redfern1' has completed CRS-4133: Oracle High Availability Services has been stopped. 2017/11/30 13:17:26 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector. 2017/11/30 13:19:06 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector. 2017/11/30 13:19:07 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node 2017/11/30 13:19:07 CLSRSC-559: Ensure that the GPnP profile data under the 'gpnp' directory in /opt/app/grid_infra/12.1.0/grid is deleted on each node before using the software in the current Grid Infrastructure home for reconfiguration.
It appears that grid infrastructure was successfully de-installed on both nodes.
Ran the following command on both nodes:
sudo rm -fR /opt/app/grid_infra
Ran the following commands to see if the voting disk had been formatted:
dd if=/dev/xvdh of=/tmp/xvdh.dmp bs=8K count=128
The output was:
128+0 records in 128+0 records out 1048576 bytes (1.0 MB) copied, 0.0257255 s, 40.8 MB/s
Ran the following command to examine the header:
hexdump -C /tmp/xvdh.dmp|head
The output confirmed that the voting disk had been formatted:
00000000 01 82 01 01 00 00 00 00 00 00 00 80 20 af 32 ed |............ .2.| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000020 4f 52 43 4c 44 49 53 4b 00 00 00 00 00 00 00 00 |ORCLDISK........| 00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000040 00 00 10 0c 00 00 01 04 56 4f 54 45 5f 30 30 30 |........VOTE_000| 00000050 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |0...............| 00000060 00 00 00 00 00 00 00 00 56 4f 54 45 00 00 00 00 |........VOTE....| 00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000080 00 00 00 00 00 00 00 00 56 4f 54 45 5f 30 30 30 |........VOTE_000| 00000090 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |0...............|
Ran the following command to reformat the voting disk (/dev/xvdh):
dd if=/dev/zero of=/dev/xvdh bs=8K count=128
The output was:
128+0 records in 128+0 records out 1048576 bytes (1.0 MB) copied, 0.004428 s, 237 MB/s