1) Assumptions
============
> Already built and configured a three-node Oracle RAC 10g Release 2 environment
OS : OEL 5.6
DATABASE : 10.2.0.1.0
ASM
node names RAC1, RAC2 and RAC3
> Each node in the existing Oracle RAC cluster has a copy of the Oracle Clusterware and Oracle Database software installed on their local disks. The current two-node Oracle RAC environment does not use shared Oracle homes for the Clusterware or Database software
> The existing Oracle RAC 10g environment makes use of a clustered file system (OCFS2) to store the two files required to be shared by Oracle Clusterware; namely the Oracle Cluster Registry (OCR) file and the Voting Disk
> Automatic Storage Management (ASM) is being used as the file system and volume manager for all Oracle physical database files (data, online redo logs, control files, archived redo logs) and a Flash Recovery Area (Used by RMAN)
> The third node in the current Oracle RAC environment named RAC3 (running the brij3 instance) will be removed from the cluster making it a two-node cluster.
2) NETWORK DETAILS OF NODES
===========================
First node details
Device* IP Address* Subnet mask *Default gateway address
eth0 * 192.168.2.131 * 255.255.255.0 * 192.168.2.1
eth1 * 10.10.10.31 * 255.255.255.0 * <leave empty>
Second node details
Device* IP Address* Subnet mask *Default gateway address
eth0 * 192.168.2.132 * 255.255.255.0 * 192.168.2.1
eth1 * 10.10.10.32 * 255.255.255.0 * <leave empty>
Third node details
Device* IP Address* Subnet mask *Default gateway address
eth0 * 192.168.2.133 * 255.255.255.0 * 192.168.2.1
eth1 * 10.10.10.33 * 255.255.255.0 * <leave empty>
3) REMOVE DATABASE INSTANCE
===========================
A ) Using DBCA
=============
The following steps can be used to remove an Oracle10g instance from a clustered database using DBCA - even if the instance on the node is not available.
First, verify that you have a good backup of the Oracle Configuration Repository (OCR) using ocrconfig:$ ocrconfig -showbackup
Next, run the DBCA from one of the nodes you are going to keep. The database should remain up as well as leaving the departing instance up and running (if it is available).
cd $ORACLE_HOME/bin
$ dbca
Within the DBCA, perform the following steps:
1) Choose "Oracle Real Application Clusters database" and click [Next].
2) Choose "Instance Management" and click [Next].
3) Choose "Delete an instance" and click [Next].
4) On the next screen, select the cluster database from which you want to remove the instance from. You will need to supply the system privilege (SYSDBA) username and password and click [Next].
5) On the next screen, a list of cluster database instances will appear. Highlight the instance you would like to delete (linux3 in my example) and click [Next].
6) If you have services configured, they will need to be reassigned. Modify the services so that each service can run on one of the remaining instances. Set "not used" for each service regarding the instance that is to be deleted. click [Finish].
7) Acknowledge the dialog box by clicking [Ok] when asked to confirm you want to delete the selected instance.
8) Acknowledge the second dialog by clicking [Ok] when asked to confirm the DBCA will remove the Oracle instance and all associated OFA directory structure. All information about this instance will be deleted.
9) After the DBCA has removed the instance, click [No] when prompted to perform another operation. The DBCA will exit.
IMPORTANT >>>>>
===========================================================
If the database is in archive log mode, the DBA may receive the following errors:
ORA-00350 or ORA-00312
This may occur because the DBCA cannot drop the current log, as it needs archived. This issue is fixed with 10.1.0.3 patchset. If the DBA encounters this error, click the [Ignore] button and when the DBCA completes, manually archive the logs for the deleted instance and drop the log group:
SQL> alter system archive log all;
SQL> alter database drop logfile group 3;
==============================================================
B) USING SRVCTL
===============
Use the srvctl command-line utility from a pre-existing / available node in the cluster to remove the instance (from the node to be removed) from the cluster. This should be run as the oracle UNIX user account as follows:
$ srvctl remove instance -d brij -i brij3
Remove instance brij3 for the database brij? (y/[n]) y
4) VERIFY REDO THREADS
======================
Verify that the redo thread for the dropped instance has been removed by querying v$log
SQL> select group#, thread#, status from v$log;
GROUP# THREAD# STATUS
---------- ---------- ---------------------------------
1 1 INACTIVE
2 1 CURRENT
3 2 INACTIVE
4 2 CURRENT
If for any reason the redo thread is not disabled then disable the thread:
SQL> alter database disable public thread 3;
5) VERIFY INSTANCE REMOVED
===========================
$ ./srvctl config database -d brij
rac1 brij1 /u01/app/oracle/product/10.2.0/db_1
rac2 brij2 /u01/app/oracle/product/10.2.0/db_1
$ ./crs_stat |grep ins
NAME=ora.brij.brij1.inst
NAME=ora.brij.brij2.inst
6) REMOVE ASM INSTANCE
========================
Stop and remove ASM instance which is running on RAC3
$ ./srvctl stop asm -n rac3
$ ./srvctl remove asm -n rac3
$ ./srvctl config asm -n rac3
If the removal of the ASM instance was successful, you should simply get your prompt back with no output. If however, you receive a record back (as shown below for RAC1) , then the removal of the ASM instance failed.
$ ./srvctl config asm -n rac1
+ASM1 /u01/app/oracle/product/10.2.0/db_1
7) RUN NETCA TO REMOVE Network Configuration
============================================
cd $ORACLE_HOME/bin
$ ./netca
Perform the following steps within the NETCA:
1) Choose "Cluster Configuration" and click [Next].
2) Only select the node you are removing and click [Next].
3) Choose "Listener Configuration" and click [Next].
4) Choose "Delete" and delete any listeners configured on the node you are removing. Acknowledge the dialog box to delete the listener configuration.
You should see below message after netca completes.
Oracle Net Services Configuration:
rac3...
Deleted listener: LISTENER_RAC3
Oracle Net Services configuration successful. The exit code is 0
8) REMOVE NODEAPPS FROM RAC3 NODE
====================================
From a pre-existing node (i.e. rac1), remove the nodeapps from the node you are removing as the root UNIX user account
[root@rac1 bin]# ./srvctl stop nodeapps -n rac3
CRS-0210: Could not find resource 'ora.rac3.LISTENER_RAC3.lsnr'.
[root@rac1 bin]# ./srvctl remove nodeapps -n rac3
Please confirm that you intend to remove the node-level applications on node rac3 (y/[n]) y
9) CHECK CURRENT CRS STATUS BY CRS_STAT
=======================================
$ ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....j1.inst application ONLINE ONLINE rac1
ora....j2.inst application ONLINE ONLINE rac2
ora.brij.db application ONLINE ONLINE rac1
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
$
As you can see above NOTHING is now running on RAC3 node.
10) UPDATE NODE LIST IN OUI
==========================
The next step is to update the node list using the updateNodeList option to the OUI as the oracle user. This procedure will remove the node to be deleted from the list of node locations maintained by the OUI by listing only those remaining nodes.
The only file that gets modified is $ORACLE_BASE/oraInventory/ContentsXML/inventory.xml. Here is the command used for removing RAC3 from the list.
Notice that the DISPLAY variable needs to be set even though the GUI does not run.
$ DISPLAY=<machine_or_ip_address>:0.0; export DISPLAY
$ORACLE_HOME/oui/bin/runInstaller -ignoreSysPrereqs -updateNodeList ORACLE_HOME=/u01/app/oracle/product/10.1.0/db_1 CLUSTER_NODES=rac1,rac2
Note that the command above may produce the following error which can safely be ignored:
PRKC-1002 : All the submitted commands did not execute successfully
In the end it will show :
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
>>> Ignoring required pre-requisite failures. Continuing...
$
11) STOP THE CRS STACK AND REMOVE THE ocr.loc FILE
===============================================
If the node to be removed is still available and running the CRS stack, the DBA will need to stop the CRS stack and remove the ocr.loc file.
These tasks should be performed as the root user account and on the node that is to be removed from the cluster.
The nosharedvar option assumes the ocr.loc file is not on a shared file system (which is the case in my example). If the file does exist on a shared file system, then specify sharedvar.
From the node to be removed (i.e. RAC3) and as the root user, run the following:
[root@rac3 install]# ./rootdelete.sh remote nosharedvar
CRS-0210: Could not find resource 'ora.rac3.LISTENER_RAC3.lsnr'.
CRS-0210: Could not find resource 'ora.rac3.ons'.
CRS-0210: Could not find resource 'ora.rac3.vip'.
CRS-0210: Could not find resource 'ora.rac3.gsd'.
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.
Checking to see if Oracle CRS stack is down...
Oracle CRS stack is not running.
Oracle CRS stack is down now.
Removing script for Oracle Cluster Ready services
Updating ocr file for downgrade
Cleaning up SCR settings in '/etc/oracle/scls_scr'
[root@rac3 install]#
12) RUN SCRIPT rootdeletenode.sh
==============================
Next, using the node name and CRS-assigned node number for the node to be deleted, run the rootdeletenode.sh command as follows. Keep in mind that this command should be run from a pre-existing / available node (i.e. rac1) in the cluster as the root UNIX user account:
$ cd $ORA_CRS_HOME/install
[root@rac1 install]# ./rootdeletenode.sh rac3,3
CRS-0210: Could not find resource 'ora.rac3.LISTENER_RAC3.lsnr'.
CRS-0210: Could not find resource 'ora.rac3.ons'.
CRS-0210: Could not find resource 'ora.rac3.vip'.
CRS-0210: Could not find resource 'ora.rac3.gsd'.
CRS-0210: Could not find resource ora.rac3.vip.
CRS nodeapps are deleted successfully
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully deleted 14 values from OCR.
Key SYSTEM.css.interfaces.noderac3 marked for deletion is not there. Ignoring.
Successfully deleted 5 keys from OCR.
Node deletion operation successful.
'rac3,3' deleted successfully
13) VERIFY NODE DELETION
========================
[root@rac1 install]# cd $ORA_CRS_HOME/bin
[root@rac1 bin]# ./olsnodes -n
rac1 1
rac2 2
No RAC3 entries should come up.
14) REMOVE CRS NODE FROM OUI
==============================
Now, switch back to the oracle UNIX user account on the same pre-existing node (rac1) and run the runInstaller command to update the OUI node list, however this time for the CRS installation ($ORA_CRS_HOME). This procedure will remove the node to be deleted from the list of node locations maintained by the OUI by listing only those remaining nodes.
The only file that I know of that gets modified is $ORACLE_BASE/oraInventory/ContentsXML/inventory.xml. Here is the command I used for removing rac3 from the list.
Notice that the DISPLAY variable needs to be set even though the GUI does not run.
$ DISPLAY=<machine_or_ip_address>:0.0; export DISPLAY
$ORACLE_HOME/oui/bin/runInstaller -ignoreSysPrereqs -updateNodeList ORACLE_HOME=/u01/app/oracle/product/10.1.0/crs CLUSTER_NODES=rac1,rac2
The OUI now contains the valid nodes that are part of the cluster!
15) MANUALLY REMOVE ORACLE 10G RAC INSTALLATION FILES
=======================================================
From the deleted node (RAC3) perform the following tasks as the root UNIX user account:
Remove ORACLE_HOME and ORA_CRS_HOME:
==========================================
# rm -rf /u01/app/oracle/product/10.2.0/db_1
# rm -rf /u01/app/oracle/product/10.2.0/crs
Remove all ASM directories from the node to be deleted (linux3) as the oracle user account:
=================================================================================
$ cd $ORACLE_BASE/admin
$ rm -rf +ASM
Remove ASMLib from the Node to be Deleted
========================================
The Linux nodes in the current Oracle RAC database make use of ASMLib so that the raw block devices managed by ASM use ASMLib calls as opposed to standard Linux I/O system calls.
We will remove the ASMLib kernel driver, supporting software, and associated directories. Run the following from the node to be deleted (rac3) as the root user account:
[root@rac3 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [ OK ]
[root@rac3 ~]# rpm -qa | grep oracleasm
oracleasm-2.6.18-238.el5debug-2.0.5-1.el5
oracleasm-2.6.18-238.el5-2.0.5-1.el5
oracleasmlib-2.0.4-1.el5
oracleasm-support-2.1.4-1.el5
oracleasm-2.6.18-238.el5xen-2.0.5-1.el5
# rpm -ev oracleasm-support-2.1.4-1.el5 oracleasm-2.6.18-238.el5xen-2.0.5-1.el5 oracleasmlib-2.0.4-1.el5
warning: /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave
# rm -f /etc/sysconfig/oracleasm.rpmsave
# rm -f /etc/rc.d/init.d/oracleasm
# rm -f /etc/rc0.d/*oracleasm*
# rm -f /etc/rc1.d/*oracleasm*
# rm -f /etc/rc2.d/*oracleasm*
# rm -f /etc/rc3.d/*oracleasm*
# rm -f /etc/rc4.d/*oracleasm*
# rm -f /etc/rc5.d/*oracleasm*
# rm -f /etc/rc6.d/*oracleasm*
Remove all init scripts and soft links (for Linux). ( For a list of init scripts and soft links for other UNIX platforms, see Metalink Note: 269320.1)
=========================================
# rm -f /etc/init.d/init.cssd
# rm -f /etc/init.d/init.crs
# rm -f /etc/init.d/init.crsd
# rm -f /etc/init.d/init.evmd
# rm -f /etc/rc2.d/K96init.crs
# rm -f /etc/rc2.d/S96init.crs
# rm -f /etc/rc3.d/K96init.crs
# rm -f /etc/rc3.d/S96init.crs
# rm -f /etc/rc5.d/K96init.crs
# rm -f /etc/rc5.d/S96init.crs
# rm -Rf /etc/oracle/scls_scr
Remove all remaining files:
========================
# rm -rf /etc/oracle
# rm -f /etc/oratab
# rm -f /etc/oraInst.loc
# rm -rf /etc/ORCLcluster
# rm -rf /u01/app/oracle/oraInventory
# rm -rf /u01/app/oracle/product
# rm -rf /u01/app/oracle/admin
# rm -f /usr/local/bin/coraenv
# rm -f /usr/local/bin/dbhome
# rm -f /usr/local/bin/oraenv
Remove all CRS/EVM entries from the file /etc/inittab:
================================================
h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null
*********************DEINSTALL ORACLE CLUSTERWARE SOFTWARE FROM THE REMOVED NODE BY running runinstaller************************************
You have successfully removed your RAC node now!