adding a new Node into the cluster

Post date: Aug 11, 2011 2:4:27 PM

Steps to run when adding a new node to the cluster

in this case I am adding a new HP-UX 11.31 node to a cluster which has HP Serviceguard extenstions - this is because our target hosts already had Serviceguard on so we had to add the HP SGEfRAC extension to our build.

cleardown excessive logs from current node that will run the addNode, for example $CRS_HOME/rdbms/audit contains many ASM audit logs

Modify the parameter JRE_MEMORY_OPTIONS to be -mx1024m or even -mx2048m in $CRS_HOME/oui/oraparm.ini

= this was required when I got the error which matched the Oracle Support Note Addnode.sh Fails With PRKC-1025 [ID 1085893.1]

Run the following cluvfy on the cluster node to check that the hardware and operating system on the new host to be added to the cluster is ready (where node3 is the name of the new node to add.

node1/grid$ cluvfy stage -post hwos -n node99

Performing post-checks for hardware and operating system setup 

Checking node reachability...

Node reachability check passed from node "node1"  

Checking user equivalence...

User equivalence check passed for user "grid" 

Checking node connectivity... 

Checking hosts config file... 

ERROR:

PRVF-4190 : Verification of the hosts config file failed 

Node connectivity passed for subnet "10.10.31.0" with node(s) node3

TCP connectivity check passed for subnet "10.10.31.0"

Interfaces found on subnet "10.10.41.0" that are likely candidates for VIP are:

node3 lan41:10.10.41.207 

Interfaces found on subnet "10.10.31.0" that are likely candidates for a private interconnect are:

node3 lan10:10.10.31.57 

Node connectivity check passed 

Check for multiple users with UID value 0 passed 

Post-check for hardware and operating system setup was successful.

Once this has passed all checks you can run a more indepth check by comparing the new node to an existing node in the cluster.

node1/grid$ cluvfy comp peer -refnode node1 -n node3 -orainv oinstall -osdba asmdba -verbose

Verifying peer compatibility

Checking peer compatibility...

Compatibility check: Physical memory [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      31.9797GB (3.35331E7KB)   31.451GB (3.2978728E7KB)  mismatched

Physical memory check failed

Compatibility check: Available memory [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      22.8951GB (2.4007304E7KB)  2.1266GB (2229860.0KB)    mismatched

Available memory check failed

Compatibility check: Swap space [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      94.4172GB (9.9003564E7KB)  93.9152GB (9.8477192E7KB)  mismatched

Swap space check failed

Compatibility check: Free disk space for "/var/tmp/" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      3.4796GB (3648672.0KB)    3.2417GB (3399192.0KB)    mismatched

Free disk space check failed

Compatibility check: User existence for "grid" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      grid(338)                 grid(338)                 matched

User existence for "grid" check passed

Compatibility check: Group existence for "oinstall" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      oinstall(200)             oinstall(200)             matched

Group existence for "oinstall" check passed

Compatibility check: Group existence for "asmdba" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      asmdba(204)               asmdba(204)               matched

Group existence for "asmdba" check passed

Compatibility check: Group membership for "grid" in "oinstall (Primary)" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      yes                       yes                       matched

Group membership for "grid" in "oinstall (Primary)" check passed

Compatibility check: Group membership for "grid" in "asmdba" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      yes                       yes                       matched

Group membership for "grid" in "asmdba" check passed

Compatibility check: Run level [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      3                         3                         matched

Run level check passed

Compatibility check: System architecture [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      ia64                      ia64                      matched

System architecture check passed

Compatibility check: Kernel version [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      HP-UX B.11.31             HP-UX B.11.31             matched

Kernel version check passed

Compatibility check: Kernel param "ksi_alloc_max" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      33600                     33600                     matched

Kernel param "ksi_alloc_max" check passed

Compatibility check: Kernel param "executable_stack" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      0                         0                         matched

Kernel param "executable_stack" check passed

Compatibility check: Kernel param "max_thread_proc" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      8000                      8000                      matched

Kernel param "max_thread_proc" check passed

Compatibility check: Kernel param "maxdsiz" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      3221225472                3221225472                matched

Kernel param "maxdsiz" check passed

Compatibility check: Kernel param "maxdsiz_64bit" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      274877906944              274877906944              matched

Kernel param "maxdsiz_64bit" check passed

Compatibility check: Kernel param "maxssiz" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      134217728                 134217728                 matched

Kernel param "maxssiz" check passed

Compatibility check: Kernel param "maxssiz_64bit" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      2147483648                2147483648                matched

Kernel param "maxssiz_64bit" check passed

Compatibility check: Kernel param "maxuprc" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      27000                     27000                     matched

Kernel param "maxuprc" check passed

Compatibility check: Kernel param "msgmni" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      4096                      4096                      matched

Kernel param "msgmni" check passed

Compatibility check: Kernel param "msgtql" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      4096                      4096                      matched

Kernel param "msgtql" check passed

Compatibility check: Kernel param "ncsize" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      42172                     42172                     matched

Kernel param "ncsize" check passed

Compatibility check: Kernel param "maxfiles_lim" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      63488                     63488                     matched

Kernel param "maxfiles_lim" check passed

Compatibility check: Kernel param "maxfiles" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      63488                     63488                     matched

Kernel param "maxfiles" check passed

Compatibility check: Kernel param "nflocks" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      32788                     32788                     matched

Kernel param "nflocks" check passed

Compatibility check: Kernel param "ninode" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      242048                    242048                    matched

Kernel param "ninode" check passed

Compatibility check: Kernel param "nkthread" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      250000                    250000                    matched

Kernel param "nkthread" check passed

Compatibility check: Kernel param "nproc" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      30000                     30000                     matched

Kernel param "nproc" check passed

Compatibility check: Kernel param "semmni" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      8192                      8192                      matched

Kernel param "semmni" check passed

Compatibility check: Kernel param "semmns" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      60000                     60000                     matched

Kernel param "semmns" check passed

Compatibility check: Kernel param "semmnu" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      27000                     27000                     matched

Kernel param "semmnu" check passed

Compatibility check: Kernel param "semvmx" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      32767                     32767                     matched

Kernel param "semvmx" check passed

Compatibility check: Package existence for "OS-Core" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      OS-Core-B.11.31-0         OS-Core-B.11.31-0         matched

Package existence for "OS-Core" check passed

Compatibility check: Package existence for "ServiceGuard" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      ServiceGuard-A.11.20.00-0  ServiceGuard-A.11.20.00-0  matched

Package existence for "ServiceGuard" check passed

Compatibility check: Package existence for "SG-RAC-Tool" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      SG-RAC-Tool-A.11.20.00-0  SG-RAC-Tool-A.11.20.00-0  matched

Package existence for "SG-RAC-Tool" check passed

Verification of peer compatibility was unsuccessful.

Checks did not pass for the following node(s):

        node3

Again all of these checks should then be marked "matched", however the physical and available memory checks never seem to match and so always marks the test as failed.

But as the difference in memory sizes is hardly worth worrying about, especially the available if the cluster node is in use, I guess we can carry on...

now you are ready to run the addNode.sh command

as this install is not using GNS I have to specify the command like so...

node1/grid$ $CRS_HOME/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={node3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={node3-vip}"

An orainstRoot.sh script needs to be run on the new host (when instructed by the addNode process)

node3/root$ cd /u01/app/oraInventory

node3/root$ ./orainstRoot.sh

Creating the Oracle inventory pointer file (/var/opt/oracle/oraInst.loc)

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

node3/root$ cd /u01/app/grid/11.2.0

node3/root$ ./root.sh