Step 1—Build the cluster. The command is:

# clmgr add cluster pha_cluster repository=hdisk2 nodes=node1,node2

You should see the following output:

# cltopinfo

Cluster Name: pha_cluster

Cluster Connection Authentication Mode: Standard

Cluster Message Authentication Mode: None

Cluster Message Encryption: None

Use Persistent Labels for Communication: No

Repository Disk: hdisk2

Cluster IP Address:

There are 2 node(s) and 1 network(s) defined

NODE node1:

Network net_ether_01

node1 10.6.51.121

NODE node2:

Network net_ether_01

node2 10.6.51.221

No resource groups defined

Step 2—Define persistent addresses for each node. The command:

# clmgr add persistent_ip 192.168.3.1 network=net_ether_01 node=node1

# clmgr a pe 192.168.3.2 network=net_ether_01 node=node2

The result:

# clmgr -v q pe

NAME="node1admin"

IPADDR="192.168.3.1"

NODE="node1"

SITE=""

NETWORK="net_ether_01"

INTERFACE=""

NETTYPE="ether"

TYPE="persistent"

ATTR="public"

GLOBALNAME=""

HADDR=""

NAME="node2admin"

IPADDR="192.168.3.2"

NODE="node2"

SITE=""

NETWORK="net_ether_01"

INTERFACE=""

NETTYPE="ether"

TYPE="persistent"

ATTR="public"

GLOBALNAME=""

HADDR=""

Step 3—Define two service addresses, one for each node’s resource group. The command:

The result:

# clmgr -v q se

NAME="appAsvc"

IPADDR="192.168.3.10"

NODE=""

SITE="ignore"

NETWORK="net_ether_01"

INTERFACE=""

NETTYPE="ether"

TYPE="service"

ATTR="public"

GLOBALNAME=""

HWADDR=""

NAME="appBsvc"

IPADDR="192.168.3.20"

NODE=""

SITE="ignore"

NETWORK="net_ether_01"

INTERFACE=""

NETTYPE="ether"

TYPE="service"

ATTR="public"

GLOBALNAME=""

HWADDR=""

Step 4—Define two application controllers, one for each node’s resource group. The command:

# clmgr add application_controller appActrl \ startscript=/lab/config/startA stopscript=/lab/config/stopA

# clmgr a ac appBctrl startscript=/lab/config/startB \ stopscript=/lab/config/stopB

The result:

# clmgr -v q ac

NAME="appActrl"

MONITORS=""

STARTSCRIPT="/lab/config/startA"

STOPSCRIPT="/lab/config/stopA"

NAME="appBctrl"

MONITORS=""

STARTSCRIPT="/lab/config/startB"

STOPSCRIPT="/lab/config/stopB"

Step 5—Define two resource groups with resources, one for each node. The command is as follows. Note: The policy parameters must be uppercase.

# clmgr add resource_group appAgroup nodes=node1,node2 startup=OHN \ fallover=FNPN fallback=NFB service_label=appAsvc \ applications=appActrl volume_group=vgA fs_before_ipaddr=true

# clmgr add resource_group appBgroup nodes=node2,node1 startup=OHN \ fallover=FNPN fallback=NFB service_label=appBsvc \ applications=appBctrl volume_group=vgB

The result:

# cltopinfo

Cluster Name: pha_cluster

Cluster Connection Authentication Mode: Standard

Cluster Message Authentication Mode: None

Cluster Message Encryption: None

Use Persistent Labels for Communication: No

Repository Disk: hdisk2

Cluster IP Address:

There are 2 node(s) and 1 network(s) defined

NODE node1:

Network net_ether_01

appBsvc 192.168.3.20

appAsvc 192.168.3.10

node1 10.6.51.121

NODE node2:

Network net_ether_01

appBsvc 192.168.3.20

appAsvc 192.168.3.10

node2 10.6.51.221

Resource Group appAgroup

Startup Policy Online On Home Node Only

Fallover Policy Fallover To Next Priority Node In The List

Fallback Policy Never Fallback

Participating Nodes node1 node2

Service IP Label appAsvc

Resource Group appBgroup

Startup Policy Online On Home Node Only

Fallover Policy Fallover To Next Priority Node In The List

Fallback Policy Never Fallback

Participating Nodes node2 node1

Service IP Label appBsvc

Step 6—Synchronize the cluster definitions with the command:

# clmgr sync cluster

The result (some output was pruned):

# clmgr sync cluster

Verification to be performed on the following:

Cluster Topology

Cluster Resources

Retrieving data from available cluster nodes. This could take a few minutes.

Start data collection on node node1

Start data collection on node node2

Verifying Cluster Topology...

Completed 10 percent of the verification checks

node1 net_ether_01

node2 net_ether_01

Verifying Cluster Resources...

Completed 40 percent of the verification checks

appActrl appAgroup

appBctrl appBgroup

Completed 100 percent of the verification checks

Remember to redo automatic error notification if configuration has changed.

Verification has completed normally.

Committing any changes, as required, to all available nodes...

Adding any necessary PowerHA SystemMirror for AIX entries to /etc/inittab and /etc/rc.net for IP Address Takeover on node node1.

Adding any necessary PowerHA SystemMirror for AIX entries to /etc/inittab and /etc/rc.net for IP Address Takeover on node node2.

Verification has completed normally.

WARNING: Multiple communication interfaces are recommended for networks that

use IP aliasing in order to prevent the communication interface from

becoming a single point of failure. There are fewer than the recommended

number of communication interfaces defined on the following node(s) for

the given network(s):

Node: Network:

---------------------------------- ------------------------------

WARNING: Network option "nonlocsrcroute" is set to 0 and will be set to 1 on during HACMP startup on the following nodes:

node1

node2

WARNING: Application monitors are required for detecting application failures

in order for HACMP to recover from them. Application monitors are started

by HACMP when the resource group in which they participate is activated.

The following application(s), shown with their associated resource group,

do not have an application monitor configured:

Step 7—Check state of cluster services. The command:

# clmgr -a state query cluster

# clmgr -cv -a name,state query node

The result:

# clmgr -a state query cluster

STATE="OFFLINE"

# clmgr -cv -a name,state query node

# NAME:STATE

node1:OFFLINE

node2:OFFLINE

Step 8—Start cluster services on both nodes with the command:

# clmgr start cluster

The result:

# clmgr -a state q cluster

STATE="STABLE"

# clmgr -cv -a name,state,raw_state q node

# NAME:STATE:RAW_STATE

node1:NORMAL:ST_STABLE

node2:NORMAL:ST_STABLE

Step 9—Check the status of the resource groups with the command:

# clmgr -cv -a name,state,current_node q rg

The result:

# clmgr -cv -a name,state,current_node q rg

# NAME:STATE:CURRENT_NODE

appAgroup:ONLINE:node1

appBgroup:ONLINE:node2

At this point, checking the resources in the resource group must be done using AIX commands. Although there’s state information for many of the cluster objects via clmgr, it doesn’t provide state information for objects like the service address or application-controller scripts. Let’s move on to some testing.

Step 10—Move the appAgroup resource group to node2 with the command:

# clmgr mv rg appAgroup node=node2

The result:

# clmgr mv rg appAgroup node=node2

Attempting to move resource group appAgroup to node node2.

Waiting for the cluster to process the resource group movement request....

Waiting for the cluster to stabilize....

Broadcast message from root@node1 (tty) at 16:29:04 ...

appA stopping

...........

Resource group movement successful.

Resource group appAgroup is online on node node2.

Cluster Name: pha_cluster

Resource Group Name: appAgroup

Primary instance(s):

The following node temporarily has the highest priority for this instance:

node2, user-requested rg_move performed on Wed Dec 1 16:28:56 2010

Node Group State

---------------------------- ---------------

node1 OFFLINE

node2 ONLINE

Resource Group Name: appBgroup

Node Group State

---------------------------- ---------------

node2 ONLINE

node1 OFFLINE

Step 11—Stop cluster services on node2 with the Move Resource Groups option. The command:

# clmgr stop node node2 manage=move

The result:

# clmgr -cv -a name,state,raw_state q node

# NAME:STATE:RAW_STATE

node1:NORMAL:ST_STABLE

node2:OFFLINE:ST_INIT

# clmgr -cv -a name,state,current_node q rg

# NAME:STATE:CURRENT_NODE

appAgroup:ONLINE:node1

appBgroup:ONLINE:node1

Step 12—Restart cluster services on node2. The command (using online alias instead of start) is:

# clmgr on node node2

The result:

# clmgr -cv -a name,state,raw_state q node

# NAME:STATE:RAW_STATE

node1:NORMAL:ST_STABLE

node2:NORMAL:ST_STABLE

# clmgr -cv -a name,state,current_node q rg

# NAME:STATE:CURRENT_NODE

appAgroup:ONLINE:node1

appBgroup:ONLINE:node1

Step 13—Move the appBgroup resource group back to node2 using the command:

# clmgr mv rg appBgroup node=node2

The result:

# clmgr -cv -a name,state,current_node q rg

# NAME:STATE:CURRENT_NODE

appAgroup:ONLINE:node1

appBgroup:ONLINE:node2

Step 14—To make a snapshot of all of this good work, use the command:

# clmgr mk sn clmgr_snap description="Snap of clmgr example cluster"

The result:

# clmgr mk sn clmgr_snap description="Snap of clmgr example cluster"

clsnapshot: Creating file /usr/es/sbin/cluster/snapshots/clmgr_snap.odm.

clsnapshot: Creating file /usr/es/sbin/cluster/snapshots/clmgr_snap.info.

clsnapshot: Executing clsnapshotinfo command on node: node1...

clsnapshot: Executing clsnapshotinfo command on node: node2...

clsnapshot: Succeeded creating Cluster Snapshot: clmgr_snap

Usability features

As shown above, many aliases or shortcuts are available in the utility. For a complete list, visit the man page. Misspell something or get a parameter wrong, and clmgr tells you clearly what is wrong. For example:

# clmgr add service_ip appAsvc network=net_ether01 netmask=255.255.255.0

ERROR: failed to create "appAsvc".

Network net_ether01 cannot be found in the configuration.

Do you want to do a query but you’re not sure what object names were used in PowerHA SystemMirror for a given class? Provide any name and consult the very user-friendly error message.

# clmgr q rg what_group

ERROR: "what_group" does not appear to exist!

Available Resource Groups:

appAgroup

appBgroup

Notice, the problem value is clearly shown, and all possible correct values are also shown. To find all off the resource groups that have the option to mount filesystems before the IP addresses are acquired:

# clmgr q rg fs_before_ipaddr=true

appAgroup

Troubleshooting and Logging

Troubleshooting a utility like clmgr is arguably not all that important. If it doesn’t work, switch to SMIT and get the task done. But, if you are trying to get clmgr to do something and you are planning to use it extensively in the future, it does produce some very good log information. The logging is done to /var/hacmp/log/clutils.log (or wherever clutils.log is kept on your system). Further, the clutils.log contains the output of the action, so if the output has overrun the scroll buffer, it’s all safely in the log. An excerpt of the log is shown here. This is the result of starting cluster services on node2.

CLMGR STARTED (9153:10354698:5177392): Wed Dec 1 16:56:43 CET 2010

CLMGR USER (9153:10354698:5177392): ::root:system

CLMGR COMMAND (9153:7471358:10354698): clmgr -T 9153 online node node2

CLMGR ACTUAL (9153:7471358:10354698): start_node node2

CLMGR RETURN (9153:7471358:10354698): 0

CLMGR STDERR -- BEGIN (9153:10354698:5177392): Wed Dec 1 17:00:10 CET 2010

CLMGR STDERR -- END (9153:10354698:5177392): Wed Dec 1 17:00:10 CET 2010

CLMGR ENDED (9153:10354698:5177392): Wed Dec 1 17:00:10 CET 2010

Some useful bits of information are contained in the output, including the user executing the command, the command entered by the user (CLMGR COMMAND) and the actual command executed. he return code is also displayed, which may be helpful if a failure had occurred.

This is just the beginning of troubleshooting. If the problem is with the executed command, not with clmgr, then other logs or the AIX configuration will have to be consulted. For example, the failure to configure the repository disk during the initial synchronization will not be solved by looking in clutils.log. That will most likely be solved using syslog.

clmgr is a Viable CLI

While earlier versions of clvt were limited, the updated clmgr is ready for mainstream use. As for the common administrative tasks in PowerHA SystemMirror, I think I’ve shown that this utility covers them all. As you can see, there’s now a viable CLI utility for PowerHA SystemMirror administration.

Many more features are available. For example, to execute a DARE operation, you could change an attribute (like adding a volume group to a resource group) using the Modify action and then synchronize as shown earlier. Additionally, since this is the utility that the IBM IBM System Director plugin for PowerHA SystemMirror uses, you’re assured the underlying infrastructure for the plugin functions well.

While this might not be the right tool for every administrative situation nor will it replace SMIT for PowerHA SystemMirror administration, tasks like starting or stopping services or performing a synchronization can be simplified requiring only a few keystrokes.

Disclaimer: All of the commands shown in this paper worked as shown during the development of the paper. Neither IBM nor the author make any guarantee that the same or similar results will be seen in other cluster environments. Use this document as a reference and guide to using the clmgr utility.