Ubuntu LXC OEL OVS ASM GNS RAC (12c)

Hi sportsfans well today I'm working on building a 2-node Oracle 12c Real Application Cluster (RAC) using Ubuntu 14.04.1 as my base operating system, and Oracle Enterprise Linux 6.5 LXC Containers for my two RAC nodes running ontop of the Ubuntu. You might be one of those people who would ask "Why would you do that?" There are many answers. One answer is "Because it seems like LXC for the first time I can remember makes building Oracle RAC on Ubuntu a very interesting and reachable goal." Or I could go with the Sir Edmund Hillary answer "Because it was there" (Mt. Everest). I think the main reason is that I am absolutely thrilled about the possibility of running Oracle RAC on Ubuntu Linux in Oracle Enterprise Linux 6.5 LXC containers.

Note that this site is under construction, however, the final result is already achieved - the 2-node 12c RAC on LXC containers on Ubuntu 14.04.1 base OS is built and running. Because therefore this is a work-in-progress (WIP) and is primarily a place at this point where I can record my own notes on building this, plus improvements and tweaks, etc., therefore I make no assurances as to the flow or readability of the content: there may be sudden breaks in flow where I add an update noting how something was redesigned or improved (or fixed).

I will be adding the details about how storage was presented and many many other details, so this blog post will grow for some time. Check back for additional information if the topic of building Oracle Grid Infrastructure on Ubuntu Linux in Oracle Enteprise Linux LXC containers is of interest to you.

Important Please Note as usual that this is not a supported configuration for a production deployment, neither from a Linux Containers perspective in general, nor from a supported Operating System perspective (Ubuntu Linux is not officially supported by Oracle Corporation for database or grid infrastructure) and, as usual Jim, if you or any of your IMF team are killed or captured, the State Department will disavow all knowledge of your mission. Good Luck, Jim. This tape will self-destruct in 5 seconds....

This build uses the following technologies:

LXC Linux Containers for the 2 RAC Nodes

Ubuntu 14.04.1 Desktop Edition 64-bit with all latest updates from the following repositores shown below for the Base OS

Oracle Enterprise Linux 6.5 for the two RAC-node LXC containers

OpenvSwitch for the networking

Oracle 12c Grid Infrastructure (GI) version 12.1.0.2.0

Oracle 12c Database (RDBMS) version 12.1.0.2.0

Oracle Grid Naming Service (GNS)

DHCP isc-dhcp-server

Named (bind9) DNS server

InMon sflow for performance monitoring

KVM (single guest is used for providing SCST Linux iSCSI SAN LUNs for the RAC)

Note : DHCP an BIND9 are deployed so as to update DNS automatically when new LXC containers are added

Note : All LXC Linux containers in this setup use OpenvSwitch for networking. They DO NOT use the lxcbr0 default LXC out-of-the-box networking solution.

Note : InMon sflow is designed to automatically add hosts to sflow monitoring, also through DNS

Note : Regarding Ubuntu repositores, below, note that I get the "ubuntu-lxc" trusty-tahr daily build updates, and I apply them when notified they are availalble

These are my currently active Ubuntu repositories. I always install all updates as soon as I am aware the are available. My Ubuntu base OS is not a "locked-down" deployment because it is on my laptop which I keep updated with all latest Ubuntu mainline updates (as well as the LXC daily build updates, as noted below).

Update 26 October 2014

Opted to use block storage and not NFS after all as originally planned for this 2-node RAC and went with the option of just mounting the entire /dev/mapper directory of the Ubuntu 14.04.1 host inside the two LXC containers. The configuration passed cluster verification for storage with respect the ASM LUNs, and I knew from building the previous standalone 12c GI that it works that way. This change was made because after checking the Oracle documentation found that whereas in 11gR2 GI clustering NFS was supported for OCR and VOTE, now in 12c, NFS is no longer supported for OCR and VOTE, and I wanted to build a supported version first time around, so went with the ASM option after all.

Important Note: I'm leaving all this NFS work I did with LXC in here because I am going to build a 2-node Oracle 11gR2 RAC using NFS, so this work will come in handy for that later, but for the purposes of how this 2-node ASM 12c RAC on LXC Linux containers was built, the NFS material herein can be skipped over.

End Update

Here are some notes for me. I'm going for a build that uses NFS as the shared storage for the two LXC RAC nodes. Why? Because presenting block storage that ASM could use to LXC containers has proved challenging. So far I had good luck building a single instance of 12c EE ASM using the technique of just mounting all of /dev/mapper (host) inside the LXC ASM guest. However, this won't work in general because mounting the entire /dev/mapper if you think about it gets kind of problematic when you think about doing that with multiple LXC containers. It lacks a certain elegance on top of which idk if it's even possible.

I found that LXC on Oracle Enterprise Linux 6.5 base operating system also likes NFS mounts inside the LXC container (running on a btrfs FS) so I'm going to make this my defacto standard for my first go at this (building RAC on Ubuntu Linux 14.04.1 base OS using Oracle Enterprise Linux 6.5 containers).

Ubuntu LXC Guide is found here.

The first thing I found after mounting some test NFS mounts was that upon reboot my LXC container would not boot. I got this message:

gstanden@vmem1:~$ sudo lxc-start -n lxcora01 -F

lxc-start: lsm/apparmor.c: apparmor_process_label_set: 200 No such file or directory - failed to change apparmor profile to lxc-container-default

lxc-start: sync.c: __sync_wait: 51 invalid sequence number 1. expected 4

lxc-start: start.c: __lxc_start: 1087 failed to spawn 'lxcora01'

lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request

lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing name=systemd:lxc/lxcora01-2

lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request

lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing perf_event:lxc/lxcora01-2

lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request

lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing memory:lxc/lxcora01-2

lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request

lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing hugetlb:lxc/lxcora01-2

lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request

lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing freezer:lxc/lxcora01-2

lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request

lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing devices:lxc/lxcora01-2

lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request

lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing blkio:lxc/lxcora01-2

lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request

lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing cpuacct:lxc/lxcora01-2

lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request

lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing cpu:lxc/lxcora01-2

lxc-start: cgmanager.c: cgm_remove_cgroup: 503 call to cgmanager_remove_sync failed: invalid request

lxc-start: cgmanager.c: cgm_remove_cgroup: 505 Error removing cpuset:lxc/lxcora01-2

lxc-start: lxc_start.c: main: 341 The container failed to start.

lxc-start: lxc_start.c: main: 345 Additional information can be obtained by setting the --logfile and --logpriority options.

gstanden@vmem1:~$

I was at Starbucks with my mom who is 80 and who I take out for coffee quite a bit, and then I keep working at the Starbucks. Well at Starbucks my LXC container with the new NFS mounts would not start. So there are several steps that thank God I found on the internet which got me through this latest fire trial walk on coals.

First, I found these links here at Technology Talk website and here at TQuerci's Weblog. These sites indicated that you had to make some changes to the /etc/apparmor.d/lxc/lxc-default file, so I made the recommended changes and that file now looks like this as shown below. Thanks very much to that website and weblog for keeping this project moving forward with that very helpful information about setup for NFS in LXC containers.

gstanden@vmem1:~$ cat /etc/apparmor.d/lxc/lxc-default

# Do not load this file. Rather, load /etc/apparmor.d/lxc-containers, which

# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default flags=(attach_disconnected,mediate_deleted) {

#include <abstractions/lxc/container-base>

# the container may never be allowed to mount devpts. If it does, it

# will remount the host's devpts. We could allow it to do it with

# the newinstance option (but, right now, we don't).

deny mount fstype=devpts,

# allow nfs mount everywhere

mount fstype=nfs,

mount fstype=nfs4,

mount fstype=rpc_pipefs,

}

gstanden@vmem1:~$

Well unfortunately, I still got some error that was simlar to the first one (but different). Unfortunately it's not available on my screen anymore and I can't cut and paste it because I stupidly did a clear screen, but you get the idea, this still did not fix it completely.

Well, Ubuntu LXC documentation in this case came to the rescue. I followed the steps here and ran this command as shown below which supposedly would fix this problem possibly (and together with another step, it did). Now as to what this command actually means, does, etc. idk. I am just trying to get Oracle RAC running in some LXC containers on Ubuntu why only God knows and since I'm still an intermediate level LXC newb, I don't entirely know what these commands did, but one thing they did actually do is get my LXC container to start normally again (with a couple of additional steps which I will also share).

sudo apparmor_parser -R /etc/apparmor.d/usr.bin.lxc-start

sudo ln -s /etc/apparmor.d/usr.bin.lxc-start /etc/apparmor.d/disabled

So after these steps, still had issue. But that turned out to be from having mucked about trying to add a second network interface to my LXC container and now knowing how, which resulted in a port hanging off my sw1 OpenvSwitch and so I had to delete that manually as shown below (you won't need to do this for sure).

sudo ovs-vsctl del-port sw1 lxcora01b

Ok so after that was done, there was one more step that was required, I had to add some single line add-in to my LXC config file as shown below (bold line at end is the one I added).

root@vmem1:~# cd /var/lib/lxc/lxcora01

root@vmem1:/var/lib/lxc/lxcora01# cat config

# Template used to create this container: /usr/share/lxc/templates/lxc-oracle

# Parameters passed to the template:

# For additional config options, please look at lxc.container.conf(5)

lxc.network.type = veth

lxc.network.flags = up

# lxc.network.link = lxcbr0

lxc.network.script.up = /etc/network/if-up.d/lxc-ifup

lxc.network.script.down = /etc/network/if-down.d/lxc-ifdown

lxc.network.veth.pair = lxcora01

# GLS 2014-10-22 Above 3 lines added for OpenvSwitch networking.

lxc.network.hwaddr = 00:16:3e:5d:ae:70

lxc.rootfs = /var/lib/lxc/lxcora01/rootfs

# Common configuration

lxc.include = /usr/share/lxc/config/oracle.common.conf

# Container configuration for Oracle Linux 6.5

lxc.arch = x86_64

lxc.utsname = lxcora01

lxc.cap.drop = sys_resource

lxc.cap.drop = setfcap setpcap

# Networking

lxc.network.name = eth0

lxc.network.mtu = 1500

lxc.network.hwaddr = fe:cb:23:4e:a9:22

lxc.cgroup.devices.allow = a # Allow access to all devices

lxc.cgroup.devices.allow = a 10:236 rwm

lxc.cgroup.devices.allow = a 252:* rwm

lxc.cgroup.devices.allow = a 8:* rwm

lxc.mount.entry = /dev/mapper /var/lib/lxc/lxcora01/rootfs/dev/mapper none defaults,bind,create=dir 0 0

lxc.mount.auto = proc sys cgroup

lxc.mount.auto = proc:rw sys:rw cgroup-full:rw

lxc.aa_allow_incomplete = 1

lxc.aa_profile = unconfined

root@vmem1:/var/lib/lxc/lxcora01#

Now the LXC container lxcora01 would start and it would also get it's DHCP ip address that is on my OpenvSwitch sw1 and it mounted it's NFS shares. Halleluluah! The NFS shares are going to be the shared storage. LXC container lxcora01 is just a sort of test bed where I am trying to confirm that I can create all the things that will be needed. Here are the NFS shares that are setup for the RAC on NFS in the LXC container as shown below. Shown below are both the output of "df -TH" and also how the /etc/fstab is setup for NFS.

[root@lxcora01 ~]# df -TH

Filesystem Type Size Used Avail Use% Mounted on

tmpfs tmpfs 4.0G 662M 3.3G 17% /dev/shm

vmem1:/shared_config nfs 964G 465G 450G 51% /shared_config

vmem1:/shared_grid nfs 964G 465G 450G 51% /shared_grid

vmem1:/shared_home nfs 964G 465G 450G 51% /shared_home

vmem1:/shared_data nfs 964G 465G 450G 51% /shared_data

[root@lxcora01 ~]# cat /etc/fstab

tmpfs /dev/shm tmpfs rw,exec,size=3500m 0 0

vmem1:/shared_config /shared_config nfs rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0 0 0

vmem1:/shared_grid /shared_grid nfs rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0 0 0

vmem1:/shared_home /shared_home nfs rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0 0 0

vmem1:/shared_data /shared_data nfs rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0 0 0

[root@lxcora01 ~]#

So I have storage. Presumably, subject to verification, I should be able to setup two new LXC containers and mount this NFS share to both of them for my shared RAC storage.

Update 26 October 2014

As previously noted in another update, I opted to use ASM block storage after all, not NFS. However, also as previously noted, I've kept all the NFS information in this blog post because at some point an 11gR2 RAC on LXC containers will be built using NFS.

This note is that to get NFS to mount at boot time, I had to add the resolution information for the NFS server (vmem1.vmem.org) to /etc/hosts, as it does not seem to be getting the information successfully at boot time from DNS. Therefore, below is my configuration for the system with respect to all these various needs. The setup involves edits to all of the following files:

/etc/hosts

/etc/nsswitch.conf

/etc/dhcp/dhclient.conf

/etc/resolv.conf

and these files in their current configuration are shown, with some relevant sections bolder where applicable, as shown below.

[root@lxcora02 ~]# cat /etc/dhcp/dhclient.conf <-- This puts the GNS information into /etc/resolv.conf on boot

prepend domain-name-servers 10.207.39.3;

append domain-name " lxcrac-gns.vmem.org";

[root@lxcora02 ~]# cat /etc/resolv.conf

options attempts:2 timeout:1 <-- This is the resolv.conf format that is required for GNS

; generated by /sbin/dhclient-script

search vmem.org lxcrac-gns.vmem.org

nameserver 10.207.39.3

nameserver 10.207.39.1

[root@lxcora02 ~]# cat /etc/hosts <-- This is required so that the NFS mounts will mount at boot

127.0.0.1 localhost lxcora02

10.207.39.88 lxcora03.vmem.org lxcora03

10.207.39.87 lxcora02.vmem.org lxcora02

10.207.39.1 vmem1.vmem.org vmem1

[root@lxcora02 ~]# cat /etc/nsswitch.conf

#

# /etc/nsswitch.conf

#

# An example Name Service Switch config file. This file should be

# sorted with the most-used services at the beginning.

#

# The entry '[NOTFOUND=return]' means that the search for an

# entry should stop if the search in the previous entry turned

# up nothing. Note that if the search failed due to some other reason

# (like no NIS server responding) then the search continues with the

# next entry.

#

# Valid entries include:

#

# nisplus Use NIS+ (NIS version 3)

# nis Use NIS (NIS version 2), also called YP

# dns Use DNS (Domain Name Service)

# files Use the local files

# db Use the local database (.db) files

# compat Use NIS on compat mode

# hesiod Use Hesiod for user lookups

# [NOTFOUND=return] Stop searching if not found so far

#

# To use db, put the "db" in front of "files" for entries you want to be

# looked up first in the databases

#

# Example:

#passwd: db files nisplus nis

#shadow: db files nisplus nis

#group: db files nisplus nis

passwd: files

shadow: files

group: files

#hosts: db files nisplus nis dns

hosts: dns files nisplus nis db <-- Want DNS to be the preferential name resolution method, so it goes first

# Example - obey only what nisplus tells us...

#services: nisplus [NOTFOUND=return] files

#networks: nisplus [NOTFOUND=return] files

#protocols: nisplus [NOTFOUND=return] files

#rpc: nisplus [NOTFOUND=return] files

#ethers: nisplus [NOTFOUND=return] files

#netmasks: nisplus [NOTFOUND=return] files

bootparams: nisplus [NOTFOUND=return] files

ethers: files

netmasks: files

networks: files

protocols: files

rpc: files

services: files

netgroup: nisplus

publickey: nisplus

automount: files nisplus

aliases: files nisplus

[root@lxcora02 ~]#

Additional update to the update (a short while later). The below shows what would seem to be the ideal settings which are changed a bit from above for /etc/resolv.conf and /etc/dhcp/dhclient.conf so that (a) all external resolutions will work and (b) resolution of RAC VIPs will also work via GNS.

[root@lxcora03 ~]# nslookup lxcora03-vip.lxcrac-gns.vmem.org

Server: 10.207.39.3

Address: 10.207.39.3#53

Name: lxcora03-vip.lxcrac-gns.vmem.org

Address: 10.207.39.93

[root@lxcora03 ~]# nslookup lxcora03-vip

;; Got recursion not available from 10.207.39.3, trying next server

Server: 10.207.39.3

Address: 10.207.39.3#53

Name: lxcora03-vip.lxcrac-gns.vmem.org

Address: 10.207.39.93

[root@lxcora03 ~]# nslookup oracle652

;; Got recursion not available from 10.207.39.3, trying next server

Server: 10.207.39.1

Address: 10.207.39.1#53

Name: oracle652.vmem.org

Address: 10.207.39.74

[root@lxcora03 ~]# cat /etc/dhcp/dhclient.conf

prepend domain-name-servers 10.207.39.3;

append domain-name " lxcrac-gns.vmem.org";

[root@lxcora03 ~]# cat /etc/resolv.conf

options attempts:2 timeout:1 rotate

; generated by /sbin/dhclient-script

search vmem.org lxcrac-gns.vmem.org

nameserver 10.207.39.3

nameserver 10.207.39.1

[root@lxcora03 ~]#

A Note about ntp

The ntp was installed according to some notes at oracle-base. It doesn't seem to run however in the LXC containers. So I uninstalled it using yum. And guess what? The Oracle has failed to startup! Even though ntp was not running, it seems has wanted it there. So I reinstalled ntp, and voila! has stack and all Oracle services started. Need to figure out what that was all about.

Adding a Second OpenvSwitch Ethernet Interface to LXC Container

Update 26 October 2014

I ended up using OpenvSwitch for all the networking and decommissioned the "lxcbr0" that I was using mostly as a convenience and out of sheer bold laziness. First, the lxcbr0 was having some issue where I could not ping or ssh between the LXC containers (which I believe occurred after I had setup OpenvSwitch for the public interface which may have somehow broken inter-container ssh), even though both were on the same 10.0.3.1 default LXC network, so opted to just create an additional OpenvSwitch for the private RAC interconnect network and completely decommission the out-of-the-box lxcbr0 default networking solution and go with an all-OpenvSwitch solution. I made some desultory efforts to solve this, but with no success on that front, decided that a better use of hair-pulling time would be to just setup the private RAC interconnect on OpenvSwitch since that is of much more interest anyway, and since OpenvSwitch is a fully-functional professional-grade layer 2 switch. Also, using OpenvSwitch makes the upcoming exercise to completely re-ip a GNS RAC a much more palatable and enjoyable exercise.

End Update

The LXC container config file used to obtain a second ethernet interface (i.e. the config file for having two ethernet interfaces in the LXC container) is shown below. The single most important reference used for getting a second network interface setup for the RAC private interconnect in the LXC container was here at Ta-Ching Chen's blog and the Network Administrator Blog here, and also some good info at box.matto.nl here, was also very helpful (this was actually the first reference that I found that got things rolling). This blog also had another page that gave the best information I had found on how to use LXC with OpenvSwitch networking. The solution for this RAC on LXC uses interface eth0 (on the OpenvSwitch network) as the public interface, and eth1, on the default LXC "lxcbr0" bridge for the private RAC interconnect network. The solution could of course use a second network defined in bind9 (named) for the private network, but it was convenient to use lxcbr0 default LXC networking for the private interconnect for this first attempt at building a 2-node Oracle 12c RAC on LXC. Below is shown the config file that is used. The two stanzas in the "Networking" section are the ethernet interfaces. Some entries of interest are bolded as shown below.

root@vmem1:~# cd /var/lib/lxc/lxcora01

root@vmem1:/var/lib/lxc/lxcora01# cat config

# Template used to create this container: /usr/share/lxc/templates/lxc-oracle

# Parameters passed to the template:

# For additional config options, please look at lxc.container.conf(5)

# Networking

lxc.network.type = veth

lxc.network.flags = up

lxc.network.script.up = /etc/network/if-up.d/lxc-ifup-sw1

lxc.network.script.down = /etc/network/if-down.d/lxc-ifdown-sw1

lxc.network.veth.pair = lxcora01-pub

lxc.network.name = eth0

lxc.network.mtu = 1500

lxc.network.hwaddr = fe:cb:23:4e:a9:22

lxc.network.type = veth

lxc.network.flags = up

lxc.network.link = lxcbr0

lxc.network.veth.pair = lxcora01-pri

lxc.network.name = eth1

lxc.network.mtu = 1500

#lxc.network.hwaddr = fe:cb:23:4e:a9:23

lxc.network.hwaddr = 00:16:3e:5d:ae:70

# Common configuration

lxc.rootfs = /var/lib/lxc/lxcora01/rootfs

lxc.include = /usr/share/lxc/config/oracle.common.conf

# Container configuration for Oracle Linux 6.5

lxc.arch = x86_64

lxc.utsname = lxcora01

lxc.cap.drop = sys_resource

lxc.cap.drop = setfcap setpcap

# Devices

lxc.cgroup.devices.allow = a # Allow access to all devices

lxc.cgroup.devices.allow = a 10:236 rwm

lxc.cgroup.devices.allow = a 252:* rwm

lxc.cgroup.devices.allow = a 8:* rwm

lxc.mount.entry = /dev/mapper /var/lib/lxc/lxcora01/rootfs/dev/mapper none defaults,bind,create=dir 0 0

lxc.mount.auto = proc sys cgroup

lxc.mount.auto = proc:rw sys:rw cgroup-full:rw

lxc.aa_allow_incomplete = 1

# GLS 2014-10-23 NFS

lxc.aa_profile = unconfined

root@vmem1:/var/lib/lxc/lxcora01#

A show of the OpenvSwitch configuration helps to show how the networking is setup for the public interface for the RAC as shown below. The lxcora01-pub port off of sw1 is shown in bold.

root@vmem1:/var/lib/lxc/lxcora01# ovs-vsctl show

2fc24710-34b5-4aa2-a32d-4e7bcb1afa1a

Bridge "sw2"

Port "sw2"

Interface "sw2"

type: internal

Port "t5"

Interface "t5"

Port "t1"

Interface "t1"

Port "t6"

Interface "t6"

Port "t3"

Interface "t3"

Port "t2"

Interface "t2"

Port "t4"

Interface "t4"

Bridge "sw1"

Port "s2"

Interface "s2"

Port "s4"

Interface "s4"

Port "s1"

Interface "s1"

Port "s6"

Interface "s6"

Port "sw1"

Interface "sw1"

type: internal

Port "s5"

Interface "s5"

Port "lxcora01-pub"

Interface "lxcora01-pub"

Port "s3"

Interface "s3"

Bridge "sw3"

Port "w4"

Interface "w4"

Port "w2"

Interface "w2"

Port "w6"

Interface "w6"

Port "w3"

Interface "w3"

Port "w5"

Interface "w5"

Port "w1"

Interface "w1"

Port "sw3"

Interface "sw3"

type: internal

ovs_version: "2.0.2"

root@vmem1:/var/lib/lxc/lxcora01#

The two scripts refernced in the config file are shown below. These scripts dynamically create the port on the OpenvSwitch and then destroy the port when the LXC container is shut down.

root@vmem1:/etc/network/if-up.d# cat lxc-ifup-sw1

#!/bin/bash

ovsBr='sw1'

ovs-vsctl --may-exist add-br ${ovsBr}

ovs-vsctl --may-exist add-port ${ovsBr} $5

root@vmem1:/etc/network/if-up.d# pwd

/etc/network/if-up.d

root@vmem1:/etc/network/if-up.d#

The similar script for shutdown is shown below.

root@vmem1:/etc/network/if-down.d# cat lxc-ifdown-sw1

#!/bin/bash

ovsBr='sw1'

ovs-vsctl --if-exists del-port ${ovsBr} $5

root@vmem1:/etc/network/if-down.d# pwd

/etc/network/if-down.d

root@vmem1:/etc/network/if-down.d# ls -lrt lxc-ifdown-sw1

-rwxr-xr-x 1 root root 67 Sep 6 20:03 lxc-ifdown-sw1

root@vmem1:/etc/network/if-down.d#

Update 26 October 2014

Couple of updates here. First. when shutting down the LXC container RAC nodes lxcora02 or lxcora03 it is best to use "shutdown -h now" and not "reboot" because the reboot sequence does not execute the "lxc-ifdown-sw*" script, and so when the LXC node tries to restart, it fails because the port on the OpenvSwitch still exists (was not destroyed at shutdown). Thus, shutdown should be used, not reboot.

Found that the "if-exists" clause on the "del-port" command above was causing the "del-port" command to fail so took it out. Here is the form of the commands which is working and which is reproducibly and reliably removing the port at node shutdown, as shown below. This format should be used (without the "if-exists" clause).

Another update, a short while later, also found that even with the "if-exists" clause removed, this command was still sometimes failing, not sure why. So in addition, I also put in an if-then to re-run the command if the first run of the command was unsuccessful. Here is the current scritping being used, so far successfully, for destroying the port on shutdown.

gstanden@vmem1:/etc/network/if-down.d$ cat lxc-ifdown-sw1

#!/bin/bash

ovsBr='sw1'

sudo ovs-vsctl del-port ${ovsBr} $5

gstanden@vmem1:/etc/network/if-down.d$ cat lxc-ifdown-sw4

#!/bin/bash

ovsBr='sw4'

sudo ovs-vsctl del-port ${ovsBr} $5

gstanden@vmem1:/etc/network/if-down.d$

So here is the version of the script for down which seems to be working every time as shown below. You can see the function has been added to re-run the del-port command if it still exists.

gstanden@vmem1:~$ cat /etc/network/if-down.d/lxc-ifdown-sw1

#!/bin/bash

ovsBr='sw1'

sudo ovs-vsctl del-port ${ovsBr} $5

function CheckPortExists

{

sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'

}

PortExists=$(CheckPortExists)

if [ "$PortExists" = "$5" ]

then

sudo ovs-vsctl del-port ${ovsBr} $5

fi

gstanden@vmem1:~$

End Update

Update 27 October 2014

Every now and then the above lxc-ifdown-sw1 script is still failing to clear out the ports from the OpenvSwitch. I'm going to give each LXC container it's own file because the only thing I can think of is that the instances may be concurrently trying to access the files and this is causing them to occassionally fail.

So even larger changes were made today later in the day and so far they seem to be working perfectly. First, I added 3 more private networks so that on this install I will add HAIP 4-network to the build. Second, I have given each interface up/down it's own file so that there will be no chance of interfaces trying to access the same up/down files. Here are those files as shown below.

gstanden@vmem1:/etc/network/if-up.d$ ls -l lxcora*

-rwxr-xr-x 1 root root 117 Oct 27 20:09 lxcora02-ifup-sw1

-rwxr-xr-x 1 root root 117 Oct 27 20:09 lxcora02-priv1-ifup-sw4

-rwxr-xr-x 1 root root 117 Oct 27 20:31 lxcora02-priv2-ifup-sw5

-rwxr-xr-x 1 root root 117 Oct 27 20:31 lxcora02-priv3-ifup-sw6

-rwxr-xr-x 1 root root 117 Oct 27 20:31 lxcora02-priv4-ifup-sw7

-rwxr-xr-x 1 root root 117 Oct 27 20:09 lxcora03-ifup-sw1

-rwxr-xr-x 1 root root 117 Oct 27 20:09 lxcora03-priv1-ifup-sw4

-rwxr-xr-x 1 root root 117 Oct 27 20:32 lxcora03-priv2-ifup-sw5

-rwxr-xr-x 1 root root 117 Oct 27 20:32 lxcora03-priv3-ifup-sw6

-rwxr-xr-x 1 root root 117 Oct 27 20:33 lxcora03-priv4-ifup-sw7

gstanden@vmem1:/etc/network/if-up.d$ cd ..

gstanden@vmem1:/etc/network$ cd if-down.d

gstanden@vmem1:/etc/network/if-down.d$ ls -l lxcora*

-rwxr-xr-x 1 root root 300 Oct 27 20:08 lxcora02-ifdown-sw1

-rwxr-xr-x 1 root root 301 Oct 27 20:20 lxcora02-priv1-ifdown-sw4

-rwxr-xr-x 1 root root 301 Oct 27 20:21 lxcora02-priv2-ifdown-sw5

-rwxr-xr-x 1 root root 301 Oct 27 20:21 lxcora02-priv3-ifdown-sw6

-rwxr-xr-x 1 root root 301 Oct 27 20:21 lxcora02-priv4-ifdown-sw7

-rwxr-xr-x 1 root root 300 Oct 27 20:09 lxcora03-ifdown-sw1

-rwxr-xr-x 1 root root 301 Oct 27 20:20 lxcora03-priv1-ifdown-sw4

-rwxr-xr-x 1 root root 301 Oct 27 20:21 lxcora03-priv2-ifdown-sw5

-rwxr-xr-x 1 root root 301 Oct 27 20:21 lxcora03-priv3-ifdown-sw6

-rwxr-xr-x 1 root root 301 Oct 27 20:21 lxcora03-priv4-ifdown-sw7

gstanden@vmem1:/etc/network/if-down.d$

And here is an example of how those files look as shown below. The files are all the same except for the name of the particular OpenvSwitch as shown below.

gstanden@vmem1:/etc/network/if-down.d$ cat lxc-ifdown-sw1

#!/bin/bash

ovsBr='sw1'

sudo ovs-vsctl del-port ${ovsBr} $5

function CheckPortExists

{

sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'

}

PortExists=$(CheckPortExists)

if [ "$PortExists" = "$5" ]

then

sudo ovs-vsctl del-port ${ovsBr} $5

fi

gstanden@vmem1:/etc/network/if-down.d$

gstanden@vmem1:/etc/network/if-down.d$ diff lxc-ifdown-sw4 lxc-ifdown-sw1

2c2

< ovsBr='sw4'

---

> ovsBr='sw1'

End Update

Edit /etc/dhcp/dhcpclient.conf

To get the /etc/resolv.conf to generate correctly at startup of the LXC container RAC node, it was necessary to edit the /etc/dhcp/dhcpclient.conf file (inside the LXC container) as shown below. Note that the "nameserver 10.0.3.1" is added automatically without need of an entry in dhcpclient.conf and it was only the need to get the LXC container to be able to resolve DNS names on the vmem.org 10.207.39.1 network which drove the need to add these lines to the dhcpclient.conf file.

gstanden@vmem1:~$ ssh root@lxcora01

root@lxcora01's password:

Last login: Fri Oct 24 10:59:59 2014 from vmem1.vmem.org

[root@lxcora01 ~]# cat /etc/dhcp/dhclient.conf

prepend domain-name-servers 10.207.39.1;

supersede domain-search "vmem.org";

[root@lxcora01 ~]#

The /etc/resolv.conf file which results from the lines added to dhcpclient.conf is shown below. This /etc/resolv.conf will resolve named and addresses on both networks.

[root@lxcora01 ~]# cat /etc/resolv.conf

; generated by /sbin/dhclient-script

search vmem.org.

nameserver 10.207.39.1

nameserver 10.0.3.1

[root@lxcora01 ~]#

For example, here is a lookup of the Grid Naming Service (GNS) DNS entry for this RAC.

[root@lxcora01 ~]# nslookup lxcrac-gns

Server: 10.207.39.1

Address: 10.207.39.1#53

Name: lxcrac-gns.vmem.org

Address: 10.207.39.3

[root@lxcora01 ~]# nslookup 10.207.39.3

Server: 10.207.39.1

Address: 10.207.39.1#53

3.39.207.10.in-addr.arpa name = lxcrac-gns.vmem.org.

[root@lxcora01 ~]#

Here's a lookup of the RAC node itself.

[root@lxcora01 ~]# nslookup lxcora01

Server: 10.207.39.1

Address: 10.207.39.1#53

Name: lxcora01.vmem.org

Address: 10.207.39.85

[root@lxcora01 ~]# nslookup 10.207.39.85

Server: 10.207.39.1

Address: 10.207.39.1#53

85.39.207.10.in-addr.arpa name = lxcora01.vmem.org.

[root@lxcora01 ~]#

Notice that if the vmem.org and associated nameserver information were not in /etc/resolv.conf it would resolve the LXC RAC node on the default LXC lxcbr0 network (private RAC interconnect network) instead of the desired public network resolution which is why the additional entries in /etc/resolv.conf are needed.

[root@lxcora01 ~]# cat /etc/resolv.conf

; generated by /sbin/dhclient-script

#search vmem.org.

#nameserver 10.207.39.1

nameserver 10.0.3.1

[root@lxcora01 ~]# nslookup lxcora01

Server: 10.0.3.1

Address: 10.0.3.1#53

Name: lxcora01

Address: 10.0.3.2

[root@lxcora01 ~]#

Setup Oracle GNS (Grid Naming Service) on LXC Container

Martin Carstenbach has a great post on how to test to be sure the GNS setup is correct here at Simplified GNS setup for RAC 11.2.0.2 and newer. That's what is about to be done. Last night the GNS was setup with help from these references. This is my first time using GNS - I've been an Oracle DBA over 20 years but kept putting this off and using static DNS entries for RAC. However , with this setup, so much work has been done setting up this laptop system to use Dynamic DNS for everything, that it was very appealing to stay consistent with that motif so here now finally using GNS which is long overdue. Some references that were found very helpful were hereand also the Oracle official documentation here and Martin Carstenbach has some very useful-looking blogs that I have not used (yet) but which look like they will eventually be consulted since they are chock full of great information here and here.

Shown below are the forward zone lookup file and the reverse zone lookup file with the GNS additions in bold which will be used on this 2-node LXC RAC. This is on the Ubuntu Host OS and included is the directory location of the zone files as well as the checkzone test. The DNS environment on this laptop uses a dynamic DNS and DHCP setup which was built from help from Lee Hutchinson's Big Dinosaur blog here.

root@vmem1:/var/lib/bind# pwd

/var/lib/bind

root@vmem1:/var/lib/bind# cat fwd.vmem.org

$ORIGIN .

$TTL 86400 ; 1 day

vmem.org IN SOA vmem1.vmem.org. postmaster.vmem.org. (

1410240112 ; serial

60 ; refresh (1 minute)

1800 ; retry (30 minutes)

604800 ; expire (1 week)

86400 ; minimum (1 day)

)

NS vmem1.vmem.org.

$ORIGIN vmem.org.

_sflow._udp TXT "txtvers=1" "polling=20" "sampling=512"

SRV 0 0 6343 vmem1

$TTL 3600 ; 1 hour

lxcora01 A 10.207.39.85

TXT "001060b0cef034d0d8b3ec569ea6615637"

$TTL 86400 ; 1 day

lxcora1 A 10.207.39.79

$TTL 3600 ; 1 hour

lxcora3 A 10.207.39.83

TXT "007e7554cced9024a1bac46cc4370293c6"

lxcora5 A 10.207.39.84

TXT "0092ff209913ebc9ca832c943e48eff86f"

$TTL 86400 ; 1 day

oracle631 A 10.207.39.72

oracle632 A 10.207.39.76

oracle652 A 10.207.39.74

vmem1 A 10.207.39.1

lxcrac-gns A 10.207.39.3

$TTL 3600 ; 1 hour

vmem2 A 10.207.39.81

TXT "3118a6c1688c312b822454df8baa10bfff"

; subdomain (lxcrac1.vmem.org) for Oracle RAC GNS

$ORIGIN lxcrac.vmem.org.

@ IN NS lxcrac-gns.vmem.org.

root@vmem1:/var/lib/bind# named-checkzone vmem.org fwd.vmem.org

zone vmem.org/IN: loaded serial 1410240112

OK

root@vmem1:/var/lib/bind#

Update 27 October 2014

So because I had forgotten to change the name of my ASM system diskgroup from the default "DATA" during the GI install to "SYSTEMDG" I tried to move the DATA over to SYSTEMDG but this has become significantly more complicated. Levi Pereira had a very comprehensive procedure, but there were some issues, and I decided to instead uninstall the entire GI and redo it. I had some other reason for a redo too. I wanted to simplify the naming convention for the GNS setup as shown below this is the revised fwd.vmem.org and rev.vmem.org files on the Ubuntu host as shown below. The GNS related parts revised are shown in bold. On this install the delegated domain will simply be "gns1.vmem.org". The VIP was changed to simply be "lxc1". Also, I missed some screenshots during the GI install that pertained to some non-fatal errors, so I'm going to be sure I get those on the next go of installing GI.

root@vmem1:/var/lib/bind# cat fwd.vmem.org

$ORIGIN .

$TTL 86400 ; 1 day

vmem.org IN SOA vmem1.vmem.org. postmaster.vmem.org. (

1410271922 ; serial

60 ; refresh (1 minute)

1800 ; retry (30 minutes)

604800 ; expire (1 week)

86400 ; minimum (1 day)

)

NS vmem1.vmem.org.

$ORIGIN vmem.org.

_sflow._udp TXT "txtvers=1" "polling=20" "sampling=512"

SRV 0 0 6343 vmem1

$TTL 3600 ; 1 hour

lxcora01 A 10.207.39.85

TXT "001060b0cef034d0d8b3ec569ea6615637"

lxcora02 A 10.207.39.87

TXT "001d6eb17bf9f9081897f98548b528bfee"

lxcora03 A 10.207.39.88

TXT "0084a3923f9bbe6559acbb98034af0c10a"

$TTL 86400 ; 1 day

lxcora1 A 10.207.39.79

$TTL 3600 ; 1 hour

lxcora3 A 10.207.39.83

TXT "007e7554cced9024a1bac46cc4370293c6"

lxcora5 A 10.207.39.84

TXT "0092ff209913ebc9ca832c943e48eff86f"

$TTL 86400 ; 1 day

oracle631 A 10.207.39.72

oracle632 A 10.207.39.76

oracle652 A 10.207.39.74

lxc1 A 10.207.39.3

vmem1 A 10.207.39.1

$TTL 3600 ; 1 hour

vmem2 A 10.207.39.81

TXT "3118a6c1688c312b822454df8baa10bfff"

$ORIGIN gns1.vmem.org

@ IN NS lxc1.vmem.org.

root@vmem1:/var/lib/bind#

And this is the reverse lookup zone file information as shown below with the added reverse lookup for lxc1 in bold.

gstanden@vmem1:~$ cat /var/lib/bind/rev.vmem.org

$ORIGIN .

$TTL 86400 ; 1 day

39.207.10.in-addr.arpa IN SOA vmem1.vmem.org. postmaster.vmem.org. (

1410271922 ; serial

3600 ; refresh (1 hour)

1800 ; retry (30 minutes)

604800 ; expire (1 week)

86400 ; minimum (1 day)

)

NS vmem1.vmem.org.

$ORIGIN 39.207.10.in-addr.arpa.

1 PTR vmem1.vmem.org.

3 PTR lxc1.vmem.org.

72 PTR oracle631.vmem.org.

76 PTR oracle632.vmem.org.

$TTL 3600 ; 1 hour

81 PTR vmem2.vmem.org.

85 PTR lxcora01.vmem.org.

87 PTR lxcora02.vmem.org.

88 PTR lxcora03.vmem.org.

$TTL 86400 ; 1 day

90 PTR oracle635.vmem.org.

39.207.10.in-addr.arpa NS vmem1.vmem.org.

vmem1 A 10.207.39.1

gstanden@vmem1:~$

Next just finished getting the /etc/resolv.conf setup for GNS. There was quite a bit of discovery and work there. Here is what the /etc/resolv.conf looks like for this system as correctly configured shown below.

[root@lxcora01 ~]# cat /etc/resolv.conf

options attempts:2 timeout:1

; generated by /sbin/dhclient-script

search vmem.org gns.vmem.org

nameserver 10.207.39.3

nameserver 10.207.39.1

nameserver 10.0.3.1

[root@lxcora01 ~]#

As many know, /etc/resolv.conf gets overwritten at each reboot (i.e. each restart of networking) so some other files have to be configured to get the desired result. For this system, configuration of /etc/dhcp/dhclient.conf was sufficient to setup the required "search" and "nameserver" options (and order of nameservers matters in this case). Shown below is the required /etc/dhcp/dhclient.conf file.

[root@lxcora01 ~]# cat /etc/dhcp/dhclient.conf

prepend domain-name-servers 10.207.39.3,10.207.39.1;

append domain-name " gns.vmem.org";

[root@lxcora01 ~]#

As shown above, the file sets the FIRST nameserver as the GNS nameserver, then sets the vmem.org BIND9 namerserver second. The third nameserver entry is probably irrelevant and is set automatically by other settings (unknown) and does not harm being there afaik, and is part of the default LXC network "lxcbr0" setup, so no effort was made to remove it.

The "options" line at the top of the /etc/resolv.conf was simply typed into the resolv.conf and it was found that it persists across reboots, so apparently the "options" directives are not overwritten when networking restarts. So, the resolv.conf is believed to be complete. There were several references that were very helpful working out how to get the required options in the /etc/resolv.conf at the pages here at good old Cyberciti, here at good old AskUbuntu (for the "append" directive), colleague Edmund White at ServerFault.

End Update

Speed Up SSH

After setting up GNS in the /etc/resolv.conf ssh became markedly slow. The following fixes were found here at the GeekStuff and also here at In Just Five Minutes and worked very well. Here is the /etc/ssh/sshd_config that is working very well with the GNS setup as shown below with the changed entries in bold. The two options that should be used are:

• GSSAPIAuthentication no

• UseDNS no

[root@lxcora02 ~]# cat /etc/ssh/sshd_config

# $OpenBSD: sshd_config,v 1.80 2008/07/02 02:24:18 djm Exp $

# This is the sshd server system-wide configuration file. See

# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/local/bin:/bin:/usr/bin

# The strategy used for options in the default sshd_config shipped with

# OpenSSH is to specify options with their default value where

# possible, but leave them commented. Uncommented options change a

# default value.

#Port 22

#AddressFamily any

#ListenAddress 0.0.0.0

#ListenAddress ::

# Disable legacy (protocol version 1) support in the server for new

# installations. In future the default will change to require explicit

# activation of protocol 1

Protocol 2

# HostKey for protocol version 1

#HostKey /etc/ssh/ssh_host_key

# HostKeys for protocol version 2

#HostKey /etc/ssh/ssh_host_rsa_key

#HostKey /etc/ssh/ssh_host_dsa_key

# Lifetime and size of ephemeral version 1 server key

#KeyRegenerationInterval 1h

#ServerKeyBits 1024

# Logging

# obsoletes QuietMode and FascistLogging

#SyslogFacility AUTH

SyslogFacility AUTHPRIV

#LogLevel INFO

# Authentication:

#LoginGraceTime 2m

#PermitRootLogin yes

#StrictModes yes

#MaxAuthTries 6

#MaxSessions 10

#RSAAuthentication yes

#PubkeyAuthentication yes

#AuthorizedKeysFile .ssh/authorized_keys

#AuthorizedKeysCommand none

#AuthorizedKeysCommandRunAs nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts

#RhostsRSAAuthentication no

# similar for protocol version 2

#HostbasedAuthentication no

# Change to yes if you don't trust ~/.ssh/known_hosts for

# RhostsRSAAuthentication and HostbasedAuthentication

#IgnoreUserKnownHosts no

# Don't read the user's ~/.rhosts and ~/.shosts files

#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!

#PasswordAuthentication yes

#PermitEmptyPasswords no

PasswordAuthentication yes

# Change to no to disable s/key passwords

#ChallengeResponseAuthentication yes

ChallengeResponseAuthentication no

# Kerberos options

#KerberosAuthentication no

#KerberosOrLocalPasswd yes

#KerberosTicketCleanup yes

#KerberosGetAFSToken no

#KerberosUseKuserok yes

# GSSAPI options

#GSSAPIAuthentication yes

GSSAPIAuthentication no

#GSSAPICleanupCredentials yes

GSSAPICleanupCredentials yes

#GSSAPIStrictAcceptorCheck yes

#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,

# and session processing. If this is enabled, PAM authentication will

# be allowed through the ChallengeResponseAuthentication and

# PasswordAuthentication. Depending on your PAM configuration,

# PAM authentication via ChallengeResponseAuthentication may bypass

# the setting of "PermitRootLogin without-password".

# If you just want the PAM account and session checks to run without

# PAM authentication, then enable this but set PasswordAuthentication

# and ChallengeResponseAuthentication to 'no'.

#UsePAM no

UsePAM yes

# Accept locale-related environment variables

AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES

AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT

AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE

AcceptEnv XMODIFIERS

#AllowAgentForwarding yes

#AllowTcpForwarding yes

#GatewayPorts no

#X11Forwarding no

X11Forwarding yes

#X11DisplayOffset 10

#X11UseLocalhost yes

#PrintMotd yes

#PrintLastLog yes

#TCPKeepAlive yes

#UseLogin no

#UsePrivilegeSeparation yes

#PermitUserEnvironment no

#Compression delayed

#ClientAliveInterval 0

#ClientAliveCountMax 3

#ShowPatchLevel no

UseDNS no

#PidFile /var/run/sshd.pid

#MaxStartups 10:30:100

#PermitTunnel no

#ChrootDirectory none

# no default banner path

#Banner none

# override default of no subsystems

Subsystem sftp /usr/libexec/openssh/sftp-server

# Example of overriding settings on a per-user basis

#Match User anoncvs

# X11Forwarding no

# AllowTcpForwarding no

# ForceCommand cvs server

[root@lxcora02 ~]#

Test GNS Configuration Using cluvfy

The configuration of the GNS can be tested as described by Martin Carstenbach here. The results of my test are shown below.

[root@lxcora01 ~]# su - grid

[grid@lxcora01 ~]$ cd grid

[grid@lxcora01 grid]$ ./runcluvfy.sh comp gns -precrsinst -domain vmem.org -vip 10.207.39.3 -verbose -n lxcora01

Verifying GNS integrity

Checking GNS integrity...

Checking if the GNS subdomain name is valid...

The GNS subdomain name "vmem.org" is a valid domain name

Checking if the GNS VIP is a valid address...

GNS VIP "10.207.39.3" resolves to a valid IP address

Checking the status of GNS VIP...

GNS integrity check passed

Verification of GNS integrity was successful.

[grid@lxcora01 grid]$

Update 27 October 2014

Here is the GNS configuration precrsinst check for the reinstall as shown below.

[grid@lxcora02 grid]$ ./runcluvfy.sh comp gns -precrsinst -domain vmem.org -vip 10.207.39.3 -verbose -n lxcora02,lxcora03

Verifying GNS integrity

Checking GNS integrity...

Checking if the GNS subdomain name is valid...

The GNS subdomain name "vmem.org" is a valid domain name

Checking if the GNS VIP is a valid address...

GNS VIP "10.207.39.3" resolves to a valid IP address

Checking the status of GNS VIP...

GNS integrity check passed

Verification of GNS integrity was successful.

[grid@lxcora02 grid]$

End Update

Clone the LXC Container to Create Second RAC Node

When the container is fully configured the quickest way to create the additional RAC node(s) is to just use the lxc-clone to clone the LXC container. The clone command is shown below.

gstanden@vmem1:~$ sudo lxc-clone -o lxcora01 -n lxcora02

Created container lxcora02 as copy of lxcora01

gstanden@vmem1:~$

The lxcora02 container needs just a couple of small manual edits since the clone utility apparently doesn't know about handling OpenvSwitch networking since the default networking is lxcbr0 and also in the original container non-default rac-meaningful names for the veth.pair were used.. The lxc-clone utility did update the MAC addresses just fine, so no issue there. However, the ports on the OpenvSwitch need to be manually edited to have the matching name for the container as shown below (manual edits in bold).

# Template used to create this container: /usr/share/lxc/templates/lxc-oracle

# Parameters passed to the template:

# For additional config options, please look at lxc.container.conf(5)

# Networking

lxc.network.type = veth

lxc.network.flags = up

lxc.network.script.up = /etc/network/if-up.d/lxc-ifup-sw1

lxc.network.script.down = /etc/network/if-down.d/lxc-ifdown-sw1

lxc.network.veth.pair = lxcora02-pub

lxc.network.name = eth0

lxc.network.mtu = 1500

lxc.network.hwaddr = 00:16:3e:16:14:93 <-- Note: lxc-clone updates MAC address automatically as part of the clone process, no manual edit needed nere.

lxc.network.type = veth

lxc.network.flags = up

lxc.network.link = lxcbr0

lxc.network.veth.pair = lxcora02-pri

lxc.network.name = eth1

lxc.network.mtu = 9000

lxc.network.hwaddr = 00:16:3e:3e:a6:b4 <-- Note: lxc-clone updates MAC address automatically as part of the clone process, no manual edit needed here.

# Common configuration

lxc.include = /usr/share/lxc/config/oracle.common.conf

# Container configuration for Oracle Linux 6.5

lxc.arch = x86_64

lxc.cap.drop = sys_resource

lxc.cap.drop = setfcap setpcap

# Devices

lxc.cgroup.devices.allow = a # Allow access to all devices

lxc.cgroup.devices.allow = a 10:236 rwm

lxc.cgroup.devices.allow = a 252:* rwm

lxc.cgroup.devices.allow = a 8:* rwm

lxc.mount.entry = /dev/mapper /var/lib/lxc/lxcora01/rootfs/dev/mapper none defaults,bind,create=dir 0 0

lxc.mount.auto = proc sys cgroup

lxc.mount.auto = proc:rw sys:rw cgroup-full:rw

lxc.aa_allow_incomplete = 1

# GLS 2014-10-23 NFS

lxc.aa_profile = unconfined

lxc.rootfs = /var/lib/lxc/lxcora02/rootfs

lxc.utsname = lxcora02 <-- Note: lxc-clone updates name automatically, no manual edit needed here.

Update 28 October 2014

Now using 4 interfaces for Oracle GI HAIP. The config file is changed for both lxcora02 and lxcora03 as shown below.

root@vmem1:/var/lib/lxc/lxcora02# cat config

# Template used to create this container: /usr/share/lxc/templates/lxc-oracle

# Parameters passed to the template:

# For additional config options, please look at lxc.container.conf(5)

# Networking

lxc.network.type = veth

lxc.network.flags = up

lxc.network.script.up = /etc/network/if-up.d/lxcora02-ifup-sw1

lxc.network.script.down = /etc/network/if-down.d/lxcora02-ifdown-sw1

lxc.network.veth.pair = lxcora02

lxc.network.name = eth0

lxc.network.mtu = 1500

lxc.network.hwaddr = 00:16:3e:16:14:93

lxc.network.type = veth

lxc.network.flags = up

lxc.network.script.up = /etc/network/if-up.d/lxcora02-priv1-ifup-sw4

lxc.network.script.down = /etc/network/if-down.d/lxcora02-priv1-ifdown-sw4

lxc.network.veth.pair = lxcora02-priv1

lxc.network.name = eth1

lxc.network.mtu = 9000

lxc.network.ipv4 = 192.210.39.10

lxc.network.type = veth

lxc.network.flags = up

lxc.network.script.up = /etc/network/if-up.d/lxcora02-priv2-ifup-sw5

lxc.network.script.down = /etc/network/if-down.d/lxcora02-priv2-ifdown-sw5

lxc.network.veth.pair = lxcora02-priv2

lxc.network.name = eth2

lxc.network.mtu = 9000

lxc.network.ipv4 = 192.211.39.10

lxc.network.type = veth

lxc.network.flags = up

lxc.network.script.up = /etc/network/if-up.d/lxcora02-priv3-ifup-sw6

lxc.network.script.down = /etc/network/if-down.d/lxcora02-priv3-ifdown-sw6

lxc.network.veth.pair = lxcora02-priv3

lxc.network.name = eth3

lxc.network.mtu = 9000

lxc.network.ipv4 = 192.212.39.10

lxc.network.type = veth

lxc.network.flags = up

lxc.network.script.up = /etc/network/if-up.d/lxcora02-priv4-ifup-sw7

lxc.network.script.down = /etc/network/if-down.d/lxcora02-priv4-ifdown-sw7

lxc.network.veth.pair = lxcora02-priv4

lxc.network.name = eth4

lxc.network.mtu = 9000

lxc.network.ipv4 = 192.213.39.10

#lxc.network.type = veth

#lxc.network.flags = up

#lxc.network.link = lxcbr0

#lxc.network.name = eth1

#lxc.network.mtu = 1500

lxc.network.hwaddr = 00:16:3e:3e:a6:b4

# Common configuration

lxc.include = /usr/share/lxc/config/oracle.common.conf

# Container configuration for Oracle Linux 6.5

lxc.arch = x86_64

lxc.cap.drop = sys_resource

lxc.cap.drop = setfcap setpcap

# Devices

lxc.cgroup.devices.allow = a # Allow access to all devices

lxc.cgroup.devices.allow = a 10:236 rwm

lxc.cgroup.devices.allow = a 252:* rwm

lxc.cgroup.devices.allow = a 8:* rwm

lxc.mount.entry = /dev/mapper /var/lib/lxc/lxcora02/rootfs/dev/mapper none defaults,bind,create=dir 0 0

lxc.mount.auto = proc sys cgroup

lxc.mount.auto = proc:rw sys:rw cgroup-full:rw

lxc.aa_allow_incomplete = 1

# GLS 2014-10-23 NFS

lxc.aa_profile = unconfined

lxc.rootfs = /var/lib/lxc/lxcora02/rootfs

lxc.utsname = lxcora02

root@vmem1:/var/lib/lxc/lxcora02#

And here is the config file for lxcora03 as shown below.

root@vmem1:/var/lib/lxc/lxcora03# cat config

# Template used to create this container: /usr/share/lxc/templates/lxc-oracle

# Parameters passed to the template:

# For additional config options, please look at lxc.container.conf(5)

# Networking

lxc.network.type = veth

lxc.network.flags = up

lxc.network.script.up = /etc/network/if-up.d/lxcora03-ifup-sw1

lxc.network.script.down = /etc/network/if-down.d/lxcora03-ifdown-sw1

lxc.network.veth.pair = lxcora03

lxc.network.name = eth0

lxc.network.mtu = 1500

lxc.network.hwaddr = 00:16:3e:88:bc:f6

lxc.network.type = veth

lxc.network.flags = up

lxc.network.script.up = /etc/network/if-up.d/lxcora03-priv1-ifup-sw4

lxc.network.script.down = /etc/network/if-down.d/lxcora03-priv1-ifdown-sw4

lxc.network.veth.pair = lxcora03-priv1

lxc.network.name = eth1

lxc.network.mtu = 9000

lxc.network.ipv4 = 192.210.39.11

lxc.network.type = veth

lxc.network.flags = up

lxc.network.script.up = /etc/network/if-up.d/lxcora03-priv2-ifup-sw5

lxc.network.script.down = /etc/network/if-down.d/lxcora03-priv2-ifdown-sw5

lxc.network.veth.pair = lxcora03-priv2

lxc.network.name = eth2

lxc.network.mtu = 9000

lxc.network.ipv4 = 192.211.39.11

lxc.network.type = veth

lxc.network.flags = up

lxc.network.script.up = /etc/network/if-up.d/lxcora03-priv3-ifup-sw6

lxc.network.script.down = /etc/network/if-down.d/lxcora03-priv3-ifdown-sw6

lxc.network.veth.pair = lxcora03-priv3

lxc.network.name = eth3

lxc.network.mtu = 9000

lxc.network.ipv4 = 192.212.39.11

lxc.network.type = veth

lxc.network.flags = up

lxc.network.script.up = /etc/network/if-up.d/lxcora03-priv4-ifup-sw7

lxc.network.script.down = /etc/network/if-down.d/lxcora03-priv4-ifdown-sw7

lxc.network.veth.pair = lxcora03-priv4

lxc.network.name = eth4

lxc.network.mtu = 9000

lxc.network.ipv4 = 192.213.39.11

#lxc.network.type = veth

#lxc.network.flags = up

#lxc.network.link = lxcbr0

#lxc.network.name = eth1

#lxc.network.mtu = 1500

lxc.network.hwaddr = 00:16:3e:11:f8:13

# Common configuration

lxc.include = /usr/share/lxc/config/oracle.common.conf

# Container configuration for Oracle Linux 6.5

lxc.arch = x86_64

lxc.cap.drop = sys_resource

lxc.cap.drop = setfcap setpcap

# Devices

lxc.cgroup.devices.allow = a # Allow access to all devices

lxc.cgroup.devices.allow = a 10:236 rwm

lxc.cgroup.devices.allow = a 252:* rwm

lxc.cgroup.devices.allow = a 8:* rwm

lxc.mount.entry = /dev/mapper /var/lib/lxc/lxcora03/rootfs/dev/mapper none defaults,bind,create=dir 0 0

lxc.mount.auto = proc sys cgroup

lxc.mount.auto = proc:rw sys:rw cgroup-full:rw

lxc.aa_allow_incomplete = 1

# GLS 2014-10-23 NFS

lxc.aa_profile = unconfined

lxc.rootfs = /var/lib/lxc/lxcora03/rootfs

lxc.utsname = lxcora03

root@vmem1:/var/lib/lxc/lxcora03#

Now start up the container and see what starts up leaving the original container lxcora01 down until verification has been done that the actual hostname and networking etc are correct (non-duplicate). Since this is a first for the author too, all these steps will be carefully documented.

Check the MTU 9000 is Working on the Private Interfaces

This link here at Myles Gray shows how to do this.

Check to see if the new container is visible to lxc-ls as shown below.

gstanden@vmem1:~$ sudo lxc-ls -f

[sudo] password for gstanden:

NAME STATE IPV4 IPV6 GROUPS AUTOSTART

------------------------------------------------

lxcora01 STOPPED - - - NO

lxcora02 STOPPED - - - NO

Here's the startup (with "-F" option used to get console output). The startup is normal and exactly the same as the source LXC container as shown below. The messages about apparmor and ulimit were present in the source LXC container lxcora01 as well, and those I am still looking for an answer/solution about, but they dont' affect the functioning of the LXC container for the Oracle install purposes so for now they are not an issue. The startup of the cloned container is shown below.

gstanden@vmem1:~$ sudo lxc-start -n lxcora02 -F

lxc-start: lsm/apparmor.c: mount_feature_enabled: 61 Permission denied - Error mounting securityfs

Welcome to Oracle Linux Server

Setting hostname lxcora02: [ OK ]

Checking filesystems

[ OK ]

Mounting local filesystems: [ OK ]

Enabling /etc/fstab swaps: [ OK ]

Entering non-interactive startup

Calling the system activity data collector (sadc)...

Bringing up loopback interface: [ OK ]

Bringing up interface eth0:

Determining IP information for eth0... done.

[ OK ]

Bringing up interface eth1:

Determining IP information for eth1... done.

[ OK ]

Starting system logger: [ OK ]

Starting rpcbind: [ OK ]

Starting NFS statd: [ OK ]

NFS filesystems queued to be mounted

Mounting filesystems: [ OK ]

Starting sshd: [ OK ]

hsflowd start: OK

Starting crond: [ OK ]

/etc/rc3.d/S96ohasd: line 153: ulimit: max locked memory: cannot modify limit: Operation not permitted

/etc/rc3.d/S96ohasd: line 155: ulimit: %CRS_LIMIT_OPENFILE%: invalid number

Starting ohasd:

chown: cannot access `/dev/mapper/asm_disk*': No such file or directory

chmod: cannot access `/dev/mapper/asm_disk*': No such file or directory

Oracle Linux Server release 6.5

Kernel 3.13.11.6 on an x86_64

lxcora02 login:

Hostname info looks good as shown below. The hostname has been updated automatically by lxc-clone and correctly shows "lxcora02" as shown below.

[root@lxcora02 ~]# uname -a

Linux lxcora02 3.13.11.6 #1 SMP Mon Sep 15 11:54:55 CDT 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@lxcora02 ~]# hostname -a

lxcora02

[root@lxcora02 ~]# hostname -f

localhost

[root@lxcora02 ~]#

The ethernet interfaces also look fine. As expected they have new MAC addresses, and also have new IP addresses as shown below.

[root@lxcora02 ~]# ifconfig

eth0 Link encap:Ethernet HWaddr 00:16:3E:16:14:93

inet addr:10.207.39.87 Bcast:10.207.39.255 Mask:255.255.255.0

inet6 addr: fe80::216:3eff:fe16:1493/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:386 errors:0 dropped:0 overruns:0 frame:0

TX packets:283 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:61965 (60.5 KiB) TX bytes:31126 (30.3 KiB)

eth1 Link encap:Ethernet HWaddr 00:16:3E:3E:A6:B4

inet addr:10.0.3.236 Bcast:10.0.3.255 Mask:255.255.255.0

inet6 addr: fe80::216:3eff:fe3e:a6b4/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1

RX packets:200 errors:0 dropped:0 overruns:0 frame:0

TX packets:15 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:36470 (35.6 KiB) TX bytes:1638 (1.5 KiB)

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:44 errors:0 dropped:0 overruns:0 frame:0

TX packets:44 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:3512 (3.4 KiB) TX bytes:3512 (3.4 KiB)

[root@lxcora02 ~]#

However, as might be expected, trying to start cluster and ASM fails. That is shown below.

[root@lxcora02 ~]# crsctl stat res -t

Oracle Clusterware infrastructure error in CRSCTL (OS PID 1227): ADR home path /u00/app/grid/diag/crs/lxcora02/crs does not exist; ADR initialization will try to create it

CRS-4639: Could not contact Oracle High Availability Services

CRS-4000: Command Status failed, or completed with errors.

[root@lxcora02 ~]#

This is ok because for this example first build, I will be ripping out the Oracle RDBMS and ASM installation. The LXC container lxcora01 was more or less the product of learning how to build Oracle ASM 12c EE on LXC containers, so rather than build a fresh container for RAC and then clone it, I've just cloned this one since it was used to work out and do all the RAC GNS networking setups,, but also has all the prerequisites for ASM installation.

For anyone who might be planning on using this guide, the way this would normally be done would be to just build a fresh Oracle Enterprise Linux container on Ubuntu 14.04.1, then do all the Oracle RAC GNS steps, install all the ASM and RDBMS prerequisites, and then do the clone. So in this guide, at least for now, there will be these extra steps documenting the rip out of the existing ASM and RDBMS from the cloned LXC container lxcora02.

I'm going to startup the original lxcora01 container to verify it still runs normally with lxcora02 running at the same time. Connected to lxcora01 and testing DNS lookup shows that the DHCP and DNS combo has added the new cloned container to the bind9 DNS automatically (thanks to the configuration skills learned from BigDino!) as shown below displaying the correct IP address lookup information for the cloned lxcora02 LXC container (note: this new container was NOT added to DNS manually. It was automatically added to the DNS configuration).

[root@lxcora01 ~]# nslookup lxcora02

Server: 10.207.39.1

Address: 10.207.39.1#53

Name: lxcora02.vmem.org

Address: 10.207.39.87

Showing how this looks below in the DNS configuration. The bolded lines show how DNS automatically added the new LXC container to /var/lib/bind/fwd.vmem.org file. Also, note to self, see how the fwd.vmem.org.jnl file is not created until a new DNS entry is automatically added. Last night the static entries for GNS were added to the fwd.vmem.org (and also to the rev.vmem.org) and the serial updated, and at that time, all that was done was that bind9 was restarted to put the GNS changes into DNS. At that time, no fwd.vmem.org.jnl file was created. I had been wondering what triggered the creation of the *.jnl zone files, and now it's clear that when a new LXC container (or KVM guest) i.e. any "new server" appears on the network, then a *.jnl file is created for the related zone files, in this case fwd.vmem.org.jnl and rev.vmem.org.jnl were both created just now when lxcora02 was automatically added.

root@vmem1:/var/lib/bind# cat fwd.vmem.org

$ORIGIN .

$TTL 86400 ; 1 day

vmem.org IN SOA vmem1.vmem.org. postmaster.vmem.org. (

1410240113 ; serial

60 ; refresh (1 minute)

1800 ; retry (30 minutes)

604800 ; expire (1 week)

86400 ; minimum (1 day)

)

NS vmem1.vmem.org.

$ORIGIN vmem.org.

_sflow._udp TXT "txtvers=1" "polling=20" "sampling=512"

SRV 0 0 6343 vmem1

$TTL 3600 ; 1 hour

lxcora01 A 10.207.39.85

TXT "001060b0cef034d0d8b3ec569ea6615637"

lxcora02 A 10.207.39.87

TXT "001d6eb17bf9f9081897f98548b528bfee"

$TTL 86400 ; 1 day

lxcora1 A 10.207.39.79

$TTL 3600 ; 1 hour

lxcora3 A 10.207.39.83

TXT "007e7554cced9024a1bac46cc4370293c6"

lxcora5 A 10.207.39.84

TXT "0092ff209913ebc9ca832c943e48eff86f"

lxcrac NS lxcrac-gns

$TTL 86400 ; 1 day

lxcrac-gns A 10.207.39.3

oracle631 A 10.207.39.72

oracle632 A 10.207.39.76

oracle652 A 10.207.39.74

vmem1 A 10.207.39.1

$TTL 3600 ; 1 hour

vmem2 A 10.207.39.81

TXT "3118a6c1688c312b822454df8baa10bfff"

root@vmem1:/var/lib/bind# pwd

/var/lib/bind

root@vmem1:/var/lib/bind# ls -lrt *.jnl

-rw-r--r-- 1 bind bind 791 Oct 24 16:26 fwd.vmem.org.jnl <-- *.jnl files were created when lxcora02 was automatically added to the zone files

-rw-r--r-- 1 bind bind 774 Oct 24 16:26 rev.vmem.org.jnl <--

root@vmem1:/var/lib/bind#

For completeness, here's the rev.vmem.org zone file showing how lxcora02 was automatically added here too.

root@vmem1:/var/lib/bind# cat rev.vmem.org

$ORIGIN .

$TTL 86400 ; 1 day

39.207.10.in-addr.arpa IN SOA vmem1.vmem.org. postmaster.vmem.org. (

1410232201 ; serial

3600 ; refresh (1 hour)

1800 ; retry (30 minutes)

604800 ; expire (1 week)

86400 ; minimum (1 day)

)

NS vmem1.vmem.org.

$ORIGIN 39.207.10.in-addr.arpa.

1 PTR vmem1.vmem.org.

3 PTR lxcrac-gns.vmem.org.

72 PTR oracle631.vmem.org.

76 PTR oracle632.vmem.org.

$TTL 3600 ; 1 hour

81 PTR vmem2.vmem.org.

85 PTR lxcora01.vmem.org.

87 PTR lxcora02.vmem.org.

$TTL 86400 ; 1 day

90 PTR oracle635.vmem.org.

39.207.10.in-addr.arpa NS vmem1.vmem.org.

vmem1 A 10.207.39.1

root@vmem1:/var/lib/bind#

Some ping and resolution and ssh tests are done as shown below. Note: the ssh from one node to the other took a few seconds - maybe 10 seconds or so - it's not as fast as I would like. Probably there are some tweaks that a network guru would know to speed the ssh and name resolution up a bit so that connection was much faster. It's not terribly slow, but slower than it should be. Here are the ping and resolution tests as shown below which all work as expected and required.

gstanden@vmem1:~$ ssh root@lxcora01

root@lxcora01's password:

Last login: Fri Oct 24 17:49:59 2014

[root@lxcora01 ~]# nslookup lxcora02

Server: 10.207.39.1

Address: 10.207.39.1#53

Name: lxcora02.vmem.org

Address: 10.207.39.87

[root@lxcora01 ~]# ping -c 3 lxcora02

PING lxcora02.vmem.org (10.207.39.87) 56(84) bytes of data.

64 bytes from lxcora02.vmem.org (10.207.39.87): icmp_seq=1 ttl=64 time=0.868 ms

64 bytes from lxcora02.vmem.org (10.207.39.87): icmp_seq=2 ttl=64 time=0.075 ms

64 bytes from lxcora02.vmem.org (10.207.39.87): icmp_seq=3 ttl=64 time=0.124 ms

--- lxcora02.vmem.org ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2003ms

rtt min/avg/max/mdev = 0.075/0.355/0.868/0.363 ms

[root@lxcora01 ~]# ssh root@lxcora02

The authenticity of host 'lxcora02 (10.207.39.87)' can't be established.

RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'lxcora02,10.207.39.87' (RSA) to the list of known hosts.

root@lxcora02's password:

Last login: Fri Oct 24 17:48:37 2014

[root@lxcora02 ~]# nslookup lxcora01

Server: 10.207.39.1

Address: 10.207.39.1#53

Name: lxcora01.vmem.org

Address: 10.207.39.85

[root@lxcora02 ~]# ping -c 3 lxcora01

PING lxcora01.vmem.org (10.207.39.85) 56(84) bytes of data.

64 bytes from lxcora01.vmem.org (10.207.39.85): icmp_seq=1 ttl=64 time=0.481 ms

64 bytes from lxcora01.vmem.org (10.207.39.85): icmp_seq=2 ttl=64 time=0.073 ms

64 bytes from lxcora01.vmem.org (10.207.39.85): icmp_seq=3 ttl=64 time=0.092 ms

--- lxcora01.vmem.org ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2001ms

rtt min/avg/max/mdev = 0.073/0.215/0.481/0.188 ms

[root@lxcora02 ~]# exit

logout

Connection to lxcora02 closed.

[root@lxcora01 ~]#

It can be seen that lxcora01 Oracle ASM and RDBMS are running fine with both the lxcora01 and it's clone lxcora02 up and running at the same time as shown below.

[root@lxcora01 ~]# crsctl stat res -t

--------------------------------------------------------------------------------

Name Target State Server State details

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA.dg

ONLINE ONLINE lxcora01 STABLE

ora.FRA.dg

ONLINE ONLINE lxcora01 STABLE

ora.IOPS.dg

ONLINE ONLINE lxcora01 STABLE

ora.LISTENER.lsnr

ONLINE ONLINE lxcora01 STABLE

ora.SYSTEMDG.dg

ONLINE ONLINE lxcora01 STABLE

ora.asm

ONLINE ONLINE lxcora01 Started,STABLE

ora.ons

OFFLINE OFFLINE lxcora01 STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.cssd

1 ONLINE ONLINE lxcora01 STABLE

ora.diskmon

1 OFFLINE OFFLINE STABLE

ora.evmd

1 ONLINE ONLINE lxcora01 STABLE

ora.vmem1.db

1 ONLINE ONLINE lxcora01 Open,STABLE

--------------------------------------------------------------------------------

[root@lxcora01 ~]#

The next steps which I will do after driving mom home from her coffee outing will be to (a) rip out the ASM and RDBMS on the lxcora02; and then (b) do an lxc-clone of lxcora02 to lxcora03 which will give a nice clean RAC node so that there is no need to rip out the ASM and RDBMS twice.

Here is the way that the OpenvSwitches look with both LXC containers running. Note that sw1 is my main OpenvSwitch for public (management) network, while sw2 and sw3 are basically created for iscsi SAN storage networks (not being used with LXC because I don't really have a good handle on how to present iSCSI block devices to LXC containers since iscsi packages don't seem to work in the containers). This RAC is going to use NFS as the storage, side note. Here's the OpenvSwitch detail as shown below. The ports to sw1 for lxcora01 and lxcora02 are shown in bold below.

gstanden@vmem1:~$ sudo ovs-vsctl show

[sudo] password for gstanden:

2fc24710-34b5-4aa2-a32d-4e7bcb1afa1a

Bridge "sw2"

Port "sw2"

Interface "sw2"

type: internal

Port "t5"

Interface "t5"

Port "t1"

Interface "t1"

Port "t6"

Interface "t6"

Port "t3"

Interface "t3"

Port "t2"

Interface "t2"

Port "t4"

Interface "t4"

Bridge "sw1"

Port "lxcora02-pub"

Interface "lxcora02-pub"

Port "s2"

Interface "s2"

Port "s4"

Interface "s4"

Port "s1"

Interface "s1"

Port "sw1"

Interface "sw1"

type: internal

Port "s6"

Interface "s6"

Port "s5"

Interface "s5"

Port "lxcora01-pub"

Interface "lxcora01-pub"

Port "s3"

Interface "s3"

Bridge "sw3"

Port "w4"

Interface "w4"

Port "w2"

Interface "w2"

Port "w3"

Interface "w3"

Port "w6"

Interface "w6"

Port "w5"

Interface "w5"

Port "w1"

Interface "w1"

Port "sw3"

Interface "sw3"

type: internal

ovs_version: "2.0.2"

gstanden@vmem1:~$

So now I have taken my mother home to her condo in my motorcar (an Eclipse Special Edition (SE) 2012) and I have watched with my wife the movie Out of Africa (1985) with Meryl Streep and Robert Redford on Turner Classic Movies (TCM) and we have shared a bottle of Trader Joe "3 Buck Chuck" white wine, and now I am going to rip out Oracle RDBMS and Oracle ASM from lxcora02 in preparation for making a clone of lxcora02 to lxcora03 and then hopefully building a 2-node RAC on the lxc nodes lxcora02 and lxcora03. Here is the first step in that process as shown below, starting DBCA so I can delete the database and database binaries. I will not show those screenshots, I already know how to do that :-) ! )(smiling man smoking a cigar!).

gstanden@vmem1:~$ ssh -Y -C oracle@lxcora02

The authenticity of host 'lxcora02 (10.207.39.87)' can't be established.

RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'lxcora02,10.207.39.87' (RSA) to the list of known hosts.

oracle@lxcora02's password:

Permission denied, please try again.

oracle@lxcora02's password:

Last login: Sun Sep 21 23:45:02 2014 from 10.0.3.1

[oracle@lxcora02 ~]$ xclock

[oracle@lxcora02 ~]$ dbca

The database had been deleted using DBCA. Next the RDBMS home and binaries are deconfigured using 12c deinstall utility as shown below. For this I refreshed my memory on how to do this at Thinking Out Loud weblog here.

gstanden@vmem1:~$ ssh -Y -C oracle@lxcora02

oracle@lxcora02's password:

Last login: Fri Oct 24 22:55:08 2014 from vmem1.vmem.org

[oracle@lxcora02 ~]$ $ORACLE_HOME/deinstall/deinstall

Checking for required files and bootstrapping ...

Please wait ...

Location of logs /u00/app/oraInventory/logs/

############ ORACLE DECONFIG TOOL START ############

######################### DECONFIG CHECK OPERATION START #########################

## [START] Install check configuration ##

Checking for existence of the Oracle home location /u00/app/oracle/product/12.1.0/dbhome_1

Oracle Home type selected for deinstall is: Oracle Single Instance Database

Oracle Base selected for deinstall is: /u00/app/oracle

Checking for existence of central inventory location /u00/app/oraInventory

## [END] Install check configuration ##

Network Configuration check config START

Network de-configuration trace file location: /u00/app/oraInventory/logs/netdc_check2014-10-24_11-15-21-PM.log

Network Configuration check config END

Database Check Configuration START

Database de-configuration trace file location: /u00/app/oraInventory/logs/databasedc_check2014-10-24_11-15-27-PM.log

Use comma as separator when specifying list of values as input

Specify the list of database names that are configured in this Oracle home []: VMEM1

###### For Database 'VMEM1' ######

Specify the type of this database (1.Single Instance Database|2.Oracle Restart Enabled Database) [1]: 2

Specify the list of nodes on which this database has instances [lxcora02]:

Specify the diagnostic destination location of the database [/u00/app/oracle/diag/rdbms/vmem1]:

Specify the storage type used by the Database ASM|FS []: ASM

Database Check Configuration END

Oracle Configuration Manager check START

OCM check log file location : /u00/app/oraInventory/logs//ocm_check9034.log

Oracle Configuration Manager check END

######################### DECONFIG CHECK OPERATION END #########################

####################### DECONFIG CHECK OPERATION SUMMARY #######################

Oracle Home selected for deinstall is: /u00/app/oracle/product/12.1.0/dbhome_1

Inventory Location where the Oracle home registered is: /u00/app/oraInventory

The following databases were selected for de-configuration : VMEM1

Database unique name : VMEM1

Storage used : ASM

Checking the config status for CCR

Oracle Home exists and CCR is configured

CCR check is finished

Do you want to continue (y - yes, n - no)? [n]: y

A log of this session will be written to: '/u00/app/oraInventory/logs/deinstall_deconfig2014-10-24_11-15-21-PM.out'

Any error messages from this session will be written to: '/u00/app/oraInventory/logs/deinstall_deconfig2014-10-24_11-15-21-PM.err'

######################## DECONFIG CLEAN OPERATION START ########################

Database de-configuration trace file location: /u00/app/oraInventory/logs/databasedc_clean2014-10-24_11-16-02-PM.log

Database Clean Configuration START VMEM1

This operation may take few minutes.

Database Clean Configuration END VMEM1

Network Configuration clean config START

Network de-configuration trace file location: /u00/app/oraInventory/logs/netdc_clean2014-10-24_11-16-35-PM.log

De-configuring Naming Methods configuration file...

Naming Methods configuration file de-configured successfully.

De-configuring Local Net Service Names configuration file...

Local Net Service Names configuration file de-configured successfully.

De-configuring backup files...

Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

Oracle Configuration Manager clean START

OCM clean log file location : /u00/app/oraInventory/logs//ocm_clean9034.log

Oracle Configuration Manager clean END

######################### DECONFIG CLEAN OPERATION END #########################

####################### DECONFIG CLEAN OPERATION SUMMARY #######################

Successfully de-configured the following database instances : VMEM1

Cleaning the config for CCR

Cleaning the CCR configuration by executing its binaries

CCR clean is finished

#######################################################################

############# ORACLE DECONFIG TOOL END #############

Using properties file /tmp/deinstall2014-10-24_11-15-03PM/response/deinstall_2014-10-24_11-15-21-PM.rsp

Location of logs /u00/app/oraInventory/logs/

############ ORACLE DEINSTALL TOOL START ############

####################### DEINSTALL CHECK OPERATION SUMMARY #######################

A log of this session will be written to: '/u00/app/oraInventory/logs/deinstall_deconfig2014-10-24_11-15-21-PM.out'

Any error messages from this session will be written to: '/u00/app/oraInventory/logs/deinstall_deconfig2014-10-24_11-15-21-PM.err'

######################## DEINSTALL CLEAN OPERATION START ########################

## [START] Preparing for Deinstall ##

Setting LOCAL_NODE to localhost

Setting CRS_HOME to false

Setting oracle.installer.invPtrLoc to /tmp/deinstall2014-10-24_11-15-03PM/oraInst.loc

Setting oracle.installer.local to false

## [END] Preparing for Deinstall ##

Setting the force flag to false

Setting the force flag to cleanup the Oracle Base

Oracle Universal Installer clean START

Detach Oracle home '/u00/app/oracle/product/12.1.0/dbhome_1' from the central inventory on the local node : Done

Delete directory '/u00/app/oracle/product/12.1.0/dbhome_1' on the local node : Done

The Oracle Base directory '/u00/app/oracle' will not be removed on local node. The directory is not empty.

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END

## [START] Oracle install clean ##

Clean install operation removing temporary directory '/tmp/deinstall2014-10-24_11-15-03PM' on node 'localhost'

## [END] Oracle install clean ##

######################### DEINSTALL CLEAN OPERATION END #########################

####################### DEINSTALL CLEAN OPERATION SUMMARY #######################

Successfully detached Oracle home '/u00/app/oracle/product/12.1.0/dbhome_1' from the central inventory on the local node.

Successfully deleted directory '/u00/app/oracle/product/12.1.0/dbhome_1' on the local node.

Oracle Universal Installer cleanup was successful.

Oracle deinstall tool successfully cleaned up temporary directories.

#######################################################################

############# ORACLE DEINSTALL TOOL END #############

[oracle@lxcora02 ~]$

Next the Grid Infrastructure is deconfigured, again refreshed my memory from Thinking Out Loud.

gstanden@vmem1:~$ ssh grid@lxcora02

grid@lxcora02's password:

Last login: Sun Sep 21 23:28:24 2014 from 10.0.3.1

[grid@lxcora02 ~]$ clear

[grid@lxcora02 ~]$ echo $GRID_HOME

/u00/app/grid/product/12.1.0/grid

[grid@lxcora02 ~]$ $GRID_HOME/deinstall/deinstall

Checking for required files and bootstrapping ...

Please wait ...

Location of logs /tmp/deinstall2014-10-24_11-21-17PM/logs/

############ ORACLE DECONFIG TOOL START ############

######################### DECONFIG CHECK OPERATION START #########################

## [START] Install check configuration ##

Checking for existence of the Oracle home location /u00/app/grid/product/12.1.0/grid

Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Standalone Server

Oracle Base selected for deinstall is: /u00/app/grid

Checking for existence of central inventory location /u00/app/oraInventory

Checking for existence of the Oracle Grid Infrastructure home /u00/app/grid/product/12.1.0/grid

## [END] Install check configuration ##

Traces log file: /tmp/deinstall2014-10-24_11-21-17PM/logs//crsdc_2014-10-24_11-21-36PM.log

Network Configuration check config START

Network de-configuration trace file location: /tmp/deinstall2014-10-24_11-21-17PM/logs/netdc_check2014-10-24_11-21-37-PM.log

Specify all Oracle Restart enabled listeners that are to be de-configured. Enter .(dot) to deselect all. [LISTENER]:

Network Configuration check config END

Asm Check Configuration START

ASM de-configuration trace file location: /tmp/deinstall2014-10-24_11-21-17PM/logs/asmcadc_check2014-10-24_11-21-43-PM.log

Automatic Storage Management (ASM) instance is detected in this Oracle home /u00/app/grid/product/12.1.0/grid.

ASM Diagnostic Destination : /u00/app/grid

ASM Diskgroups :

ASM diskstring : <Default>

Diskgroups will not be dropped

If you want to retain the existing diskgroups or if any of the information detected is incorrect, you can modify by entering 'y'. Do you want to modify above information (y|n) [n]:

Database Check Configuration START

Database de-configuration trace file location: /tmp/deinstall2014-10-24_11-21-17PM/logs/databasedc_check2014-10-24_11-22-03-PM.log

Database Check Configuration END

######################### DECONFIG CHECK OPERATION END #########################

####################### DECONFIG CHECK OPERATION SUMMARY #######################

Oracle Grid Infrastructure Home is: /u00/app/grid/product/12.1.0/grid

The following nodes are part of this cluster: null

The cluster node(s) on which the Oracle home deinstallation will be performed are:null

Oracle Home selected for deinstall is: /u00/app/grid/product/12.1.0/grid

Inventory Location where the Oracle home registered is: /u00/app/oraInventory

Following Oracle Restart enabled listener(s) will be de-configured: LISTENER

ASM instance will be de-configured from this Oracle home

Do you want to continue (y - yes, n - no)? [n]: y

A log of this session will be written to: '/tmp/deinstall2014-10-24_11-21-17PM/logs/deinstall_deconfig2014-10-24_11-21-36-PM.out'

Any error messages from this session will be written to: '/tmp/deinstall2014-10-24_11-21-17PM/logs/deinstall_deconfig2014-10-24_11-21-36-PM.err'

######################## DECONFIG CLEAN OPERATION START ########################

Database de-configuration trace file location: /tmp/deinstall2014-10-24_11-21-17PM/logs/databasedc_clean2014-10-24_11-22-19-PM.log

ASM de-configuration trace file location: /tmp/deinstall2014-10-24_11-21-17PM/logs/asmcadc_clean2014-10-24_11-22-19-PM.log

ASM Clean Configuration START

ASM Clean Configuration END

Network Configuration clean config START

Network de-configuration trace file location: /tmp/deinstall2014-10-24_11-21-17PM/logs/netdc_clean2014-10-24_11-22-22-PM.log

De-configuring Oracle Restart enabled listener(s): LISTENER

De-configuring listener: LISTENER

Stopping listener: LISTENER

Warning: Failed to stop listener. Listener may not be running.

Deleting listener: LISTENER

Listener deleted successfully.

Listener de-configured successfully.

De-configuring Listener configuration file...

Listener configuration file de-configured successfully.

De-configuring Naming Methods configuration file...

Naming Methods configuration file de-configured successfully.

De-configuring Local Net Service Names configuration file...

Local Net Service Names configuration file de-configured successfully.

De-configuring backup files...

Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

---------------------------------------->

Run the following command as the root user or the administrator on node "localhost".

/tmp/deinstall2014-10-24_11-21-17PM/perl/bin/perl -I/tmp/deinstall2014-10-24_11-21-17PM/perl/lib -I/tmp/deinstall2014-10-24_11-21-17PM/crs/install /tmp/deinstall2014-10-24_11-21-17PM/crs/install/roothas.pl -force -deconfig -paramfile "/tmp/deinstall2014-10-24_11-21-17PM/response/deinstall_OraGI12Home1.rsp"

Press Enter after you finish running the above commands

<----------------------------------------

######################### DECONFIG CLEAN OPERATION END #########################

####################### DECONFIG CLEAN OPERATION SUMMARY #######################

ASM instance was de-configured successfully from the Oracle home

Following Oracle Restart enabled listener(s) were de-configured successfully: LISTENER

Oracle Restart was already stopped and de-configured on node "localhost"

Oracle Restart is stopped and de-configured successfully.

#######################################################################

############# ORACLE DECONFIG TOOL END #############

Using properties file /tmp/deinstall2014-10-24_11-21-17PM/response/deinstall_2014-10-24_11-21-36-PM.rsp

Location of logs /tmp/deinstall2014-10-24_11-21-17PM/logs/

############ ORACLE DEINSTALL TOOL START ############

####################### DEINSTALL CHECK OPERATION SUMMARY #######################

A log of this session will be written to: '/tmp/deinstall2014-10-24_11-21-17PM/logs/deinstall_deconfig2014-10-24_11-21-36-PM.out'

Any error messages from this session will be written to: '/tmp/deinstall2014-10-24_11-21-17PM/logs/deinstall_deconfig2014-10-24_11-21-36-PM.err'

######################## DEINSTALL CLEAN OPERATION START ########################

## [START] Preparing for Deinstall ##

Setting LOCAL_NODE to localhost

Setting CRS_HOME to true

Setting oracle.installer.invPtrLoc to /tmp/deinstall2014-10-24_11-21-17PM/oraInst.loc

Setting oracle.installer.local to false

## [END] Preparing for Deinstall ##

Setting the force flag to false

Setting the force flag to cleanup the Oracle Base

Oracle Universal Installer clean START

Detach Oracle home '/u00/app/grid/product/12.1.0/grid' from the central inventory on the local node : Done

Delete directory '/u00/app/grid/product/12.1.0/grid' on the local node : Done

Delete directory '/u00/app/oraInventory' on the local node : Done

The Oracle Base directory '/u00/app/grid' will not be removed on local node. The directory is not empty.

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END

## [START] Oracle install clean ##

Clean install operation removing temporary directory '/tmp/deinstall2014-10-24_11-21-17PM' on node 'localhost'

## [END] Oracle install clean ##

######################### DEINSTALL CLEAN OPERATION END #########################

####################### DEINSTALL CLEAN OPERATION SUMMARY #######################

Successfully detached Oracle home '/u00/app/grid/product/12.1.0/grid' from the central inventory on the local node.

Successfully deleted directory '/u00/app/grid/product/12.1.0/grid' on the local node.

Successfully deleted directory '/u00/app/oraInventory' on the local node.

Oracle Universal Installer cleanup was successful.

Run 'rm -r /etc/oraInst.loc' as root on node(s) 'localhost' at the end of the session.

Run 'rm -r /opt/ORCLfmap' as root on node(s) 'localhost' at the end of the session.

Run 'rm -r /etc/oratab' as root on node(s) 'localhost' at the end of the session.

Oracle deinstall tool successfully cleaned up temporary directories.

#######################################################################

############# ORACLE DEINSTALL TOOL END #############

[grid@lxcora02 ~]$

And here are the results of running the root portion of the GI deinstall as shown below.

gstanden@vmem1:~$ ssh root@lxcora02

root@lxcora02's password:

Last login: Fri Oct 24 23:10:13 2014

[root@lxcora02 ~]# /tmp/deinstall2014-10-24_11-21-17PM/perl/bin/perl -I/tmp/deinstall2014-10-24_11-21-17PM/perl/lib -I/tmp/deinstall2014-10-24_11-21-17PM/crs/install /tmp/deinstall2014-10-24_11-21-17PM/crs/install/roothas.pl -force -deconfig -paramfile "/tmp/deinstall2014-10-24_11-21-17PM/response/deinstall_OraGI12Home1.rsp"

Using configuration parameter file: /tmp/deinstall2014-10-24_11-21-17PM/response/deinstall_OraGI12Home1.rsp

CRS-4639: Could not contact Oracle High Availability Services

CRS-4000: Command Stop failed, or completed with errors.

CRS-4639: Could not contact Oracle High Availability Services

CRS-4000: Command Delete failed, or completed with errors.

CLSU-00100: operating system function: opendir failed with error data: 2

CLSU-00101: operating system error message: No such file or directory

CLSU-00103: error location: scrsearch1

CLSU-00104: additional error information: cant open scr home dir scls_scr_getval

CRS-4639: Could not contact Oracle High Availability Services

CRS-4000: Command Stop failed, or completed with errors.

2014/10/24 23:24:13 CLSRSC-337: Successfully deconfigured Oracle Restart stack

[root@lxcora02 ~]#

Remove the entire $ORACLE_HOME and $GRID_HOME top level directory, u00 as shown below.

[root@lxcora02 /]# rm -Rf u00

[root@lxcora02 /]#

Configure and Test Jumbo Frames on Private Interconnect Interfaces

[root@lxcora02 ~]# ping -c 3 -M do -s 8972 10.207.39.11

PING 10.207.39.11 (10.207.39.11) 8972(9000) bytes of data.

ping: local error: Message too long, mtu=1500

ping: local error: Message too long, mtu=1500

ping: local error: Message too long, mtu=1500

--- 10.207.39.11 ping statistics ---

3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3012ms

[root@lxcora02 ~]# ping -c 3 -M do -s 8972 192.210.39.11

PING 192.210.39.11 (192.210.39.11) 8972(9000) bytes of data.

8980 bytes from 192.210.39.11: icmp_seq=1 ttl=64 time=0.189 ms

8980 bytes from 192.210.39.11: icmp_seq=2 ttl=64 time=0.062 ms

8980 bytes from 192.210.39.11: icmp_seq=3 ttl=64 time=0.047 ms

--- 192.210.39.11 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 1999ms

rtt min/avg/max/mdev = 0.047/0.099/0.189/0.064 ms

[root@lxcora02 ~]# ping -c 3 -M do -s 8972 192.211.39.11

PING 192.211.39.11 (192.211.39.11) 8972(9000) bytes of data.

8980 bytes from 192.211.39.11: icmp_seq=1 ttl=64 time=0.143 ms

8980 bytes from 192.211.39.11: icmp_seq=2 ttl=64 time=0.046 ms

8980 bytes from 192.211.39.11: icmp_seq=3 ttl=64 time=0.062 ms

--- 192.211.39.11 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 1999ms

rtt min/avg/max/mdev = 0.046/0.083/0.143/0.043 ms

[root@lxcora02 ~]# ping -c 3 -M do -s 8972 192.212.39.11

PING 192.212.39.11 (192.212.39.11) 8972(9000) bytes of data.

8980 bytes from 192.212.39.11: icmp_seq=1 ttl=64 time=0.156 ms

8980 bytes from 192.212.39.11: icmp_seq=2 ttl=64 time=0.096 ms

8980 bytes from 192.212.39.11: icmp_seq=3 ttl=64 time=0.068 ms

--- 192.212.39.11 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 1998ms

rtt min/avg/max/mdev = 0.068/0.106/0.156/0.038 ms

[root@lxcora02 ~]# ping -c 3 -M do -s 8972 192.213.39.11

PING 192.213.39.11 (192.213.39.11) 8972(9000) bytes of data.

8980 bytes from 192.213.39.11: icmp_seq=1 ttl=64 time=0.082 ms

8980 bytes from 192.213.39.11: icmp_seq=2 ttl=64 time=0.066 ms

8980 bytes from 192.213.39.11: icmp_seq=3 ttl=64 time=0.094 ms

--- 192.213.39.11 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 1998ms

rtt min/avg/max/mdev = 0.066/0.080/0.094/0.015 ms

[root@lxcora02 ~]#

Now test from the other node as shown below.

[root@lxcora03 ~]# ping -c 3 -M do -s 8972 10.207.39.10

PING 10.207.39.10 (10.207.39.10) 8972(9000) bytes of data.

ping: local error: Message too long, mtu=1500

ping: local error: Message too long, mtu=1500

ping: local error: Message too long, mtu=1500

--- 10.207.39.10 ping statistics ---

3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2094ms

[root@lxcora03 ~]# ping -c 3 -M do -s 8972 192.210.39.10

PING 192.210.39.10 (192.210.39.10) 8972(9000) bytes of data.

8980 bytes from 192.210.39.10: icmp_seq=1 ttl=64 time=0.360 ms

8980 bytes from 192.210.39.10: icmp_seq=2 ttl=64 time=0.072 ms

8980 bytes from 192.210.39.10: icmp_seq=3 ttl=64 time=0.065 ms

--- 192.210.39.10 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 1998ms

rtt min/avg/max/mdev = 0.065/0.165/0.360/0.138 ms

[root@lxcora03 ~]# ping -c 3 -M do -s 8972 192.211.39.10

PING 192.211.39.10 (192.211.39.10) 8972(9000) bytes of data.

8980 bytes from 192.211.39.10: icmp_seq=1 ttl=64 time=0.033 ms

8980 bytes from 192.211.39.10: icmp_seq=2 ttl=64 time=0.048 ms

8980 bytes from 192.211.39.10: icmp_seq=3 ttl=64 time=0.041 ms

--- 192.211.39.10 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 1998ms

rtt min/avg/max/mdev = 0.033/0.040/0.048/0.009 ms

[root@lxcora03 ~]# ping -c 3 -M do -s 8972 192.212.39.10

PING 192.212.39.10 (192.212.39.10) 8972(9000) bytes of data.

8980 bytes from 192.212.39.10: icmp_seq=1 ttl=64 time=0.136 ms

8980 bytes from 192.212.39.10: icmp_seq=2 ttl=64 time=0.112 ms

8980 bytes from 192.212.39.10: icmp_seq=3 ttl=64 time=0.040 ms

--- 192.212.39.10 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2000ms

rtt min/avg/max/mdev = 0.040/0.096/0.136/0.040 ms

[root@lxcora03 ~]# ping -c 3 -M do -s 8972 192.213.39.10

PING 192.213.39.10 (192.213.39.10) 8972(9000) bytes of data.

8980 bytes from 192.213.39.10: icmp_seq=1 ttl=64 time=0.037 ms

8980 bytes from 192.213.39.10: icmp_seq=2 ttl=64 time=0.061 ms

8980 bytes from 192.213.39.10: icmp_seq=3 ttl=64 time=0.045 ms

--- 192.213.39.10 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 1998ms

rtt min/avg/max/mdev = 0.037/0.047/0.061/0.012 ms

[root@lxcora03 ~]#

Install cvuqdisk rpm Package on Both RAC nodes

gstanden@vmem1:~$ ssh root@lxcora02

root@lxcora02's password:

Last login: Tue Oct 28 00:11:19 2014

[root@lxcora02 ~]# cd /home/grid

[root@lxcora02 grid]# cd grid

[root@lxcora02 grid]# cd rpm

[root@lxcora02 rpm]# rpm -Uvh cvuqdisk-1.0.9-1.rpm

Preparing... ########################################### [100%]

Using default group oinstall to install package

1:cvuqdisk ########################################### [100%]

[root@lxcora02 rpm]# ssh lxcora03

root@lxcora03's password:

Last login: Tue Oct 28 00:10:46 2014

[root@lxcora03 ~]# cd /home/grid/grid/rpm

[root@lxcora03 rpm]# rpm -Uvh cvuqdisk-1.0.9-1.rpm

Preparing... ########################################### [100%]

Using default group oinstall to install package

1:cvuqdisk ########################################### [100%]

[root@lxcora03 rpm]#

Establish No-password ssh between RAC nodes

The following steps are done to establish no-password ssh between nodes. The GUI installer can do this automatically in 12c, but still I prefer to do it ahead of time and verify it is set prior to install. The first step was that since the lxcora02 was cloned, it still had the public key in the .ssh directory for lxcora01 so need to delete key and create new for lxcora02 as shown below.

[grid@lxcora02 .ssh]$ ls -lrt

total 16

-rw-r--r-- 1 grid oinstall 395 Sep 18 22:06 id_rsa.pub

-rw------- 1 grid oinstall 1675 Sep 18 22:06 id_rsa

-rw-r--r-- 1 grid oinstall 395 Sep 18 22:06 authorized_keys

-rw-r--r-- 1 grid oinstall 1958 Oct 24 23:34 known_hosts

[grid@lxcora02 .ssh]$ cd ..

[grid@lxcora02 ~]$ rm .ssh/id_rsa*

[grid@lxcora02 ~]$

[grid@lxcora02 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/grid/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/grid/.ssh/id_rsa.

Your public key has been saved in /home/grid/.ssh/id_rsa.pub.

The key fingerprint is:

60:41:ce:f4:10:a0:1b:3e:5e:5e:77:dd:8d:b6:da:93 grid@lxcora02

The key's randomart image is:

+--[ RSA 2048]----+

| .o*. |

| . + + |

| o = . |

| . o . . . . o |

| + . . S . . + .|

| . + . . . . . |

| . . .. |

| oE |

| . .. |

+-----------------+

[grid@lxcora02 ~]$ cd .ssh

[grid@lxcora02 .ssh]$ ls -lrt

total 16

-rw-r--r-- 1 grid oinstall 395 Sep 18 22:06 authorized_keys

-rw-r--r-- 1 grid oinstall 1958 Oct 24 23:34 known_hosts

-rw-r--r-- 1 grid oinstall 395 Oct 24 23:37 id_rsa.pub

-rw------- 1 grid oinstall 1675 Oct 24 23:37 id_rsa

[grid@lxcora02 .ssh]$ cat authorized_keys id_rsa.pub > new

[grid@lxcora02 .ssh]$ more new

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAx3VNFs0b7Qr4rOjP3myGVY+F6dm1VpBxcUDm1tdcX/c77SZ/a9l1gkiAX0OcgpyWZII1gUJfm/qVi7U94lI+KTBCq3bzhxA78OnSHnw/eGpk/EuzAZ9+3T9RtJoqv44KBdLOUBU+Q7NxCQAhiaD21qNUp9CWYpHQdt1z4xG9u

fGsQPjK5+6mxFjsk2eJZwXKD2KhaKqgBsULkf2wJq6Hg5S+Gta3uKLz4mSrtx7zeblapL4VJWlX7mqLI77M+QEEt2qienDBtoLet1MEy8UGe4EN3aS8rU2MDyin+FrT2OAYv5DHKuYrm2bFb7loddAf+lANwVWaKzfOl2Yy7i8BSw== grid@lxcora01

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqIVZGpnGL5WjAUaQLPCK5sjBdXXoBs7kL827cMySCzR7KeOSTxk4OLDDhVK349B1hJn1eiyTDR7NQ7jd0bu2+IfEn3ICchIg1LMdsLMb3xj93hNOSbcNhjVeqVWqF4GVXKY2lcCLaJQ6dELuhfvdkuZ18uwjhebnnCt0ESf7b

cqwAm7AxQbrqHf2A/7W9iICGdqL2KtYXch5DVYITA2+wmb5Ww1DTd3KNs1aBOCC8N80DZRaOX4DTeZ7zh+LK7lleKB3aTTmiK5tT/3d/HD2BoCdbZbtOd2ro/QDlZLs0Zh9Av6xiGtyaufoeS0ycCIoZoEESqZrSfAu8vkyqMREMw== grid@lxcora02

[grid@lxcora02 .ssh]$ mv new authorized_keys

[grid@lxcora02 .ssh]$ cat authorized_keys

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAx3VNFs0b7Qr4rOjP3myGVY+F6dm1VpBxcUDm1tdcX/c77SZ/a9l1gkiAX0OcgpyWZII1gUJfm/qVi7U94lI+KTBCq3bzhxA78OnSHnw/eGpk/EuzAZ9+3T9RtJoqv44KBdLOUBU+Q7NxCQAhiaD21qNUp9CWYpHQdt1z4xG9ufGsQPjK5+6mxFjsk2eJZwXKD2KhaKqgBsULkf2wJq6Hg5S+Gta3uKLz4mSrtx7zeblapL4VJWlX7mqLI77M+QEEt2qienDBtoLet1MEy8UGe4EN3aS8rU2MDyin+FrT2OAYv5DHKuYrm2bFb7loddAf+lANwVWaKzfOl2Yy7i8BSw== grid@lxcora01

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqIVZGpnGL5WjAUaQLPCK5sjBdXXoBs7kL827cMySCzR7KeOSTxk4OLDDhVK349B1hJn1eiyTDR7NQ7jd0bu2+IfEn3ICchIg1LMdsLMb3xj93hNOSbcNhjVeqVWqF4GVXKY2lcCLaJQ6dELuhfvdkuZ18uwjhebnnCt0ESf7bcqwAm7AxQbrqHf2A/7W9iICGdqL2KtYXch5DVYITA2+wmb5Ww1DTd3KNs1aBOCC8N80DZRaOX4DTeZ7zh+LK7lleKB3aTTmiK5tT/3d/HD2BoCdbZbtOd2ro/QDlZLs0Zh9Av6xiGtyaufoeS0ycCIoZoEESqZrSfAu8vkyqMREMw== grid@lxcora02

[grid@lxcora02 .ssh]$

Next copy the authorized_keys file to lxcora01 as shown below.

[grid@lxcora02 .ssh]$ scp authorized_keys grid@lxcora01:/home/grid/.ssh/.

grid@lxcora01's password:

authorized_keys 100% 790 0.8KB/s 00:00

[grid@lxcora02 .ssh]$

Next test that no-password ssh works and login to the nodes as variations of the hostname. Run the test_login.sh script on both nodes until the only output is the date as shown below.

[grid@lxcora02 .ssh]$ chmod 775 test_login.sh

[grid@lxcora02 .ssh]$ ./test_login.sh

Fri Oct 24 23:44:43 EDT 2014

Fri Oct 24 23:44:43 EDT 2014

Fri Oct 24 23:44:43 EDT 2014

Fri Oct 24 23:45:07 EDT 2014

Fri Oct 24 23:45:21 EDT 2014

Fri Oct 24 23:45:35 EDT 2014

The authenticity of host 'lxcora01.vmem.org (10.207.39.85)' can't be established.

RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'lxcora01.vmem.org' (RSA) to the list of known hosts.

Fri Oct 24 23:46:13 EDT 2014

The authenticity of host 'lxcora02.vmem.org (10.207.39.87)' can't be established.

RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'lxcora02.vmem.org' (RSA) to the list of known hosts.

Fri Oct 24 23:46:34 EDT 2014

[grid@lxcora02 .ssh]$ ./test_login.sh

Fri Oct 24 23:46:49 EDT 2014

Fri Oct 24 23:46:50 EDT 2014

Fri Oct 24 23:46:50 EDT 2014

Fri Oct 24 23:47:14 EDT 2014

Fri Oct 24 23:47:28 EDT 2014

Fri Oct 24 23:47:42 EDT 2014

Fri Oct 24 23:48:01 EDT 2014

Fri Oct 24 23:48:20 EDT 2014

[grid@lxcora02 .ssh]$ cat test_login.sh

ssh grid@localhost date

ssh grid@127.0.0.1 date

ssh grid@lxcora02 date

ssh grid@lxcora01 date

ssh grid@10.207.39.87 date

ssh grid@10.207.39.85 date

ssh grid@lxcora01.vmem.org date

ssh grid@lxcora02.vmem.org date

[grid@lxcora02 .ssh]$

Now scp the script to the other node and run until output is only date and time on the other node (lxcora01) as shown below.

[grid@lxcora02 .ssh]$ scp test_login.sh grid@lxcora01:/home/grid/.

test_login.sh 100% 212 0.2KB/s 00:00

[grid@lxcora02 .ssh]$

Then login to lxcora01 as grid and run test_login.sh script until output is only date and time as shown below.

[grid@lxcora01 ~]$ ./test_login.sh

Fri Oct 24 23:52:18 EDT 2014

Fri Oct 24 23:52:18 EDT 2014

The authenticity of host 'lxcora02 (10.207.39.87)' can't be established.

RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'lxcora02,10.207.39.87' (RSA) to the list of known hosts.

Fri Oct 24 23:52:45 EDT 2014

Fri Oct 24 23:52:45 EDT 2014

Fri Oct 24 23:52:59 EDT 2014

The authenticity of host '10.207.39.85 (10.207.39.85)' can't be established.

RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '10.207.39.85' (RSA) to the list of known hosts.

Fri Oct 24 23:53:14 EDT 2014

The authenticity of host 'lxcora01.vmem.org (10.207.39.85)' can't be established.

RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'lxcora01.vmem.org' (RSA) to the list of known hosts.

Fri Oct 24 23:53:35 EDT 2014

The authenticity of host 'lxcora02.vmem.org (10.207.39.87)' can't be established.

RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'lxcora02.vmem.org' (RSA) to the list of known hosts.

Fri Oct 24 23:53:55 EDT 2014

[grid@lxcora01 ~]$ ./test_login.sh

Fri Oct 24 23:54:00 EDT 2014

Fri Oct 24 23:54:00 EDT 2014

Fri Oct 24 23:54:24 EDT 2014

Fri Oct 24 23:54:24 EDT 2014

Fri Oct 24 23:54:38 EDT 2014

Fri Oct 24 23:54:52 EDT 2014

Fri Oct 24 23:55:12 EDT 2014

Fri Oct 24 23:55:31 EDT 2014

[grid@lxcora01 ~]$

Now it is necessary to setup the final NFS mount configurations and also setup for Normal Redundancy (mirrored). So change in plans. After looking at the Oracle 12c ASM guide find that NFS is no longer supported for OCR and VOTE as it was in 11gR2. So, will use lxc-device command to add the required non-multipath disk onto the lxc RAC nodes. Actually, this was already done. These are LUNS delivered from a KVM guest (oracle652.vmem.org) running SCST Linux SAN (used because it has a switch for 4K / 512 logical setting and 4K / 512 physical setting of sector sizes, so that LUNs can be delivered for any format combination. The Ubuntu 14.04.1 laptop OS is the iSCSI client. The LUNs are then added to the lxc nodes using the lxc-device add command which is persistend across lxc reboots. It would be preferable to use some sort of UDEV rule in the lxc nodes to ensure that the lun is persistent across reboots with respect to actual identity.

At the moment I need to resize the iSCSI LUN so I use the method at my blog documented here. For completeness of this blog, the steps are shown below.

[root@oracle651 scst_lxcora01]# dd if=/dev/zero of=/scst_lxcora01/AsmTst_02.img bs=512 count=0 seek=10M

0+0 records in

0+0 records out

0 bytes (0 B) copied, 0.000411897 s, 0.0 kB/s

[root@oracle651 scst_lxcora01]# ls -lrt AsmTst_02.img

total 377860

-rw-r--r-- 1 root root 5368709120 Oct 25 00:01 AsmTst_02.img

[root@oracle651 scst_lxcora01]# scstadmin -resync_dev AsmTst_02

Collecting current configuration: done.

-> Making requested changes.

-> Setting device attribute 'resync_size' to value '1' for device 'AsmTst_02': done.

-> Done.

All done.

[root@oracle651 scst_lxcora01]# scstadmin -write_config /etc/scst.conf

Collecting current configuration: done.

Writing current configuration to file '/etc/scst.conf'..

All done.

[root@oracle651 scst_lxcora01]# scstadmin -list_group

Collecting current configuration: done.

Driver: iscsi

Target: iqn.2014-08.org.vmem:oracle651.san.asm.luns

Driver/target 'iscsi/iqn.2014-08.org.vmem:oracle651.san.asm.luns' has no associated LUNs.

Group: oracle631

Assigned LUNs:

LUN Device

-----------------

1 AsmDat631_01

2 AsmDat631_02

3 AsmDat631_03

4 AsmDat631_04

5 AsmDat631_05

6 AsmDat631_06

7 AsmDat631_07

8 AsmDat631_08

Assigned Initiators:

Initiator

----------------------------------

iqn.2014-08.org.vmem:oracle631.asm

iqn.2014-09.org.vmem:lxcora3.asm

Group: lxcora01

Assigned LUNs:

LUN Device

--------------

1 AsmTst_01

2 AsmTst_02

3 AsmTst_03

Assigned Initiators:

Initiator

-------------------------------------

iqn.2014-09.org.vmem1:oracle.asm.luns

All done.

[root@oracle651 scst_lxcora01]# exit

Next on the Ubuntu laptop host, use iscsiadm to update with the new size as shown below.

gstanden@vmem1:~$ sudo iscsiadm -m node -R

[sudo] password for gstanden:

Rescanning session [sid: 1, target: iqn.2014-08.org.vmem:oracle651.san.asm.luns, portal: 10.207.40.74,3260]

Rescanning session [sid: 2, target: iqn.2014-08.org.vmem:oracle651.san.asm.luns, portal: 10.207.41.74,3260]

gstanden@vmem1:~$

Check that the updated size has been detected as shown below.

Here are some notes from the install of RAC on the LXC containers. This is the root.sh output from lxcora02 (first node) as shown below. The root.sh execution successful should look something like as shown below.

[root@lxcora02 ~]# /u00/app/grid/product/12.1.0/grid/root.sh

Performing root user operation.

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u00/app/grid/product/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.

The contents of "oraenv" have not changed. No need to overwrite.

The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u00/app/grid/product/12.1.0/grid/crs/install/crsconfig_params

2014/10/25 18:15:01 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

TFA-00012: Oracle Trace File Analyzer (TFA) requires BASH shell. Please install bash and try again.

2014/10/25 18:15:10 CLSRSC-4004: Failed to install Oracle Trace File Analyzer (TFA) Collector. Grid Infrastructure operations will continue.

2014/10/25 18:15:11 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful

root wallet

root wallet cert

root cert export

peer wallet

profile reader wallet

pa wallet

peer wallet keys

pa wallet keys

peer cert request

pa cert request

peer cert

pa cert

peer root cert TP

profile reader root cert TP

pa root cert TP

peer pa cert TP

pa peer cert TP

profile reader pa cert TP

profile reader peer cert TP

peer user cert

pa user cert

2014/10/25 18:15:42 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

CRS-2672: Attempting to start 'ora.evmd' on 'lxcora02'

CRS-2672: Attempting to start 'ora.mdnsd' on 'lxcora02'

CRS-2676: Start of 'ora.mdnsd' on 'lxcora02' succeeded

CRS-2676: Start of 'ora.evmd' on 'lxcora02' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'lxcora02'

CRS-2676: Start of 'ora.gpnpd' on 'lxcora02' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lxcora02'

CRS-2672: Attempting to start 'ora.gipcd' on 'lxcora02'

CRS-2676: Start of 'ora.cssdmonitor' on 'lxcora02' succeeded

CRS-2676: Start of 'ora.gipcd' on 'lxcora02' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'lxcora02'

CRS-2672: Attempting to start 'ora.diskmon' on 'lxcora02'

CRS-2676: Start of 'ora.diskmon' on 'lxcora02' succeeded

CRS-2676: Start of 'ora.cssd' on 'lxcora02' succeeded

ASM created and started successfully.

Disk Group DATA created successfully.

CRS-2672: Attempting to start 'ora.crf' on 'lxcora02'

CRS-2672: Attempting to start 'ora.storage' on 'lxcora02'

CRS-2676: Start of 'ora.storage' on 'lxcora02' succeeded

CRS-2676: Start of 'ora.crf' on 'lxcora02' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'lxcora02'

CRS-2676: Start of 'ora.crsd' on 'lxcora02' succeeded

CRS-4256: Updating the profile

Successful addition of voting disk b5c728b6fd2c4f5dbf311ac955d78133.

Successful addition of voting disk 2a797f2ddb824feabfd38214d88f30b7.

Successful addition of voting disk 21cdfa0bb8c04f26bf07f6901e789466.

Successfully replaced voting disk group with +DATA.

CRS-4256: Updating the profile

CRS-4266: Voting file(s) successfully replaced

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE b5c728b6fd2c4f5dbf311ac955d78133 (/dev/mapper/asm_disk26) [DATA]

2. ONLINE 2a797f2ddb824feabfd38214d88f30b7 (/dev/mapper/asm_disk27) [DATA]

3. ONLINE 21cdfa0bb8c04f26bf07f6901e789466 (/dev/mapper/asm_disk28) [DATA]

Located 3 voting disk(s).

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'lxcora02'

CRS-2673: Attempting to stop 'ora.crsd' on 'lxcora02'

CRS-2677: Stop of 'ora.crsd' on 'lxcora02' succeeded

CRS-2673: Attempting to stop 'ora.storage' on 'lxcora02'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'lxcora02'

CRS-2673: Attempting to stop 'ora.gpnpd' on 'lxcora02'

CRS-2677: Stop of 'ora.storage' on 'lxcora02' succeeded

CRS-2673: Attempting to stop 'ora.crf' on 'lxcora02'

CRS-2673: Attempting to stop 'ora.ctssd' on 'lxcora02'

CRS-2673: Attempting to stop 'ora.evmd' on 'lxcora02'

CRS-2673: Attempting to stop 'ora.asm' on 'lxcora02'

CRS-2677: Stop of 'ora.mdnsd' on 'lxcora02' succeeded

CRS-2677: Stop of 'ora.gpnpd' on 'lxcora02' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'lxcora02' succeeded

CRS-2677: Stop of 'ora.crf' on 'lxcora02' succeeded

CRS-2677: Stop of 'ora.evmd' on 'lxcora02' succeeded

CRS-2677: Stop of 'ora.asm' on 'lxcora02' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'lxcora02'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'lxcora02' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'lxcora02'

CRS-2677: Stop of 'ora.cssd' on 'lxcora02' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'lxcora02'

CRS-2677: Stop of 'ora.gipcd' on 'lxcora02' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'lxcora02' has completed

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Starting Oracle High Availability Services-managed resources

CRS-2672: Attempting to start 'ora.mdnsd' on 'lxcora02'

CRS-2672: Attempting to start 'ora.evmd' on 'lxcora02'

CRS-2676: Start of 'ora.mdnsd' on 'lxcora02' succeeded

CRS-2676: Start of 'ora.evmd' on 'lxcora02' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'lxcora02'

CRS-2676: Start of 'ora.gpnpd' on 'lxcora02' succeeded

CRS-2672: Attempting to start 'ora.gipcd' on 'lxcora02'

CRS-2676: Start of 'ora.gipcd' on 'lxcora02' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lxcora02'

CRS-2676: Start of 'ora.cssdmonitor' on 'lxcora02' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'lxcora02'

CRS-2672: Attempting to start 'ora.diskmon' on 'lxcora02'

CRS-2676: Start of 'ora.diskmon' on 'lxcora02' succeeded

CRS-2676: Start of 'ora.cssd' on 'lxcora02' succeeded

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'lxcora02'

CRS-2672: Attempting to start 'ora.ctssd' on 'lxcora02'

CRS-2676: Start of 'ora.ctssd' on 'lxcora02' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'lxcora02' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'lxcora02'

CRS-2676: Start of 'ora.asm' on 'lxcora02' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'lxcora02'

CRS-2676: Start of 'ora.storage' on 'lxcora02' succeeded

CRS-2672: Attempting to start 'ora.crf' on 'lxcora02'

CRS-2676: Start of 'ora.crf' on 'lxcora02' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'lxcora02'

CRS-2676: Start of 'ora.crsd' on 'lxcora02' succeeded

CRS-6023: Starting Oracle Cluster Ready Services-managed resources

CRS-6017: Processing resource auto-start for servers: lxcora02

CRS-6016: Resource auto-start has completed for server lxcora02

CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources

CRS-4123: Oracle High Availability Services has been started.

2014/10/25 18:31:31 CLSRSC-343: Successfully started Oracle Clusterware stack

CRS-2672: Attempting to start 'ora.asm' on 'lxcora02'

CRS-2676: Start of 'ora.asm' on 'lxcora02' succeeded

CRS-2672: Attempting to start 'ora.DATA.dg' on 'lxcora02'

CRS-2676: Start of 'ora.DATA.dg' on 'lxcora02' succeeded

2014/10/25 18:33:02 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

[root@lxcora02 ~]#

Results of root.sh on the second node (lxcora03) are shown below.

[root@lxcora03 ~]# /u00/app/grid/product/12.1.0/grid/root.sh

Performing root user operation.

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u00/app/grid/product/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.

The contents of "oraenv" have not changed. No need to overwrite.

The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u00/app/grid/product/12.1.0/grid/crs/install/crsconfig_params

2014/10/25 18:35:38 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

TFA-00012: Oracle Trace File Analyzer (TFA) requires BASH shell. Please install bash and try again.

2014/10/25 18:35:47 CLSRSC-4004: Failed to install Oracle Trace File Analyzer (TFA) Collector. Grid Infrastructure operations will continue.

2014/10/25 18:35:48 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful

2014/10/25 18:37:15 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Starting Oracle High Availability Services-managed resources

CRS-2672: Attempting to start 'ora.mdnsd' on 'lxcora03'

CRS-2672: Attempting to start 'ora.evmd' on 'lxcora03'

CRS-2676: Start of 'ora.mdnsd' on 'lxcora03' succeeded

CRS-2676: Start of 'ora.evmd' on 'lxcora03' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'lxcora03'

CRS-2676: Start of 'ora.gpnpd' on 'lxcora03' succeeded

CRS-2672: Attempting to start 'ora.gipcd' on 'lxcora03'

CRS-2676: Start of 'ora.gipcd' on 'lxcora03' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lxcora03'

CRS-2676: Start of 'ora.cssdmonitor' on 'lxcora03' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'lxcora03'

CRS-2672: Attempting to start 'ora.diskmon' on 'lxcora03'

CRS-2676: Start of 'ora.diskmon' on 'lxcora03' succeeded

CRS-2676: Start of 'ora.cssd' on 'lxcora03' succeeded

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'lxcora03'

CRS-2672: Attempting to start 'ora.ctssd' on 'lxcora03'

CRS-2676: Start of 'ora.ctssd' on 'lxcora03' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'lxcora03' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'lxcora03'

CRS-2676: Start of 'ora.asm' on 'lxcora03' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'lxcora03'

CRS-2676: Start of 'ora.storage' on 'lxcora03' succeeded

CRS-2672: Attempting to start 'ora.crf' on 'lxcora03'

CRS-2676: Start of 'ora.crf' on 'lxcora03' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'lxcora03'

CRS-2676: Start of 'ora.crsd' on 'lxcora03' succeeded

CRS-6017: Processing resource auto-start for servers: lxcora03

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'lxcora02'

CRS-2672: Attempting to start 'ora.net1.network' on 'lxcora03'

CRS-2676: Start of 'ora.net1.network' on 'lxcora03' succeeded

CRS-2672: Attempting to start 'ora.ons' on 'lxcora03'

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'lxcora02' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'lxcora02'

CRS-2677: Stop of 'ora.scan1.vip' on 'lxcora02' succeeded

CRS-2672: Attempting to start 'ora.scan1.vip' on 'lxcora03'

CRS-2676: Start of 'ora.ons' on 'lxcora03' succeeded

CRS-2676: Start of 'ora.scan1.vip' on 'lxcora03' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'lxcora03'

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'lxcora03' succeeded

CRS-6016: Resource auto-start has completed for server lxcora03

CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources

CRS-4123: Oracle High Availability Services has been started.

2014/10/25 18:41:14 CLSRSC-343: Successfully started Oracle Clusterware stack

2014/10/25 18:41:26 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

[root@lxcora03 ~]#

Ok so I'm going to paste in all the screenshots that I have of the GUI grid install hopefully today, but I wanted to upload the shots of the running system (crsctl output) of the 12c 2-node RAC GI running perfectly on Ubuntu 14.04.1 inside of Oracle Enteprise Linux 6.5 conntainers running on the Ubuntu so that I could share this result even while I'm still finishing up the documentation at this blog. I'm stoked! Finally! Oracle RAC on Ubuntu without endless tweaks!! All of the taste, none of the calories!! The background is a webpage at the QNX website.

And here is the GNS post-install validation step from cluvy.

Install RAC Database on LXC Containers GI

Setup ssh user equivalence as shown below.

gstanden@vmem1:~$ ssh oracle@lxcora02

oracle@lxcora02's password:

Last login: Sun Oct 26 22:17:26 2014 from vmem1.vmem.org

[oracle@lxcora02 ~]$ cd .ssh

[oracle@lxcora02 .ssh]$ ls

[oracle@lxcora02 .ssh]$ cd ..

[oracle@lxcora02 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/oracle/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/oracle/.ssh/id_rsa.

Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.

The key fingerprint is:

4a:15:3d:26:86:ce:7a:47:71:0e:3e:0f:b8:42:8c:ec oracle@lxcora02

The key's randomart image is:

+--[ RSA 2048]----+

| ... |

| . =.= |

| . o o +.B . |

| o o +.= . |

| . . ..oS+ |

| E o.o.. . |

| o.. |

| |

| |

+-----------------+

[oracle@lxcora02 ~]$ cd .ssh

[oracle@lxcora02 .ssh]$ cat id_rsa.pub > authorized_keys

[oracle@lxcora02 .ssh]$ scp authorized_keys oracle@lxcora03

[oracle@lxcora02 .ssh]$ scp authorized_keys oracle@lxcora03:/home/oracle/.

The authenticity of host 'lxcora03 (10.207.39.88)' can't be established.

RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'lxcora03,10.207.39.88' (RSA) to the list of known hosts.

oracle@lxcora03's password:

authorized_keys 100% 397 0.4KB/s 00:00

[oracle@lxcora02 .ssh]$ ssh lxcora03

oracle@lxcora03's password:

Last login: Fri Oct 24 23:14:34 2014 from vmem1.vmem.org

[oracle@lxcora03 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/oracle/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/oracle/.ssh/id_rsa.

Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.

The key fingerprint is:

3f:47:5f:6a:2a:c0:c2:e2:b2:96:99:a6:62:22:08:4b oracle@lxcora03

The key's randomart image is:

+--[ RSA 2048]----+

| |

| |

| |

| |

| . .S . .|

|.E . o o. . . o |

|+. = . . .o . + |

|*.B . .o o |

|*+.o .. |

+-----------------+

[oracle@lxcora03 ~]$ mv authorized_keys ./.ssh/.

[oracle@lxcora03 ~]$ cd .ssh

[oracle@lxcora03 .ssh]$ cat id_rsa.pub >> authorized_keys

[oracle@lxcora03 .ssh]$ scp authorized_keys oracle@lxcora02:/home/oracle/.ssh/.

The authenticity of host 'lxcora02 (10.207.39.87)' can't be established.

RSA key fingerprint is 4f:62:a6:23:e4:58:2e:e2:1b:a4:1c:39:28:a0:ff:df.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'lxcora02,10.207.39.87' (RSA) to the list of known hosts.

oracle@lxcora02's password:

authorized_keys 100% 794 0.8KB/s 00:00

[oracle@lxcora03 .ssh]$ ssh lxcora02

Last login: Sun Oct 26 22:22:15 2014 from vmem1.vmem.org

[oracle@lxcora02 ~]$ ssh lxcora03

Last login: Sun Oct 26 22:24:22 2014 from lxcora02.vmem.org

[oracle@lxcora03 ~]$ exit

logout

Connection to lxcora03 closed.

[oracle@lxcora02 ~]$ exit

logout

Connection to lxcora02 closed.

[oracle@lxcora03 .ssh]$

Create and run the test_login.sh script until output is as shown below to establish user equivalency for linux user "oracle" on both lxcora02 and lxcora03 nodes.

[oracle@lxcora03 .ssh]$ ./test_login.sh

Sun Oct 26 22:36:09 EDT 2014

Sun Oct 26 22:36:09 EDT 2014

Sun Oct 26 22:36:09 EDT 2014

Sun Oct 26 22:36:09 EDT 2014

Sun Oct 26 22:36:09 EDT 2014

Sun Oct 26 22:36:09 EDT 2014

Sun Oct 26 22:36:09 EDT 2014

Sun Oct 26 22:36:09 EDT 2014

[oracle@lxcora03 .ssh]$ ssh oracle@lxcora02

Last login: Sun Oct 26 22:33:04 2014 from vmem1.vmem.org

[oracle@lxcora02 ~]$ cd .ssh

[oracle@lxcora02 .ssh]$ ./test_login.sh

Sun Oct 26 22:36:24 EDT 2014

Sun Oct 26 22:36:24 EDT 2014

Sun Oct 26 22:36:24 EDT 2014

Sun Oct 26 22:36:24 EDT 2014

Sun Oct 26 22:36:24 EDT 2014

Sun Oct 26 22:36:25 EDT 2014

Sun Oct 26 22:36:25 EDT 2014

Sun Oct 26 22:36:25 EDT 2014

[oracle@lxcora02 .ssh]$ cat test_login.sh

ssh oracle@localhost date

ssh oracle@127.0.0.1 date

ssh oracle@lxcora03 date

ssh oracle@lxcora02 date

ssh oracle@10.207.39.88 date

ssh oracle@10.207.39.87 date

ssh oracle@lxcora02.vmem.org date

ssh oracle@lxcora03.vmem.org date

[oracle@lxcora02 .ssh]$