Oracle RAC 6-node 12c GNS ASM Flex Cluster Ubuntu 15.04 Install

GENERAL NOTES

What is this page?  It is a guide that provides a set of scripts so that you can prepare an Ubuntu 15.04 64-bit desktop to run Oracle Enterprise software products (such as Oracle Enterprise Edition 12c RAC GNS ASM Flex Cluster)  using LXC Linux Containers on an OpenvSwitch network. 

Why would you want to build this?  To have an elastic and dense deployment of Oracle Enterprise Edtion 12c RAC nodes on Ubuntu 15.04 64-bit desktop edition that runs at bare metal speed with NO hypervisor.  This project is very useful for RAC training schools.  The scripts can be rerun to destroy and rebuild the cluster in the cycle typical of an Oracle RAC training program, all on Ubuntu Linux, the premier free linux desktop OS, ideal for students.

Note that currently running Oracle Enteprise Edition software in LXC containers ontop of Ubuntu Linux is not supported by Oracle, so you would have to run this configuration and support it on your own or simply just use it for educational and training purposes.  It should not be used for production in cases where you need fully-supported deployment.

The setup uses the following main components as shown below
  • bind9
  • isc-dhcp-server
  • openvswitch
  • lxc
[ Note: that this LXC containerized approach to Oracle Enteprise Software product deployment on Linux(es) can theoretically be used on any Linux distribution, such as trivially RedHat, but theoretically also Gentoo and really any distribution that supports LXC Linux Containers. Of course the details could be wildly different as to how to duplicate this Ubuntu work, because on Gentoo for example, you'd have to rewrite all the package commands as "emerge" commands. ]

I was able to create a 12-node LXC-container RAC on my Lenovo W520 Mobile Workstation:
  • Intel® Core™ i7-2720QM CPU @ 2.20GHz × 8
  • 32Gb of RAM
  • ST1000LM014-1EJ164 (SM28) (1Tb Hybrid Disk with SSD Cache by Seagate)

The load averages begin to increase non-linearly after about 8-node RAC, and at 12-node RAC the load averages as measured by top are close to 100.  Thus, the 6-lxc container RAC is about the upper limit for a high-end desktop or laptop.  Some specs from the running 12-node RAC are shown below.   Note that one interesting feature of LXC containerized RAC is that each LXC node can access ALL memory available on the Ubuntu host.  The 12-node containerized RAC is at the very edge of the upper-limit for this workstation.

  • Memory Usage:  15 Gb
Note:  September 18, 2015:  The software for this project is now maintained at github.  The bundles attached at this webpage should not be used as the primary source for the software.  Go to https://github.com/gstanden/orabuntu-lxc to download the latest zip bundle of the software.  This page can be used as a supplementary guide for configuring the software that you download from github.

NOTE:  The install bundle of files are now a github project at "orabuntu-lxc" at github.

Gilbert Standen
Yonkers, NY
September 13, 2015

INSTALL INSTRUCTIONS START HERE


DO A FRESH INSTALL OF UBUNTU 15.04 64-bit DESKTOP SOFTWARE.   ONLY RUN THE SCRIPT BUNDLE ON A FRESH INSTALL.

NOTE:  When you install the Ubuntu 15.04 be sure to take the "set time automatically over the internet" options which is to say be sure NTP is configured and running on your Ubuntu 15.04 install. This is needed because this guide uses the Ubuntu host server as a local NTP source for the LXC containers.

Do not do an install on a customized and been-running-for-awhile Ubuntu 15.04 - such install is at your own risk!!  These scripts are not yet sufficiently bulletproofed for that type of install and could overwrite your existing customizations on an Ubuntu 15.04 that has been running for some time and has customizations.

UPDATE YOUR FRESH INSTALL OF UBUNTU 15.04 64-bit DESKTOP SOFTWARE BEFORE DOING THIS INSTALL. 

IF YOUR UBUNTU 15.04 IS UP-TO-DATE YOU WILL GET THE FOLLOWING DIALOG WHEN YOU RUN THE UBUNTU SOFTWARE UPDATER UTILITY, AS SHOWN BELOW.


Get this software from github now, do not use the early development bundles at the bottom of this blog.

https://github.com/gstanden/orabuntu-lxc

Don't use the bundles at the end of this blog anymore, the software has moved to github.  This google sites page is just the guide now.

Note also that the IP address assignments are designed to have the IP 4th triplet match the container name, i.e. "lxcora10" has a public IP address of "10.207.39.10", and lxcora11 has a public IP address of "10.207.39.11", and so on. and all the ASM and private networks are the same, so that the IP can be easily remembered for the container to which the IP is assigned.

Oracle Support Document # 1918426.1
12.1.0.2 root script fails to start ora.ctssd if nodes name length are not the same

The root.sh script will error out on any nodes that have a different number of characters in the length of the node name compared to the first node with the following error as shown below.  The latest version at github ensures that all lxc container nodes have names of the form { lxcora00, lxcora01, ..., lxcora99 }.  Since Oracle 12c allows a maximum of 64 RAC nodes, this fix should allow for any number of nodes up to the maximum the software allows.

CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2015/09/13 13:10:16 CLSRSC-117: Failed to start Oracle Clusterware stack

Died at /u00/app/grid/product/12.1.0/grid/crs/install/crsinstall.pm line 914.
The command '/u00/app/grid/product/12.1.0/grid/perl/bin/perl -I/u00/app/grid/product/12.1.0/grid/perl/lib -I/u00/app/grid/product/12.1.0/grid/crs/install /u00/app/grid/product/12.1.0/grid/crs/install/rootcrs.pl ' execution failed


Now open a terminal session on the Ubuntu desktop and go the the ~/Downloads subdirectory of your user account.  For example, on my desktop this directory will be:

/home/oracle/Downloads

because I installed this Ubuntu 15.04 using "oracle" as the default username. 

Now gunzip and untar the bundle as shown below.

oracle@W521:~$ cd Downloads

oracle@W521:~/Downloads$ ls -lrt
total 52
-rw-rw-r-- 1 oracle oracle 51614 Sep  3 21:54 ubuntu-lxc-oracle.v9.tar.gz

oracle@W521:~/Downloads$ gunzip ubuntu-lxc-oracle.v9.tar.gz

oracle@W521:~/Downloads$ tar -xvf ubuntu-lxc-oracle.v9.tar

ubuntu-host.lst
ubuntu-host.tar
lxc-config.lst
lxc-config.tar
lxc-lxcora01.lst
lxc-lxcora01.tar
lxc-lxcora0x.lst
lxc-lxcora0x.tar
scst-files.tar
ubuntu-services-1.sh
ubuntu-services-2a.sh
ubuntu-services-2b.sh
ubuntu-services-3a.sh
ubuntu-services-3b.sh
ubuntu-services-3c.sh
ubuntu-services-3d.sh
ubuntu-host-backup.sh
rc.local
dhclient.conf
lxc-services.sh
install_grid.sh
edit_bashrc
crt_links_v2.sh
ubuntu-lxc-oracle.lst.copy
README

oracle@W521:~/Downloads$

Run the scripts in this order as shown below.  Some scripts will reboot the host when finished.

oracle@W521:~/Downloads$ ls ubuntu-services* | sort

ubuntu-services-1.sh
ubuntu-services-2a.sh
ubuntu-services-2b.sh
ubuntu-services-3a.sh
ubuntu-services-3b.sh
ubuntu-services-3c.sh
ubuntu-services-3d.sh


oracle@W521:~/Downloads$


If you have not previously run any sudo commands, the ubuntu-services-1.sh script will stop to prompt for your user password so that sudo commands can be run as shown below.  You probably won't need to enter your sudo password again, this will be the only time.

oracle@W521:~/Downloads$ ./ubuntu-services-1.sh

============================================
Verify network up....
============================================

PING google.com (74.125.196.113) 56(84) bytes of data.
64 bytes from yk-in-f113.1e100.net (74.125.196.113): icmp_seq=1 ttl=41 time=31.8 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 31.882/31.882/31.882/0.000 ms

[sudo] password for oracle:

In this example, the default user account created at install of the OS was "oracle".   Yours might be /home/jsmith/Downloads if your name was Jim Smith and you selected "jsmith" as your username when you installed the Ubuntu base OS on your laptop or desktop.

The ubuntu-services-3c.sh script script needs to have a parameter passed in (the number of RAC nodes you want).  I have a Lenovo W520 top-of-the-line mobile workstation with a quad-core processor and 32 Gb of RAM, so I choose "6" for "six RAC node LXC containers".  If you give no parameter at all, it defaults to "2" so if you have an "ordinary" desktop with 4Gb of RAM or maybe 8Gb of RAM, just pass in "2" as the parameter or take the default which is also "2".

================================================
Now run ubuntu-services-3c.sh X                
Note that ubuntu-services-3c.sh takes an input 
variable X which is the number of LXC RAC nodes
you wish to create.  If X is not entered, the  
build defaults to a 2-node RAC cluster.  If X is
set to 6 it will create a 6-node RAC           
================================================

oracle@W521:~/Downloads$ ./ubuntu-services-3c.sh 6

When you get this message below, in most cases you will answer "N" because you should be installing on a fresh copy of Ubuntu 15.04 64-bit desktop and there should be no previously-created containers.  However, this code exists in the script because I'm working on dealing with the case that containers of the same name already exist and in general working towards making this script bundle safe for a customized been-running-for-awhile deployment of Ubuntu 15.04 64-bit desktop.  Usually, just answer "N" to this, especially if as shown below, no containers are found matching the naming criteria output shows empty set {    } as shown below. 

!!! WARNING !!!
===========================================
Destruction of cloned containers ( Y / N )
===========================================

ClonedContainersExist =

Existing containers in the set {  } have been found.  <-- the "{  }" means no containers were found, so just answer "N".

These containers match the names of containers that are about to be created.
Please answer Y to destroy the existing containers or N to keep them
!!! WARNING:  ANSWERING Y WILL DESTROY EXISTING CONTAINERS !!!
Destroy existing containers?  [ Y | N ]:
N

Final output shows all Oracle-ready lxcora0x containers are running and ready for Oracle RAC Grid Infrastructure install as shown below.

================================================
Starting LXC clone containers for Oracle       
================================================

starting a container...
lxcora01
next command will be:   sudo lxc-start -n lxcora01
sleeping 20 seconds...
starting a container...
lxcora02
next command will be:   sudo lxc-start -n lxcora02
sleeping 20 seconds...
starting a container...
lxcora03
next command will be:   sudo lxc-start -n lxcora03
sleeping 20 seconds...
starting a container...
lxcora04
next command will be:   sudo lxc-start -n lxcora04
sleeping 20 seconds...
starting a container...
lxcora05
next command will be:   sudo lxc-start -n lxcora05
sleeping 20 seconds...
starting a container...
lxcora06
next command will be:   sudo lxc-start -n lxcora06
sleeping 20 seconds...
================================================
Waiting for final container initialization...  
================================================

================================================
LXC containers for Oracle started.             
================================================


NAME      STATE    IPV4                                                                                                    IPV6  GROUPS  AUTOSTART 
--------------------------------------------------------------------------------------------------------------------------------------------------
lxcora00  STOPPED  -                                                                                                       -     -       NO        
lxcora01  RUNNING  10.207.39.10, 172.220.40.10, 172.221.40.10, 192.210.39.10, 192.211.39.10, 192.212.39.10, 192.213.39.10  -     -       NO        
lxcora02  RUNNING  10.207.39.11, 172.220.40.11, 172.221.40.11, 192.210.39.11, 192.211.39.11, 192.212.39.11, 192.213.39.11  -     -       NO        
lxcora03  RUNNING  10.207.39.12, 172.220.40.12, 172.221.40.12, 192.210.39.12, 192.211.39.12, 192.212.39.12, 192.213.39.12  -     -       NO        
lxcora04  RUNNING  10.207.39.13, 172.220.40.13, 172.221.40.13, 192.210.39.13, 192.211.39.13, 192.212.39.13, 192.213.39.13  -     -       NO        
lxcora05  RUNNING  10.207.39.14, 172.220.40.14, 172.221.40.14, 192.210.39.14, 192.211.39.14, 192.212.39.14, 192.213.39.14  -     -       NO        
lxcora06  RUNNING  10.207.39.15, 172.220.40.15, 172.221.40.15, 192.210.39.15, 192.211.39.15, 192.212.39.15, 192.213.39.15  -     -       NO        
oracle@W521:~/Downloads$

SCST SAN SETUP

[ A side note about running SCST (which is covered next):  This guide uses SCST, which requires a custom kernel, because SCST is one of the few Linux SAN open source offerings that support 4K logical / 4K physical LUN presentation.  This blog grew out of some ideas I had while working at Violin Memory to create a desktop "workshop" where customers could experiment with 4K native format storage with Oracle software products, so a Linux SAN that fully supports 4K as well as the traditional 512-byte format was needed.  Also, SCST is behind-the-scenes a component of VMOS5 Violin Memory arrays, and also some generations of Kaminario all-flash arrays.

However, if you will not need 4K support for your Linux SAN, you could just as easily use Linux TGT which does not require the kernel build steps, and can just be configured and used without a custom SCST kernel build. So, if you prefer, you can build out your Linux SAN storage for this containerized RAC using TGT although I'm not providing those instructions because they are more trivial and guides for TGT setup are readily available on the net for example here.  Note that if you go the TGT route, you won't be able to experiment with Oracle and 4K native storage interactions, which are interesting and never dull to say the least. ]

Next untar the scst-files.tar file and cd to scst-files to create the SCST Linux SAN which will provide the storage LUNs for the Oracle 12c Grid Infrastructure install as shown below.  Run the scripts in the order shown below.


oracle@W521:~/Downloads$ ls -lrt scst*

-rw-rw-r-- 1 oracle oracle 40960 Sep  2 21:57 scst-files.tar

oracle@W521:~/Downloads$ tar -xf scst-files.tar

oracle@W521:~/Downloads$ cd scst-files

oracle@W521:~/Downloads/scst-files$ ls create-scst* | sort

create-scst-1a.sh    <-- Run the scripts in this order !
create-scst-1b.sh
create-scst-1c.sh
create-scst-1d.sh
create-scst-2a.sh
create-scst-2b.sh
create-scst-3.sh
create-scst-4a.sh
create-scst-4b.sh
create-scst-5a.sh
create-scst-5b.sh
create-scst.sh

oracle@W521:~/Downloads/scst-files$

When you run the create-scst-1d.sh script, you will get these prompts of the form "(NEW)" and just hit enter for each one (accepts default value) as shown below.  See bold section at end of following blockcode quote where the first prompt for "(NEW)" appears. 

oracle@W521:~/Downloads/scst-files$ ./create-scst-1d.sh

for i in debian.master/d-i/kernel-versions.in debian.master/control.stub.in; do    \
  new=`echo $i | sed 's/\.in$//'`;                    \
  cat $i | sed -e 's/PKGVER/3.19.0/g'                              \
        -e 's/ABINUM/26/g'                                       \
    -e 's/SRCPKGNAME/linux/g'                             \
    -e 's/=HUMAN=/64 bit x86/g'                                    \
  > $new;                                \
done
flavours="debian.master/control.d/vars.generic debian.master/control.d/vars.generic-lpae debian.master/control.d/vars.lowlatency debian.master/control.d/vars.powerpc-e500mc debian.master/control.d/vars.powerpc-smp debian.master/control.d/vars.powerpc64-emb debian.master/control.d/vars.powerpc64-smp debian.master/control.d/vars.scst";\
for i in $flavours; do                            \
  /bin/bash -e debian/scripts/control-create $i |                \
    sed -e 's/PKGVER/3.19.0/g'                                  \
    -e 's/ABINUM/26/g'                                       \
    -e 's/SRCPKGNAME/linux/g'                             \
    -e 's/=HUMAN=/64 bit x86/g'                                    \
    >> debian.master/control.stub;                        \
done
cp debian.master/control.stub debian.master/control
rm -rf /home/oracle/Downloads/linux-3.19.0/debian/build/modules /home/oracle/Downloads/linux-3.19.0/debian/build/firmware            \
    /home/oracle/Downloads/linux-3.19.0/debian/build/kernel-versions /home/oracle/Downloads/linux-3.19.0/debian/build/package-list    \
    /home/oracle/Downloads/linux-3.19.0/debian/build/debian.master
mkdir -p /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64/
cp debian.master/d-i/modules/* /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64/
mkdir -p /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64/
cp debian.master/d-i/firmware/* /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64/
cp debian.master/d-i/package-list debian.master/d-i/kernel-versions /home/oracle/Downloads/linux-3.19.0/debian/build/
touch /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64/kernel-image
# kernel-wedge needs to poke around in debian.master/
ln -nsf /home/oracle/Downloads/linux-3.19.0/debian /home/oracle/Downloads/linux-3.19.0/debian/build/debian
# Some files may need to differ between architectures
if [ -d debian.master/d-i/modules-amd64 ]; then            \
    cp debian.master/d-i/modules-amd64/* \
    /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64/;                \
fi
if [ -d debian.master/d-i/firmware-amd64 ]; then            \
    cp debian.master/d-i/firmware-amd64/* \
    /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64/;                \
fi
# Remove unwanted stuff for this architecture
if [ -r "debian.master/d-i/exclude-modules.amd64" ]; then        \
    (cat debian.master/d-i/exclude-modules.amd64;            \
     ls /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64/) | sort | uniq -d |    \
    (cd /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64/; xargs rm -f);        \
fi
if [ -r "debian.master/d-i/exclude-firmware.amd64" ]; then        \
    (cat debian.master/d-i/exclude-firmware.amd64;            \
     ls /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64/) | sort | uniq -d |    \
    (cd /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64/; xargs rm -f);    \
fi
# Per flavour module lists
flavour_modules=`ls debian.master/d-i/modules.amd64-* 2>/dev/null`    \
    || true;                        \
if [ "$flavour_modules" != "" ]; then                \
    for flav in $flavour_modules; do                \
    name=`echo $flav | sed 's/.*\/modules.amd64-//'`;    \
    mkdir /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name;        \
    (cd /home/oracle/Downloads/linux-3.19.0/debian/build/modules/; tar cf - `cat ../$flav`) |    \
        (cd /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name/; tar xf -);    \
    touch /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name/kernel-image;    \
    done;                            \
fi
flavour_firmware=`ls debian.master/d-i/firmware.amd64-* 2>/dev/null`    \
    || true;                        \
if [ "$flavour_firmware" != "" ]; then                \
    for flav in $flavour_firmware; do                \
    name=`echo $flav | sed 's/.*\/firmware.amd64-//'`;    \
    mkdir /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name;        \
    (cd /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/; tar cf - `cat ../$flav`) |    \
        (cd /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name/; tar xf -);\
    touch /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name/kernel-image;    \
    done;                            \
fi
# Some files may need to differ between flavours
flavour_module_dirs=`ls -d debian.master/d-i/modules-amd64-* 2>/dev/null`\
    || true;                        \
if [ "$flavour_module_dirs" ]; then                \
    for flav in $flavour_module_dirs; do            \
    name=`echo $flav | sed 's/.*\/modules-amd64-//'`;    \
    [ -d /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name ] ||        \
        cp -a /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64            \
        modules/amd64-$name;                \
    cp $flav/* /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name/;    \
    done;                            \
fi
flavour_firmware_dirs=`ls -d debian.master/d-i/firmware-amd64-* 2>/dev/null`\
    || true;                        \
if [ "$flavour_firmware_dirs" ]; then                \
    for flav in $flavour_firmware_dirs; do            \
    name=`echo $flav | sed 's/.*\/firmware-amd64-//'`;    \
    [ -d /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name ] ||        \
        cp -a /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64            \
        firmware/amd64-$name;            \
    cp $flav/* /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name/;    \
    done;                            \
fi
# Remove unwanted stuff for each flavour
flavour_exclude=`ls debian.master/d-i/exclude-modules.amd64-* 2>/dev/null`\
    || true;                        \
if [ "$flavour_exclude" ]; then                \
    for flav in $flavour_exclude; do                \
    name=`echo $flav | sed 's/.*\/exclude-modules.amd64-//'`;\
    [ -d /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name ] ||        \
        cp -a /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64             \
        /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name;        \
    (cat $flav;                        \
     ls /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name) |        \
     sort |    uniq -d |                    \
        (cd /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64-$name/;        \
         xargs rm -f);                    \
    done;                            \
fi
flavour_exclude=`ls debian.master/d-i/exclude-firmware.amd64-* 2>/dev/null`\
    || true;                        \
if [ "$flavour_exclude" ]; then                \
    for flav in $flavour_exclude; do                \
    name=`echo $flav | sed 's/.*\/exclude-firmware.amd64-//'`;\
    [ -d /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name ] ||        \
        cp -a /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64             \
        /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name;        \
    (cat $flav;                        \
     ls /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name) |        \
     sort |    uniq -d |                    \
        (cd /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64-$name/;        \
         xargs rm -f);                    \
    done;                            \
fi
if [ ! -d /home/oracle/Downloads/linux-3.19.0/debian/build/modules/x86_64 ]; then        \
    mkdir -p /home/oracle/Downloads/linux-3.19.0/debian/build/modules/x86_64;        \
    cp /home/oracle/Downloads/linux-3.19.0/debian/build/modules/amd64/*            \
        /home/oracle/Downloads/linux-3.19.0/debian/build/modules/x86_64;        \
fi
if [ ! -d /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/x86_64 ]; then        \
    mkdir -p /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/x86_64;        \
    cp /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/amd64/*            \
        /home/oracle/Downloads/linux-3.19.0/debian/build/firmware/x86_64;        \
fi
cp debian.master/control.stub debian/control.stub
cd /home/oracle/Downloads/linux-3.19.0/debian/build && LANG=C kernel-wedge gen-control > /home/oracle/Downloads/linux-3.19.0/debian/control
Use of uninitialized value $builddep in string ne at /usr/share/kernel-wedge/commands/gen-control line 43, <KVERS> line 6.
Use of uninitialized value $builddep in split at /usr/share/kernel-wedge/commands/gen-control line 44, <KVERS> line 6.
Use of uninitialized value $builddep in string ne at /usr/share/kernel-wedge/commands/gen-control line 43, <KVERS> line 8.
Use of uninitialized value $builddep in split at /usr/share/kernel-wedge/commands/gen-control line 44, <KVERS> line 8.
Use of uninitialized value $builddep in string ne at /usr/share/kernel-wedge/commands/gen-control line 43, <KVERS> line 10.
Use of uninitialized value $builddep in split at /usr/share/kernel-wedge/commands/gen-control line 44, <KVERS> line 10.
Use of uninitialized value $builddep in string ne at /usr/share/kernel-wedge/commands/gen-control line 43, <KVERS> line 12.
Use of uninitialized value $builddep in split at /usr/share/kernel-wedge/commands/gen-control line 44, <KVERS> line 12.
dh_testdir
dh_testroot
dh_clean
# d-i stuff
rm -rf debian.master/d-i-amd64
# Generated on the fly.
rm -f debian.master/d-i/firmware/kernel-image
# normal build junk
rm -rf debian.master/abi/3.19.0-26.28
rm -rf /home/oracle/Downloads/linux-3.19.0/debian/build
rm -f /home/oracle/Downloads/linux-3.19.0/debian/stamps/stamp-*
rm -rf debian.master/linux-*
# This gets rid of the d-i packages in control
cp -f debian.master/control.stub debian.master/control
cp debian.master/changelog debian/changelog
# Install the copyright information.
cp debian.master/copyright debian/copyright
# If we have a reconstruct script use it.
[ -f debian.master/reconstruct ] && bash -x debian.master/reconstruct
+ '[' '!' -L ubuntu/vbox/vboxguest/include ']'
+ ln -sf ../include ubuntu/vbox/vboxguest/include
+ '[' '!' -L ubuntu/vbox/vboxguest/r0drv ']'
+ ln -sf ../r0drv ubuntu/vbox/vboxguest/r0drv
+ '[' '!' -L ubuntu/vbox/vboxsf/include ']'
+ ln -sf ../include ubuntu/vbox/vboxsf/include
+ '[' '!' -L ubuntu/vbox/vboxsf/r0drv ']'
+ ln -sf ../r0drv ubuntu/vbox/vboxsf/r0drv
+ '[' '!' -L ubuntu/vbox/vboxvideo/include ']'
+ ln -sf ../include ubuntu/vbox/vboxvideo/include
+ exit 0
dh_testdir;
/bin/bash -e debian/scripts/misc/kernelconfig updateconfigs
* Run silentoldconfig (yes=0) on amd64/config.flavour.generic ...
make[1]: Entering directory '/home/oracle/Downloads/linux-3.19.0'
make[2]: Entering directory '/home/oracle/Downloads/linux-3.19.0/build'
  HOSTCC  scripts/basic/fixdep
  GEN     ./Makefile
  HOSTCC  scripts/kconfig/conf.o
  SHIPPED scripts/kconfig/zconf.tab.c
  SHIPPED scripts/kconfig/zconf.lex.c
  SHIPPED scripts/kconfig/zconf.hash.c
  HOSTCC  scripts/kconfig/zconf.tab.o
  HOSTLD  scripts/kconfig/conf
scripts/kconfig/conf --silentoldconfig Kconfig
.config:3896:warning: override: M686 changes choice state
.config:7457:warning: override: TREE_RCU changes choice state
.config:8637:warning: override: TICK_CPU_ACCOUNTING changes choice state
.config:8640:warning: override: USB_DWC2_HOST changes choice state
*
* Restart config...
*
*
* Networking options
*
Packet socket (PACKET) [Y/n/m/?] y
  Packet: sockets monitoring interface (PACKET_DIAG) [M/n/y/?] m
Unix domain sockets (UNIX) [Y/n/m/?] y
  UNIX: socket monitoring interface (UNIX_DIAG) [M/n/y/?] m
Transformation user configuration interface (XFRM_USER) [M/n/y/?] m
Transformation sub policy support (XFRM_SUB_POLICY) [N/y/?] n
Transformation migrate database (XFRM_MIGRATE) [N/y/?] n
Transformation statistics (XFRM_STATISTICS) [Y/n/?] y
PF_KEY sockets (NET_KEY) [M/n/y/?] m
  PF_KEY MIGRATE (NET_KEY_MIGRATE) [N/y/?] n
TCP/IP networking (INET) [Y/n/?] y
  TCP/IP zero-copy transfer completion notification (TCP_ZERO_COPY_TRANSFER_COMPLETION_NOTIFICATION) [N/y/?] (NEW)

After answering many of these "(NEW)" prompts and accepting default value "(NEW)" you will eventually get the following prompt about "config.flavour.scst" as shown below.

Notice that all prompts should be answered with "n" in this phase EXCEPT the line for "amd64/config.flavour.scst? [Y/n] Y" which as you can see should be answered with "Y" as shown below (bolded line).  Be sure that ONLY this config is answered with "Y" .  All other config line prompts should be answered with "n" as shown below.

check-config: 43/43 checks passed -- exit 0
rm -rf build
dh_testdir;
/bin/bash -e debian/scripts/misc/kernelconfig editconfigs
Do you want to edit config: amd64/config.flavour.generic? [Y/n] n
Do you want to edit config: amd64/config.flavour.lowlatency? [Y/n] n

Do you want to edit config: amd64/config.flavour.scst? [Y/n] Y


Answer "n" to the rest of the "config" prompts.  There will be an error message about some builds as shown below, but these are expected and can be ignored because we only care about the build on amd64 architecture, which is successful. 

*** ERROR: 12 config-check failures detected  <-- This error is ok, ignore

When you choose the "Y" for config.flavour.scst you will be presented with a linux GUI screen that will popup automatically.  This utility is so that you can configure the new SCST options.  Screenshots are shown below to guide you through choosing the required SCST options for the kernel.

This is the screen you first get as shown below.



Use the "down arrow" key on your keyboard to move down to "Networking support --->" as shown below.


Use the down arrow key to move down to "Networking options --->" as shown below.


Use the down arrow key to move down to "TCP/IP zero-copy transfer completion notification" and press the spacebar on the keyboard to put a " * " symbol in the brackets, i.e. [ * ] as shown below (selects the feature).


Now use the right-arrow key on the keyboard to move the lower highlighted value from "<Select>" to "<Exit>", and then hit <Enter> to exit this screen, as shown below.


Repeat the same sequence of steps to "<Exit>" the next screen too, as shown below.


Now use the down-arrow key on the keyboard to select "Device Drivers --->" as shown below and hit <Enter> to select.


Now use the down-arrow key on the keyboard to move down to "SCSI target (SCST) support --->" and hit <Enter> to select option.


The "< > SCSI target (SCST) support" option is displayed.  Press the spacebar to enable the large number of options associated with this and to put a " *M" in the < M> to select all options as shown in the next screenshot following.


Once the "<M>" is in the option, all the additional options appear.  No need to make anymore changes, this is it for this screen.  Now use the right arrow key as before to exit from this screen back up to the main screen as shown in the next screen following.


Next screenshot shows using right-arrow key to exit back up to the previous screen, as shown below.


Again, use the right-arrow key to select "<Exit>" to exit up to the next higher screen as shown below.


Again, use right-arrow screen to select "<Exit>" to move back up to the next higher screen as shown below.


This is the final screen.  Just accept the default "<Yes>" to accept all the change you have made as shown below.


Now the kernel build will continue on and will run for a relatively long time such as 1 hour or more.  It can take quite a long time for the kernel to build.  You can go to sleep if it's late and check it in the morning, or go have a few beers and some tacos and come back in 90 minutes or so to see if the kernel build has finished.

When the kernel build finishes, these are the last few lines where you get back to the prompt as shown below.

flock -w 60 /home/oracle/Downloads/linux-3.19.0/debian/.LOCK dh_gencontrol -plinux-cloud-tools-3.19.0-26-scst
dh_md5sums -plinux-cloud-tools-3.19.0-26-scst
dh_builddeb -plinux-cloud-tools-3.19.0-26-scst
dpkg-deb: building package `linux-cloud-tools-3.19.0-26-scst' in `../linux-cloud-tools-3.19.0-26-scst_3.19.0-26.28_amd64.deb'.
oracle@W521:~/Downloads/scst-files$

Once the kernel build has finished as shown above, you need to continue with the next script to be run as shown below.  The first few output lines of the next script create-scst-2a.sh is shown below.

oracle@W521:~/Downloads/scst-files$ ./create-scst-2a.sh
-rw-r--r-- 1 root root  9499358 Sep  3 23:09 linux-headers-3.19.0-26_3.19.0-26.28_all.deb
-rw-r--r-- 1 root root   873666 Sep  4 00:03 linux-headers-3.19.0-26-scst_3.19.0-26.28_amd64.deb
-rw-r--r-- 1 root root 55869032 Sep  4 00:03 linux-image-3.19.0-26-scst_3.19.0-26.28_amd64.deb
[sudo] password for oracle:

Next run create-scst-2b.sh script. This builds scst modules.

Now it's time to run create-scst-3.sh script.

Now run create-scst-4a.sh.  During create-scst-4a.sh you will get the prompt shown below, and you should answer "y" to this prompt.

Performing this action may result in lost or corrupt data, are you sure you wish to continue (y/[n]) ? y

When the create-scst-4a.sh script has finished, it will wait for 30 seconds, and will present the following output (this is the expected output which shows your SCST SAN has successfully been configured and is ready with 3 LUNs for Oracle Grid Infrastructure install.

======================================================
Verify that SCST SAN is fully configured and ready   
Sleeping for 30 seconds...                           
======================================================


Collecting current configuration: done.

Driver: iscsi
Target: iqn.2015-08.org.vmem:w520.san.asm.luns

Driver/target 'iscsi/iqn.2015-08.org.vmem:w520.san.asm.luns' has no associated LUNs.

Group: lxc1

Assigned LUNs:

    LUN  Device        
    --------------------
    0    asm_systemdg_00
    1    asm_systemdg_01
    2    asm_systemdg_02

Assigned Initiators:

    Initiator
    -------------------------------------
    iqn.2014-09.org.vmem1:oracle.asm.luns



All done.

Next run create-scst-4b.sh script.  The expected output is shown below.  Notice that the LUNs are discovered on many networks, but that we only login to SCST LUNs on the two OpenvSwitch networks, sw2 and sw3, that are dedicated as LUN storage networks.  Notice also that the last step does an "ls -l /dev/mapper" and shows that we now have 3 multipath LUNs that are intended for the SYSTEMDG Oracle ASM diskgroup of the 12c ASM Flex Cluster Grid Infrastructure install.

oracle@W521:~/Downloads/scst-files$ ./create-scst-4b.sh

10.207.40.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.168.1.7:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.213.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.0.3.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.207.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.210.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
172.221.40.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.207.41.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
172.220.40.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.207.29.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.211.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.212.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.168.122.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.207.41.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.168.1.7:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.213.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.0.3.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.207.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.210.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.207.40.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
172.221.40.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
172.220.40.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
10.207.29.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.211.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.212.39.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns
192.168.122.1:3260,1 iqn.2015-08.org.vmem:w520.san.asm.luns

Logging in to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.41.1,3260] (multiple)
Login to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.41.1,3260] successful.
Logging in to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.40.1,3260] (multiple)
Login to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.40.1,3260] successful.

total 0
crw------- 1 root root  10, 236 Sep  4 00:12 control
brw-rw---- 1 root disk 252,   2 Sep  4 00:24 mpath2
brw-rw---- 1 root disk 252,   0 Sep  4 00:24 mpath1
brw-rw---- 1 root disk 252,   1 Sep  4 00:24 mpath0


oracle@W521:~/Downloads/scst-files$
,
Now run create-scst-5a.sh script.  This script if it works as designed should auto-create the /etc/multipath.conf file you will need to configure your multipath LUNs in /dev/mapper with friendly names and with other settings designed for the Oracle deployment.  Some details of what create-scst-5a.sh does is shown below.  The multipath.conf file that is autocreated for my Lenovo W520 mobile workstation is shown below.  You will likely be on different laptop or desktop and different disk and your multipath.conf will be different in the "blacklist" and "devices" stanzas, but the script has been designed to configure the multipath.conf for your system.  If you have issues, please email me at "gilstanden@hotmail.com" so that I can improve the script to work for all types of combos of machine and disks.

oracle@W521:~/Downloads/scst-files$ ./create-scst-5a.sh

oracle@W521:~/Downloads/scst-files$ more multipath.conf

blacklist {
    devnode      "sd[a]$"
    wwid         "350004cf20fba5cfd"
    device {
        vendor   "ATA"
        product  "ST2000LM003HN-M"
        revision "0001"
    }
}
defaults {
    user_friendly_names  yes
}
devices {
    device {
    vendor               "SCST_FIO"
    product              "asm*"
    revision             "310"
    path_grouping_policy group_by_serial
    getuid_callout       "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
    hardware_handler     "0"
    features             "1 queue_if_no_path"
    fast_io_fail_tmo     5
    dev_loss_tmo         30
    failback             immediate
    rr_weight            uniform
    no_path_retry        fail
    path_checker         tur
    rr_min_io            4
    path_selector        "round-robin 0"
    }
}
multipaths {
     multipath {
         wwid 26537623737636234
         alias asm_systemdg_00
     }
     multipath {
         wwid 26634313565346333
         alias asm_systemdg_01
     }
     multipath {
         wwid 26330663234633561
         alias asm_systemdg_02
     }
}
oracle@W521:~/Downloads/scst-files$

Now run create-scst-5b.sh script.  See the output of this script below for details.  Note that it has assigned the user-friendly names to the mpathX lunx in /dev/mapper.  If the multipath.conf was created correctly for your system, you will also see the LUN "friendly names" in /dev/mapper also.  Notice highlighed in bold the friendly names and the true multiapath (2 paths) that have been achieved for this SCST SAN for Oracle.

oracle@W521:~/Downloads/scst-files$ ./create-scst-5b.sh
Logging out of session [sid: 1, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.41.1,3260]
Logout of [sid: 1, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.41.1,3260] successful.
Logging out of session [sid: 2, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.40.1,3260]
Logout of [sid: 2, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.40.1,3260] successful.

===========================================================================
Login to SCST target...                                                   
Verify that login to SCST target is successful...                         
===========================================================================

Logging in to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.41.1,3260] (multiple)
Login to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.41.1,3260] successful.
Logging in to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.40.1,3260] (multiple)
Login to [iface: default, target: iqn.2015-08.org.vmem:w520.san.asm.luns, portal: 10.207.40.1,3260] successful.

===========================================================================
Verify /dev/mapper output...should be using multipath friendly names...   
Sleeping 5 seconds...                                                    
===========================================================================
 
total 0
crw------- 1 root root  10, 236 Sep  4 00:12 control
brw-rw---- 1 root disk 252,   1 Sep  4 00:32 asm_systemdg_02
brw-rw---- 1 root disk 252,   0 Sep  4 00:32 asm_systemdg_01
brw-rw---- 1 root disk 252,   2 Sep  4 00:32 asm_systemdg_00

===========================================================================
Verify multipath -ll -v2 output ...                                       
Sleeping 5 seconds...                                                    
===========================================================================

asm_systemdg_02 (26330663234633561) dm-1 SCST_FIO,asm_systemdg_02
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 9:0:0:2 sdg 8:96 active ready running
  `- 8:0:0:2 sdd 8:48 active ready running

asm_systemdg_01 (26634313565346333) dm-0 SCST_FIO,asm_systemdg_01
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 9:0:0:1 sdf 8:80 active ready running
  `- 8:0:0:1 sdc 8:32 active ready running

asm_systemdg_00 (26537623737636234) dm-2 SCST_FIO,asm_systemdg_00
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 9:0:0:0 sde 8:64 active ready running
  `- 8:0:0:0 sdb 8:16 active ready running

oracle@W521:~/Downloads/scst-files$

This concludes the series of "create-scst-*.sh" scripts.  If all has run as intended, the SCST SAN is up and running and ready.  You can run some checks on both the LXC Containers and the SCST SAN as shown below.


oracle@W521:~/Downloads/scst-files$ sudo lxc-ls -f
NAME      STATE    IPV4  IPV6  GROUPS  AUTOSTART 
------------------------------------------------
lxcora00  STOPPED  -     -     -       NO        
lxcora01  STOPPED  -     -     -       NO        
lxcora02  STOPPED  -     -     -       NO        
lxcora03  STOPPED  -     -     -       NO        
lxcora04  STOPPED  -     -     -       NO        
lxcora05  STOPPED  -     -     -       NO        
lxcora06  STOPPED  -     -     -       NO        
oracle@W521:~/Downloads/scst-files$ sudo scstadmin -list_group

Collecting current configuration: done.

Driver: iscsi
Target: iqn.2015-08.org.vmem:w520.san.asm.luns

Driver/target 'iscsi/iqn.2015-08.org.vmem:w520.san.asm.luns' has no associated LUNs.

Group: lxc1

Assigned LUNs:

    LUN  Device        
    --------------------
    0    asm_systemdg_00
    1    asm_systemdg_01
    2    asm_systemdg_02

Assigned Initiators:

    Initiator
    -------------------------------------
    iqn.2014-09.org.vmem1:oracle.asm.luns



All done.
oracle@W521:~/Downloads/scst-files$

Notice that when the Ubuntu host was last rebooted it booted into the "-scst" kernel that we built as shown below.

oracle@W521:~/Downloads/scst-files$ uname -a
Linux W521 3.19.0-26-scst #28 SMP Thu Sep 3 23:08:42 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
oracle@W521:~/Downloads/scst-files$

At this point, you might as well kick off the download of the Oracle 12c Grid Infrastructure software from http://www.oracle.com/downloads and then also while it is downloading you can start up the LXC RAC node containers as shown below. 

Notice that it's important to be sure that on all containers, that the "10.207.39.x" public interface comes up and is the first interface listed.  It sometimes takes a minute or so for the public interface IP to show up on the output of "sudo lxc-ls -f" so give it at least 30 seconds or so to show up.
Make sure that all containers have got their public interfaces up before starting any Oracle software installation by using "sudo lxc-ls -f" to check.


oracle@W521:~/Downloads/scst-files$ sudo lxc-start -n lxcora01
oracle@W521:~/Downloads/scst-files$ sudo lxc-ls -f
NAME      STATE    IPV4                                                                                      IPV6  GROUPS  AUTOSTART 
------------------------------------------------------------------------------------------------------------------------------------
lxcora00  STOPPED  -                                                                                         -     -       NO        
lxcora01  RUNNING  172.220.40.10, 172.221.40.10, 192.210.39.10, 192.211.39.10, 192.212.39.10, 192.213.39.10  -     -       NO        
lxcora02  STOPPED  -                                                                                         -     -       NO        
lxcora03  STOPPED  -                                                                                         -     -       NO        
lxcora04  STOPPED  -                                                                                         -     -       NO        
lxcora05  STOPPED  -                                                                                         -     -       NO        
lxcora06  STOPPED  -                                                                                         -     -       NO        
oracle@W521:~/Downloads/scst-files$ sudo lxc-ls -f
NAME      STATE    IPV4                                                                                                    IPV6  GROUPS  AUTOSTART 
--------------------------------------------------------------------------------------------------------------------------------------------------
lxcora00  STOPPED  -                                                                                                       -     -       NO        
lxcora01  RUNNING  10.207.39.10, 172.220.40.10, 172.221.40.10, 192.210.39.10, 192.211.39.10, 192.212.39.10, 192.213.39.10  -     -       NO        
lxcora02  STOPPED  -                                                                                                       -     -       NO        
lxcora03  STOPPED  -                                                                                                       -     -       NO        
lxcora04  STOPPED  -                                                                                                       -     -       NO        
lxcora05  STOPPED  -                                                                                                       -     -       NO        
lxcora06  STOPPED  -                                                                                                       -     -       NO        
oracle@W521:~/Downloads/scst-files$ sudo lxc-start -n lxcora02
oracle@W521:~/Downloads/scst-files$ sudo lxc-start -n lxcora03
oracle@W521:~/Downloads/scst-files$ sudo lxc-start -n lxcora04
oracle@W521:~/Downloads/scst-files$ sudo lxc-start -n lxcora05
oracle@W521:~/Downloads/scst-files$ sudo lxc-start -n lxcora06
oracle@W521:~/Downloads/scst-files$ sudo lxc-ls -f
NAME      STATE    IPV4                                                                                                    IPV6  GROUPS  AUTOSTART 
--------------------------------------------------------------------------------------------------------------------------------------------------
lxcora00  STOPPED  -                                                                                                       -     -       NO        
lxcora01  RUNNING  10.207.39.10, 172.220.40.10, 172.221.40.10, 192.210.39.10, 192.211.39.10, 192.212.39.10, 192.213.39.10  -     -       NO        
lxcora02  RUNNING  10.207.39.11, 172.220.40.11, 172.221.40.11, 192.210.39.11, 192.211.39.11, 192.212.39.11, 192.213.39.11  -     -       NO        
lxcora03  RUNNING  10.207.39.12, 172.220.40.12, 172.221.40.12, 192.210.39.12, 192.211.39.12, 192.212.39.12, 192.213.39.12  -     -       NO        
lxcora04  RUNNING  10.207.39.13, 172.220.40.13, 172.221.40.13, 192.210.39.13, 192.211.39.13, 192.212.39.13, 192.213.39.13  -     -       NO        
lxcora05  RUNNING  10.207.39.14, 172.220.40.14, 172.221.40.14, 192.210.39.14, 192.211.39.14, 192.212.39.14, 192.213.39.14  -     -       NO        
lxcora06  RUNNING  172.220.40.15, 172.221.40.15, 192.210.39.15, 192.211.39.15, 192.212.39.15, 192.213.39.15                -     -       NO        
oracle@W521:~/Downloads/scst-files$ sudo lxc-ls -f
NAME      STATE    IPV4                                                                                                    IPV6  GROUPS  AUTOSTART 
--------------------------------------------------------------------------------------------------------------------------------------------------
lxcora00  STOPPED  -                                                                                                       -     -       NO        
lxcora01  RUNNING  10.207.39.10, 172.220.40.10, 172.221.40.10, 192.210.39.10, 192.211.39.10, 192.212.39.10, 192.213.39.10  -     -       NO        
lxcora02  RUNNING  10.207.39.11, 172.220.40.11, 172.221.40.11, 192.210.39.11, 192.211.39.11, 192.212.39.11, 192.213.39.11  -     -       NO        
lxcora03  RUNNING  10.207.39.12, 172.220.40.12, 172.221.40.12, 192.210.39.12, 192.211.39.12, 192.212.39.12, 192.213.39.12  -     -       NO        
lxcora04  RUNNING  10.207.39.13, 172.220.40.13, 172.221.40.13, 192.210.39.13, 192.211.39.13, 192.212.39.13, 192.213.39.13  -     -       NO        
lxcora05  RUNNING  10.207.39.14, 172.220.40.14, 172.221.40.14, 192.210.39.14, 192.211.39.14, 192.212.39.14, 192.213.39.14  -     -       NO        
lxcora06  RUNNING  172.220.40.15, 172.221.40.15, 192.210.39.15, 192.211.39.15, 192.212.39.15, 192.213.39.15                -     -       NO        
oracle@W521:~/Downloads/scst-files$ sudo lxc-ls -f
NAME      STATE    IPV4                                                                                                    IPV6  GROUPS  AUTOSTART 
--------------------------------------------------------------------------------------------------------------------------------------------------
lxcora00  STOPPED  -                                                                                                       -     -       NO        
lxcora01  RUNNING  10.207.39.10, 172.220.40.10, 172.221.40.10, 192.210.39.10, 192.211.39.10, 192.212.39.10, 192.213.39.10  -     -       NO        
lxcora02  RUNNING  10.207.39.11, 172.220.40.11, 172.221.40.11, 192.210.39.11, 192.211.39.11, 192.212.39.11, 192.213.39.11  -     -       NO        
lxcora03  RUNNING  10.207.39.12, 172.220.40.12, 172.221.40.12, 192.210.39.12, 192.211.39.12, 192.212.39.12, 192.213.39.12  -     -       NO        
lxcora04  RUNNING  10.207.39.13, 172.220.40.13, 172.221.40.13, 192.210.39.13, 192.211.39.13, 192.212.39.13, 192.213.39.13  -     -       NO        
lxcora05  RUNNING  10.207.39.14, 172.220.40.14, 172.221.40.14, 192.210.39.14, 192.211.39.14, 192.212.39.14, 192.213.39.14  -     -       NO        
lxcora06  RUNNING  172.220.40.15, 172.221.40.15, 192.210.39.15, 192.211.39.15, 192.212.39.15, 192.213.39.15                -     -       NO        
oracle@W521:~/Downloads/scst-files$ sudo lxc-ls -f
NAME      STATE    IPV4                                                                                                    IPV6  GROUPS  AUTOSTART 
--------------------------------------------------------------------------------------------------------------------------------------------------
lxcora00  STOPPED  -                                                                                                       -     -       NO        
lxcora01  RUNNING  10.207.39.10, 172.220.40.10, 172.221.40.10, 192.210.39.10, 192.211.39.10, 192.212.39.10, 192.213.39.10  -     -       NO        
lxcora02  RUNNING  10.207.39.11, 172.220.40.11, 172.221.40.11, 192.210.39.11, 192.211.39.11, 192.212.39.11, 192.213.39.11  -     -       NO        
lxcora03  RUNNING  10.207.39.12, 172.220.40.12, 172.221.40.12, 192.210.39.12, 192.211.39.12, 192.212.39.12, 192.213.39.12  -     -       NO        
lxcora04  RUNNING  10.207.39.13, 172.220.40.13, 172.221.40.13, 192.210.39.13, 192.211.39.13, 192.212.39.13, 192.213.39.13  -     -       NO        
lxcora05  RUNNING  10.207.39.14, 172.220.40.14, 172.221.40.14, 192.210.39.14, 192.211.39.14, 192.212.39.14, 192.213.39.14  -     -       NO        
lxcora06  RUNNING  10.207.39.15, 172.220.40.15, 172.221.40.15, 192.210.39.15, 192.211.39.15, 192.212.39.15, 192.213.39.15  -     -       NO        
oracle@W521:~/Downloads/scst-files$

Once the Oracle Grid Infrastructure 12c (12.1.0.2.0) install media is downloaded, you can scp it to the lxcora01 LXC containers as the "grid" user as shown below.  After scp'g the install media into the lxcora01 LXC containers, ssh into the lxcora01 container as "grid" user using the "-Y" and "-C" switches to enable x-windows for the Oracle installer step, make sure the install media is owned by "grid:oinstall" and unzip, first the "1of2" file and then the "2of2" file.

You can run "xclock" from the command prompt to verify that X is working.

Logged into container as "grid" linux user, unzip the install media as shown in the next following section.

oracle@W521:~/Downloads$ ls -lrt linuxam*

-rw-rw-r-- 1 oracle oracle  646972897 Sep  4 00:58 linuxamd64_12102_grid_2of2.zip
-rw-rw-r-- 1 oracle oracle 1747043545 Sep  4 01:04 linuxamd64_12102_grid_1of2.zip

oracle@W521:~/Downloads$ scp linuxamd64_12102_grid_*.zip grid@lxcora01:~/.
grid@lxcora01's password:

linuxamd64_12102_grid_1of2.zip        100% 1666MB 333.2MB/s   00:05   

linuxamd64_12102_grid_2of2.zip        100%  617MB 308.5MB/s   00:02
  

oracle@W521:~/Downloads$ ssh -Y -C grid@lxcora01
grid@lxcora01's password:
/usr/bin/xauth:  creating new authority file /home/grid/.Xauthority

[grid@lxcora01 ~]$ ls -lrt

total 2337920
drwxr-xr-x 3 grid oinstall       4096 Sep  3 22:27 grid
-rwxr-xr-x 1 grid oinstall         28 Sep  3 22:27 edit_bashrc
-rw-r--r-- 1 grid oinstall 1747043545 Sep  4 01:05 linuxamd64_12102_grid_1of2.zip
-rw-r--r-- 1 grid oinstall  646972897 Sep  4 01:05 linuxamd64_12102_grid_2of2.zip

[grid@lxcora01 ~]$

When you unzip the first Oracle install media file, you can say "y" to overwrite the cvuqdisk rpm file as shown below.  This occurs because the cvuqdisk rpm was installed much earlier as a one-off step so that the cloned containers would already have this package installed at clone time.

[grid@lxcora01 ~]$ unzip linuxamd64_12102_grid_1of2.zip
Archive:  linuxamd64_12102_grid_1of2.zip
replace grid/rpm/cvuqdisk-1.0.9-1.rpm? [y]es, [n]o, [A]ll, [N]one, [r]ename: y

Once both of the Oracle Grid install media files have been unzipped, you can cd into the grid directory (/home/grid/grid) to begin the GRID install with "./runInstaller" as shown below.

[grid@lxcora01 ~]$ ls -lrt
total 2337928
-rwxr-xr-x 1 grid oinstall         28 Sep  3 22:27 edit_bashrc
-rw-r--r-- 1 grid oinstall 1747043545 Sep  4 01:05 linuxamd64_12102_grid_1of2.zip
-rw-r--r-- 1 grid oinstall  646972897 Sep  4 01:05 linuxamd64_12102_grid_2of2.zip
drwxr-xr-x 7 grid oinstall       4096 Sep  4 01:11 grid
[grid@lxcora01 ~]$ cd grid
[grid@lxcora01 grid]$ ./runInstaller

The ntp check must PASS in the installer.  If it fails, you can open a terminal and check to be sure all ntpd processes are running and verify that the command:

ntpq -p

has correct output.

When evaluating the detail of the error, this part of the error details screen helps to point out which nodes have issue.

PRVF-5408 : NTP Time Server "198.55.111.5" is common only to the following nodes "lxcora06,lxcora05,lxcora03,lxcora02,lxcora01"

The above indicates to correct the NTP issues on node lxcora04.

oracle0@W520:~$ ssh root@lxcora4
The authenticity of host 'lxcora4 (10.207.39.14)' can't be established.
RSA key fingerprint is 22:8d:f3:73:49:4c:82:a8:29:a0:00:e1:63:c7:64:89.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora4,10.207.39.14' (RSA) to the list of known hosts.

[root@lxcora4 ~]# ntpq -p
ntpq: read: Connection refused   <-- Incorrect output


[root@lxcora4 ~]# ntpd           <-- This is how you start NTP on the containers (note: "service ntpd start" will NOT work)

[root@lxcora4 ~]# ntpq -p        <-- Expected correct output returned this time


     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 vmem1.vmem.org  198.55.111.5     3 u    3   64    1    0.555    0.076   0.000

[root@lxcora4 ~]

has valid output, and then retry the checks again for the installer.

oracle@W521:~$ ssh root@lxcora01
Last login: Fri Sep  4 01:16:09 2015 from 10.207.39.1
[root@lxcora01 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*vmem1.vmem.org  132.163.4.102    2 u   49   64  377    0.077    0.012   0.053
[root@lxcora01 ~]# ssh lxcora02
The authenticity of host 'lxcora02 (10.207.39.11)' can't be established.
RSA key fingerprint is a9:85:d2:96:d8:38:56:fa:ba:0b:64:be:f0:a7:59:c4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora02,10.207.39.11' (RSA) to the list of known hosts.
root@lxcora02's password:
[root@lxcora02 ~]# ntpq -p
ntpq: read: Connection refused
[root@lxcora02 ~]# ntpd
[root@lxcora02 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 vmem1.vmem.org  132.163.4.102    2 u    2   64    1    0.693    0.097   0.000
[root@lxcora02 ~]# ssh lxcora03
The authenticity of host 'lxcora03 (10.207.39.12)' can't be established.
RSA key fingerprint is a9:85:d2:96:d8:38:56:fa:ba:0b:64:be:f0:a7:59:c4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora03,10.207.39.12' (RSA) to the list of known hosts.
root@lxcora03's password:
[root@lxcora03 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*vmem1.vmem.org  132.163.4.102    2 u   38   64  377    0.209    0.046   0.054
[root@lxcora03 ~]# ssh lxcora04
The authenticity of host 'lxcora04 (10.207.39.13)' can't be established.
RSA key fingerprint is a9:85:d2:96:d8:38:56:fa:ba:0b:64:be:f0:a7:59:c4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora04,10.207.39.13' (RSA) to the list of known hosts.
root@lxcora04's password:
[root@lxcora04 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*vmem1.vmem.org  132.163.4.102    2 u   60   64  377    0.179    0.029   0.074
[root@lxcora04 ~]# ssh lxcora05
The authenticity of host 'lxcora05 (10.207.39.14)' can't be established.
RSA key fingerprint is a9:85:d2:96:d8:38:56:fa:ba:0b:64:be:f0:a7:59:c4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora05,10.207.39.14' (RSA) to the list of known hosts.
root@lxcora05's password:
[root@lxcora05 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*vmem1.vmem.org  132.163.4.102    2 u   62  128  377    0.389    0.052   0.045
[root@lxcora05 ~]# ssh lxcora06
The authenticity of host 'lxcora06 (10.207.39.15)' can't be established.
RSA key fingerprint is a9:85:d2:96:d8:38:56:fa:ba:0b:64:be:f0:a7:59:c4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora06,10.207.39.15' (RSA) to the list of known hosts.
root@lxcora06's password:
[root@lxcora06 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*vmem1.vmem.org  132.163.4.102    2 u   56   64  377    0.463    0.001   0.090
[root@lxcora06 ~]#

Once the Oracle Grid Infrastructure (GI) installer kicks off, you can monitor the installer actions using "tail -f" on the installer log as shown below.  The exact name of your install log file will differ of course, but it will be similar name in format.

[root@lxcora01 logs]# pwd

/u00/app/12.1.0/oraInventory/logs


[root@lxcora01 logs]# ls -lrt

total 1356
-rw------- 1 grid oinstall     124 Sep  4 01:43 oraInstall2015-09-04_01-15-18AM.out
-rw------- 1 grid oinstall       0 Sep  4 01:43 oraInstall2015-09-04_01-15-18AM.err
-rw-r----- 1 grid oinstall     314 Sep  4 01:43 silentInstall2015-09-04_01-15-18AM.log
-rw-r--r-- 1 grid oinstall 1378661 Sep  4 01:43 installActions2015-09-04_01-15-18AM.log

[root@lxcora01 logs]# tail -f installActions2015-09-04_01-15-18AM.log

Note that some of the screenshots near the end of the GI install show the Ubuntu OS screen and the output of the above "tail -f" on the install log.  Below are the install screens and options to choose for this installation.




The values for the screen below, such as "lxc1" and "lxc1-scan.gns1.vmem.org" should be typed in exactly as shonw below.  Be sure to include "." (period symbols) where shown (they are small here and hard to see, examine the screenshot carefully!).  All the values shown must be entered exactly as shown because the install bundle has configured bind9 (aka "named") to resolve these values, so simply just use the exact value below.


Below the addtional lxcora0x nodes need to be added to the list of nodes as shown below.

NOTE: The v13 bundles starts container numbering at "lxcora10" so the below screenshot is OUT OF DATE and needs update.  The naming of containers starts at lxcora10 to workaround an Oracle bug related to the number of characters in the node names.


Pressing the "Add" button will bring up this dialog as shown below. Add each of the additional nodes in the same way.  The Node Role is always "HUB" and the "Virtual Hostname" is always "AUTO" so just accept the defaults for each pop-up add cluster node information dialog box.


The final list of nodes on the screen is shown below (for 6 nodes).  If you chose fewer nodes then you will of course add fewer nodes here.


The below screen looks almost identical to the previous one, but note that the SSH connectivity lower part of screen has been activiated by clicking on the "SSH connectivity" button.  Enter "grid" as the password and then click on "Setup" to setup no-password ssh for "grid" user across all LXC nodes.




Note, if for any reason passowrdless SSH setup fails, just try the tool again. Click "Next" to go to the next screen.


Now that passwordless SSH is setup, the next screen is the interface setup.  Use the dropdowns to allocate the interfaces exactly as shown below.  The Private is a 4-bond HAIP for the interconnect, and the two ASM interfaces are for the ASM Flex Cluster network.  These are all OpenvSwitch networks.  All the networking for this RAC is OpenvSwitch.


The below screen will not show the "Disk Path" values until you change the "Disk Discovery Path" as shown next screen following below by clicking on "Change Discovery Path" and changing it to "/dev/mapper/asm*" as shown below. 




Set a password for the accounts as shown below.


Select "Do not use IPMI" option.


Do not register with EM Cloud Control unless you are already running it.


Accept the defaults for the groups as shown below and click next.


Set the "Oracle base" and the "Software location" exactly as shown below.


Accept the default Inventory Directory exactly as shown below.


Type in the password which is "root" for the root user so that the installation software can automatically run the root installation scripts automatically.  I have tested this option and it works very reliably.  If you monitor the install log with the "tail -f" that was shown before the screenshots section, you can monitor the progress of the root.sh scripts running.


The Oracle pre-install Cluster Verification Utility runs as shown below.



The "Network Time Protocol" check should SUCCEED (should be no warnings for NTP).  If you get an NTP warning as shown below, it should be addressed and the cluster verification step rerun.


You can click on the "(more details)" link and get the popup dialog with more details on the Cluster Verification check details for that check, as shown below.


After checking all the LXC containers, one container was found to have NTP not running,  NTP was started on that LXC containers, the "ntpq -p" command was run on that LXC container, and then the verification check was rerun using the "Check Again" button.  This time there was no error for NTP, which is the desired result.

The check warnings and failures below are the ONLY warnings and failures that should occur.  The "OS kernel parameter" warnings for {rmem_default, rmem_max, wmem_default, and wmem_max} can be safely ignored because we have set these values at the host level (in the Ubuntu sysctl.conf file) and since all LXC containers share the same kernel with the host, these are actually set where they need to be.  The only reason that check fails is because the check itself does not know how to check the host kernel values.

The Device Checks for ASM is a known error about not being able to validate the path on all nodes and can be safely ignored.  The same is true of the resolv.conf integrity check, the DHCP check, and the /boot mount check, all of which can be safely ignored.

Therefore, next check the "Ignore All" box as long as only the below warnings and failed messages looks just like the screenshot below with the exact same warnings, and click on "Next" after checking the "Ignore All" box.


Check the "Ignore All" button in the upper-right-corner of the screen as shown below.  Then click "Next" to continue.


Click on "Yes" to continue with the install and confirm that the warnings are being ignored.


The install summary screen is displayed as shown below.


Save the response file so that a "silent" install could be done on another subsequent install.  Click on "Save" to save the response file.


The install begins and the progress bar indicates progress of the install as shown below.



When the installer gets to running the root scripts, it asks for confirmation to run them.  click on "Yes' to run the root scripts.


The root scripts can be monitored using the "tail -f" as previously described and as shown below.



Sometimes the Oracle Cluster Verification may fail.  The Cluster Verification Utillity can be re-run by clicking on "Retry".  In this case as seen in the remaining screenshots, the second time Oracle Cluster Veriication Utility ran, it was successful.  However, even if CVU is not successful, it usually does not affect the install, and you would find the install was totally successful.  So if after a couple of retries of the CVU it still shows some checks failed, then just ignore and move on to the next screen.  Note the CVU check can take a few minutes.






The install is successful.  Close the installer.



Here is the output below of how the initially installed Oracle 12c ASM Flex Cluster will look immediately after the runInstaller session completes.  Notice how only 3 ASM instances and associated services are up. This is because the default cardinality of ASM in a Flex Cluster is "3".  This can be changed so that ALL services are up using the command:

[root@lxcora01 ] srvctl modify asm -count 6

for example if as in this case the ASM Flex Cluster has 6 nodes.

So, as shown below, running the command to modify ASM and re-run "crsctl stat res -t" gives the following output as shown below.  Now as can be seen below, ALL installed and configured services have status ONLINE ONLINE including ALL of the 6 ASM instances and associated ASMNET1LSNR services.

[root@lxcora01 logs]# which srvctl

/u00/app/grid/product/12.1.0/grid/bin/srvctl

[root@lxcora01 logs]# srvctl modify asm -count 6

[root@lxcora01 logs]# crsctl stat res -t

--------------------------------------------------------------------------------
Name           Target  State        Server                   State details      
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       lxcora01                 STABLE
               ONLINE  ONLINE       lxcora02                 STABLE
               ONLINE  ONLINE       lxcora03                 STABLE
               ONLINE  ONLINE       lxcora04                 STABLE
               ONLINE  ONLINE       lxcora05                 STABLE
               ONLINE  ONLINE       lxcora06                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       lxcora01                 STABLE
               ONLINE  ONLINE       lxcora02                 STABLE
               ONLINE  ONLINE       lxcora03                 STABLE
               ONLINE  ONLINE       lxcora04                 STABLE
               ONLINE  ONLINE       lxcora05                 STABLE
               ONLINE  ONLINE       lxcora06                 STABLE
ora.SYSTEMDG.dg
               ONLINE  ONLINE       lxcora01                 STABLE
               ONLINE  ONLINE       lxcora02                 STABLE
               ONLINE  ONLINE       lxcora03                 STABLE
               ONLINE  ONLINE       lxcora04                 STABLE
               ONLINE  ONLINE       lxcora05                 STABLE
               ONLINE  ONLINE       lxcora06                 STABLE
ora.net1.network
               ONLINE  ONLINE       lxcora01                 STABLE
               ONLINE  ONLINE       lxcora02                 STABLE
               ONLINE  ONLINE       lxcora03                 STABLE
               ONLINE  ONLINE       lxcora04                 STABLE
               ONLINE  ONLINE       lxcora05                 STABLE
               ONLINE  ONLINE       lxcora06                 STABLE
ora.ons
               ONLINE  ONLINE       lxcora01                 STABLE
               ONLINE  ONLINE       lxcora02                 STABLE
               ONLINE  ONLINE       lxcora03                 STABLE
               ONLINE  ONLINE       lxcora04                 STABLE
               ONLINE  ONLINE       lxcora05                 STABLE
               ONLINE  ONLINE       lxcora06                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       lxcora06                 STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       lxcora02                 STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       lxcora03                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       lxcora01                 169.254.39.117 192.2
                                                             10.39.10 192.211.39.
                                                             10,STABLE
ora.asm
      1        ONLINE  ONLINE       lxcora01                 Started,STABLE
      2        ONLINE  ONLINE       lxcora06                 Started,STABLE
      3        ONLINE  ONLINE       lxcora02                 Started,STABLE
      4        ONLINE  ONLINE       lxcora04                 Started,STABLE
      5        ONLINE  ONLINE       lxcora05                 Started,STABLE
      6        ONLINE  ONLINE       lxcora03                 Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       lxcora01                 STABLE
ora.gns
      1        ONLINE  ONLINE       lxcora02                 STABLE
ora.gns.vip
      1        ONLINE  ONLINE       lxcora02                 STABLE
ora.lxcora01.vip
      1        ONLINE  ONLINE       lxcora01                 STABLE
ora.lxcora02.vip
      1        ONLINE  ONLINE       lxcora02                 STABLE
ora.lxcora03.vip
      1        ONLINE  ONLINE       lxcora03                 STABLE
ora.lxcora04.vip
      1        ONLINE  ONLINE       lxcora04                 STABLE
ora.lxcora05.vip
      1        ONLINE  ONLINE       lxcora05                 STABLE
ora.lxcora06.vip
      1        ONLINE  ONLINE       lxcora06                 STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       lxcora01                 Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       lxcora01                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       lxcora06                 STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       lxcora02                 STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       lxcora03                 STABLE
--------------------------------------------------------------------------------
[root@lxcora01 logs]#

The "iotop" package can be installed for monitoring IO across all the instances as shown below.  Once installed run "sudo iotop" to start the monitor.

oracle@W521:~/Downloads$ sudo iotop
[sudo] password for oracle:
sudo: iotop: command not found
oracle@W521:~/Downloads$ sudo apt-get install iotop
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following packages were automatically installed and are no longer required:
  linux-headers-3.19.0-15 linux-headers-3.19.0-15-generic linux-image-3.19.0-15-generic linux-image-extra-3.19.0-15-generic
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
  iotop
0 upgraded, 1 newly installed, 0 to remove and 8 not upgraded.
Need to get 23.8 kB of archives.
After this operation, 127 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ vivid/universe iotop amd64 0.6-1 [23.8 kB]
Fetched 23.8 kB in 1s (17.2 kB/s)      
Selecting previously unselected package iotop.
(Reading database ... 236524 files and directories currently installed.)
Preparing to unpack .../archives/iotop_0.6-1_amd64.deb ...
Unpacking iotop (0.6-1) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up iotop (0.6-1) ...
oracle@W521:~/Downloads$

You can view the OpenvSwitch at the Ubuntu host level using the command "sudo ovs-vsctl show".

To login from the Ubuntu host to the Oracle in the containers, I recommend installing Oracle Instant Client as described here.  The Oracle RAC is readily accessible from the Ubuntu host over sqlplus.

To ssh into the containers, just use:

ssh root@lxcora01  (or whichever container you want to ssh to).

To stop the container:

sudo lxc-stop -n lxcora01

To start the container

sudo lxc-start -n lxcora01

To list all your containers

sudo lxc-ls -f

Note that when you reboot the desktop, you'll need to login manually to SCST on 10.207.40.1 and 10.207.41.1, the storage networks using the appropriate iscsiadm command (which can be found in the create-scst-*.sh scripts) to get the LUNs active in /dev/mapper on the Ubuntu host.  You'll then need to start the LXC containers,which will take care of mapping the storage LUNs into themselves and starting up Oracle GI automatically in the usual way.


Also note that it can take up to 15 seconds of so for the DHCP provided public IP for each RAC node (the 10.207.39.x network) to come up.  Be sure that IP comes up.  It's best to start the containers one by one and verify each one has started up it's public IP ok using sudo lxc-ls -f before starting the next containers.

10-lxc-node containerized 12.1.0.2.0 Oracle RAC ASM Flex Cluster with GNS running on Ubuntu 15.04 64-bit desktop shown below.

oracle0@W520:~/orabuntu-lxc$ crsstat
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details      
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       lxcora10                 STABLE
               ONLINE  ONLINE       lxcora11                 STABLE
               ONLINE  ONLINE       lxcora12                 STABLE
               ONLINE  ONLINE       lxcora13                 STABLE
               ONLINE  ONLINE       lxcora14                 STABLE
               ONLINE  ONLINE       lxcora15                 STABLE
               ONLINE  ONLINE       lxcora16                 STABLE
               ONLINE  ONLINE       lxcora17                 STABLE
               ONLINE  ONLINE       lxcora18                 STABLE
               ONLINE  ONLINE       lxcora19                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       lxcora10                 STABLE
               ONLINE  ONLINE       lxcora11                 STABLE
               ONLINE  ONLINE       lxcora12                 STABLE
               ONLINE  ONLINE       lxcora13                 STABLE
               ONLINE  ONLINE       lxcora14                 STABLE
               ONLINE  ONLINE       lxcora15                 STABLE
               ONLINE  ONLINE       lxcora16                 STABLE
               ONLINE  ONLINE       lxcora17                 STABLE
               ONLINE  ONLINE       lxcora18                 STABLE
               ONLINE  ONLINE       lxcora19                 STABLE
ora.SYSTEMDG.dg
               ONLINE  ONLINE       lxcora10                 STABLE
               ONLINE  ONLINE       lxcora11                 STABLE
               OFFLINE OFFLINE      lxcora12                 STABLE
               ONLINE  ONLINE       lxcora13                 STABLE
               OFFLINE OFFLINE      lxcora14                 STABLE
               OFFLINE OFFLINE      lxcora15                 STABLE
               OFFLINE OFFLINE      lxcora16                 STABLE
               OFFLINE OFFLINE      lxcora17                 STABLE
               OFFLINE OFFLINE      lxcora18                 STABLE
               OFFLINE OFFLINE      lxcora19                 STABLE
ora.net1.network
               ONLINE  ONLINE       lxcora10                 STABLE
               ONLINE  ONLINE       lxcora11                 STABLE
               ONLINE  ONLINE       lxcora12                 STABLE
               ONLINE  ONLINE       lxcora13                 STABLE
               ONLINE  ONLINE       lxcora14                 STABLE
               ONLINE  ONLINE       lxcora15                 STABLE
               ONLINE  ONLINE       lxcora16                 STABLE
               ONLINE  ONLINE       lxcora17                 STABLE
               ONLINE  ONLINE       lxcora18                 STABLE
               ONLINE  ONLINE       lxcora19                 STABLE
ora.ons
               ONLINE  ONLINE       lxcora10                 STABLE
               ONLINE  ONLINE       lxcora11                 STABLE
               ONLINE  ONLINE       lxcora12                 STABLE
               ONLINE  ONLINE       lxcora13                 STABLE
               ONLINE  ONLINE       lxcora14                 STABLE
               ONLINE  ONLINE       lxcora15                 STABLE
               ONLINE  ONLINE       lxcora16                 STABLE
               ONLINE  ONLINE       lxcora17                 STABLE
               ONLINE  ONLINE       lxcora18                 STABLE
               ONLINE  ONLINE       lxcora19                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       lxcora12                 STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       lxcora10                 STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       lxcora11                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       lxcora10                 169.254.28.57 192.21
                                                             0.39.10 192.211.39.1
                                                             0,STABLE
ora.asm
      1        ONLINE  ONLINE       lxcora10                 Started,STABLE
      2        ONLINE  ONLINE       lxcora11                 Started,STABLE
      3        ONLINE  ONLINE       lxcora13                 Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       lxcora14                 STABLE
ora.gns
      1        ONLINE  ONLINE       lxcora10                 STABLE
ora.gns.vip
      1        ONLINE  ONLINE       lxcora10                 STABLE
ora.lxcora10.vip
      1        ONLINE  ONLINE       lxcora10                 STABLE
ora.lxcora11.vip
      1        ONLINE  ONLINE       lxcora11                 STABLE
ora.lxcora12.vip
      1        ONLINE  ONLINE       lxcora12                 STABLE
ora.lxcora13.vip
      1        ONLINE  ONLINE       lxcora13                 STABLE
ora.lxcora14.vip
      1        ONLINE  ONLINE       lxcora14                 STABLE
ora.lxcora15.vip
      1        ONLINE  ONLINE       lxcora15                 STABLE
ora.lxcora16.vip
      1        ONLINE  ONLINE       lxcora16                 STABLE
ora.lxcora17.vip
      1        ONLINE  ONLINE       lxcora17                 STABLE
ora.lxcora18.vip
      1        ONLINE  ONLINE       lxcora18                 STABLE
ora.lxcora19.vip
      1        ONLINE  ONLINE       lxcora19                 STABLE
ora.lxcora20.vip
      1        ONLINE  INTERMEDIATE lxcora12                 FAILED OVER,STABLE
ora.lxcora21.vip
      1        ONLINE  INTERMEDIATE lxcora11                 FAILED OVER,STABLE
ora.lxcora22.vip
      1        ONLINE  INTERMEDIATE lxcora10                 FAILED OVER,STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       lxcora10                 Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       lxcora18                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       lxcora12                 STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       lxcora10                 STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       lxcora11                 STABLE
--------------------------------------------------------------------------------
oracle0@W520:~/orabuntu-lxc$





ċ
ubuntu-lxc-oracle.v1.tar.gz
(48k)
Gilbert Standen,
Sep 7, 2015, 3:00 PM
ċ
ubuntu-lxc-oracle.v10.tar.gz
(51k)
Gilbert Standen,
Sep 7, 2015, 3:00 PM
ċ
ubuntu-lxc-oracle.v11.tar.gz
(52k)
Gilbert Standen,
Sep 7, 2015, 6:11 PM
ċ
ubuntu-lxc-oracle.v12.tar
(480k)
Gilbert Standen,
Sep 13, 2015, 11:13 AM
ċ
ubuntu-lxc-oracle.v13.tar
(480k)
Gilbert Standen,
Sep 13, 2015, 9:49 PM
ċ
ubuntu-lxc-oracle.v2.tar.gz
(49k)
Gilbert Standen,
Sep 7, 2015, 3:00 PM
ċ
ubuntu-lxc-oracle.v3.tar.gz
(50k)
Gilbert Standen,
Sep 7, 2015, 3:00 PM
ċ
ubuntu-lxc-oracle.v4.tar.gz
(50k)
Gilbert Standen,
Sep 7, 2015, 3:00 PM
ċ
ubuntu-lxc-oracle.v5.tar.gz
(50k)
Gilbert Standen,
Sep 7, 2015, 3:00 PM
ċ
ubuntu-lxc-oracle.v6.tar.gz
(50k)
Gilbert Standen,
Sep 7, 2015, 3:00 PM
ċ
ubuntu-lxc-oracle.v8.tar.gz
(50k)
Gilbert Standen,
Sep 7, 2015, 3:00 PM
ċ
ubuntu-lxc-oracle.v9.tar.gz
(51k)
Gilbert Standen,
Sep 7, 2015, 3:00 PM
Comments