This blog post is intended to be a complete,
comprehensive step-by-step how-to for installing a 6-node Oracle 12c RAC GNS ASM
Flex Cluster on Ubuntu 15.04 using Oracle Enterprise Linux 6.5 LXC Linux
Containers and OpenvSwitch. The guide that is being created here now is based on a fresh install of Ubuntu 15.04 Vivid Vervet 64-bit desktop edition running on a Lenovo W520 mobile workstation. The way I did the LUN presentation to Oracle running in the LXC containers previously was to build a KVM Oracle Enterprise Linux (OEL) 6.5 guest and then compile SCST into the kernel of that KVM guest. Since SCST has to be compiled into the kernel, installing it on the base Ubuntu 15.04 OS presents the problem that the kernel will have to be recompiled every time there is an update, and with Ubuntu that is typically going to be very frequently. The downside is that KVM and SCST do not provide all the great of IO relatively speaking to what one can get in a Linux Container, and so all Oracle IO to the LUNs on the KVM guest running SCST has to go through KVM which will degrade IO performance to the Oracle DB measurably. A less than optimal solution from that point of view. I haven't got a particularly good solution at this point for a better mousetrap for presenting the LUNs other than using a KVM guest, because, the nice thing about KVM is that I can set the logical and physical sector size on the LUNs with KVM, and SCST also has a manual setting for 4K physical sector size LUNs, so this KVM-SCST combo allows this lab system to present both 512/512 and 4096/4096 and 512/4096 LUN configurations to Oracle so that the setup can be used to play with Oracle using 4K native LUNs and/or 4K native LUNs with 512 emulation mode. This guide is also a de-facto recipe for creating an OpenvSwitch-based Ubuntu Linux [ desktop | laptop ] networking environment that accomplishes a number of key goals as enumerated below. Design AssumptionsThis set
of procedures has been tested and built on a fresh install of the
following Ubuntu distributions. Installation on matching Ubuntu
installations which have been running for a few months or years and have
been customized could possibly introduce variations of existing
configuration that could cause the steps described in this blog to have
results different from the desired and expected results, YMMV. This
blog to reiterate assumes a fresh install of the following
distributions, and has been built and tested successfully on all of the
following distributions of Ubuntu Linux.
Note,
if installing Ubuntu 14.04 or 14.10 or 15.04 fresh for this work, do
NOT accept the "download updates during install" option. The reason is
because these updates contain a number of CVE kernel security updates,
and this procedure uses the post install Ubuntu CVE updates to check and
verify correct operation of Oracle Ksplice Uptrack, so, in other words,
Oracle Ksplice will handle the post-install CVE kernel security
updates. Once KSplice operation has been so verified, Ubuntu Software
Updater can be used for updates going forward as normal. Oracle KSplice
can be used for urgent CVE kernel security updates with zero downtime
as needed. This particular blog is for Ubuntu 15.04 64-bit desktop Vivid Vervet. DefinitionsDEU Dense elastici unit. What formerly was called a "VM" but which now, with the advent of Linux Containers, requires imho a new term which includes density and elasticity solutions such as LXC which do not use hardware virtualization and as such are not "machines" at all. So in this blog, a "DEU" will be used as the all-inclusive term to describe a single LXC Linux Container". VM A type of DEU which uses a hypervisor to virtualize (emulate using software) the hardware layers of an actual physical machine and provides what is often called a "guest" operating system. LXC The
term LXC will be used interchangeably according to context to either
mean the software called LXC which is a Linux Container software in the
same category as OpenVZ and others, and in other contexts, LXC may be
used to denote a specific single LXC DEU which is running on LXC
software. Design Features
The
installation of bind9 is constructed so as not to interfere with the
default Ubuntu desktop; that is, both bind9 and dnsmasq-base coexist
peacefully and successfully so that even with the OpenvSwitch and bind9
installed, all the features of Ubuntu NetworkManager are preserved,
including use of wifi network manager, VPN manager, bluetooth, and in
general ALL default and configurable features of Ubuntu Network
Manager. Install Synaptic Package Manager (optional)Synaptic
Package Manager is not required, but has some very nice features and
since old-timers to Ubuntu sometimes like it, so it
can be installed. It was the GUI for package management in Ubuntu Linux
desktop prior to the introduction of the "Ubuntu Software Manager" and
so some folks like to have it in addtion to the Software Manager.
Both tools can coexist, and synaptic can sometimes be very useful for
some tasks.gstanden@u1504:~$ sudo apt-get install synaptic[sudo] password for gstanden: Reading package lists... DoneBuilding dependency tree Reading state information... DoneThe following extra packages will be installed: docbook-xml libept1.4.12 librarian0 libvte-2.90-9 libvte-2.90-common rarian-compat sgml-dataSuggested packages: docbook docbook-dsssl docbook-xsl docbook-defguide perlsgml w3-recs opensp libxml2-utils dwww menu deborphan taskselThe following NEW packages will be installed: docbook-xml libept1.4.12 librarian0 libvte-2.90-9 libvte-2.90-common rarian-compat sgml-data synaptic0 upgraded, 8 newly installed, 0 to remove and 53 not upgraded.Need to get 2,413 kB of archives.After this operation, 14.2 MB of additional disk space will be used.Do you want to continue? [Y/n] YGet:1 http://us.archive.ubuntu.com/ubuntu/ vivid/main sgml-data all 2.0.10 [173 kB]Get:2 http://us.archive.ubuntu.com/ubuntu/ vivid/main docbook-xml all 4.5-7.2 [336 kB]Get:3 http://us.archive.ubuntu.com/ubuntu/ vivid/main libept1.4.12 amd64 1.0.12.1 [125 kB]Get:4 http://us.archive.ubuntu.com/ubuntu/ vivid/main libvte-2.90-common all 1:0.36.3-1ubuntu2 [24.5 kB]Get:5 http://us.archive.ubuntu.com/ubuntu/ vivid/main libvte-2.90-9 amd64 1:0.36.3-1ubuntu2 [278 kB]Get:6 http://us.archive.ubuntu.com/ubuntu/ vivid/main librarian0 amd64 0.8.1-6 [50.4 kB]Get:7 http://us.archive.ubuntu.com/ubuntu/ vivid/main rarian-compat amd64 0.8.1-6 [55.7 kB]Get:8 http://us.archive.ubuntu.com/ubuntu/ vivid/universe synaptic amd64 0.81.3 [1,370 kB]Fetched
2,413 kB in 6s (355
kB/s)
Selecting previously unselected package sgml-data.(Reading database ... 173095 files and directories currently installed.)Preparing to unpack .../sgml-data_2.0.10_all.deb ...Unpacking sgml-data (2.0.10) ...Selecting previously unselected package docbook-xml.Preparing to unpack .../docbook-xml_4.5-7.2_all.deb ...Unpacking docbook-xml (4.5-7.2) ...Selecting previously unselected package libept1.4.12:amd64.Preparing to unpack .../libept1.4.12_1.0.12.1_amd64.deb ...Unpacking libept1.4.12:amd64 (1.0.12.1) ...Selecting previously unselected package libvte-2.90-common.Preparing to unpack .../libvte-2.90-common_1%3a0.36.3-1ubuntu2_all.deb ...Unpacking libvte-2.90-common (1:0.36.3-1ubuntu2) ...Selecting previously unselected package libvte-2.90-9.Preparing to unpack .../libvte-2.90-9_1%3a0.36.3-1ubuntu2_amd64.deb ...Unpacking libvte-2.90-9 (1:0.36.3-1ubuntu2) ...Selecting previously unselected package librarian0.Preparing to unpack .../librarian0_0.8.1-6_amd64.deb ...Unpacking librarian0 (0.8.1-6) ...Selecting previously unselected package rarian-compat.Preparing to unpack .../rarian-compat_0.8.1-6_amd64.deb ...Unpacking rarian-compat (0.8.1-6) ...Selecting previously unselected package synaptic.Preparing to unpack .../synaptic_0.81.3_amd64.deb ...Unpacking synaptic (0.81.3) ...Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...Processing triggers for doc-base (0.10.6) ...Scrollkeeper was installed, forcing re-registration of all documents.Unregistering 35 doc-base files, re-registering 35 doc-base files...Registering documents with scrollkeeper...Processing triggers for gnome-menus (3.10.1-0ubuntu5) ...Processing triggers for desktop-file-utils (0.22-1ubuntu3) ...Processing triggers for bamfdaemon (0.5.1+15.04.20150202-0ubuntu1) ...Rebuilding /usr/share/applications/bamf-2.index...Processing triggers for mime-support (3.58ubuntu1) ...Processing triggers for hicolor-icon-theme (0.14-0ubuntu1) ...Processing triggers for man-db (2.7.0.2-5) ...Setting up sgml-data (2.0.10) ...Setting up libept1.4.12:amd64 (1.0.12.1) ...Setting up libvte-2.90-common (1:0.36.3-1ubuntu2) ...Setting up libvte-2.90-9 (1:0.36.3-1ubuntu2) ...Setting up librarian0 (0.8.1-6) ...Setting up synaptic (0.81.3) ...Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...Setting up docbook-xml (4.5-7.2) ...Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...Setting up rarian-compat (0.8.1-6) ...Processing triggers for libc-bin (2.21-0ubuntu4) ...gstanden@u1504:~$Install Oracle Ksplice (optional)Oracle Ksplice is a free product for Ubuntu Linux which allows updates of kernel software with no downtime. Oracle Ksplice is not required for this project, but as Wim Coaekerts has pointed out here, Oracle Ksplice is likely to play a big role for organizations using Linux Containers, because Ksplice allows zero downtime kernel security patching. So it is included in this blog as part of this build, especially since Ksplice is free to install and use the service for Ubuntu Linux.
Oracle Ksplice will be a key part of any LXC Linux Container deployment
because it allows all LXC Containers to continue running during
security updates to the single kernel used by all LXC containers on the
system. Download Oracle KSplice UptrackThe
Oracle KSplice website already has support for Ubuntu 15.04 64-bit
deskop Vivid Vervet. Not only that, but the install instructions at the
KSplice website worked perfectly and were much simpler than what had to
be done (in my experience anyway) before when I did this install on
14.04 and 14.10. The instructions for installing Oracle KSplice onto
Ubuntu 15.04 are here at Oracle KSplice Website
and I followed the instructions there verbatim and the install went
perfectly according to those instructions no glitches. Click on the red
text "Show Ubuntu Installation Instructions" and just follow them
exactly.  My
install screen shots for Oracle KSplice were as shown below. After
clicking on the "Download" red button in the screenshot above, you get
the download dialog on the Ubuntu desktop. Choose the "open with Ubuntu
Software Center (default)" option as shown below.  The
"Ubuntu Software Center" opens. Click the "Install" button at the
upper right hand side of the screen below the black bar. The install of
KSplice begins as shown below.  A
dialog box opens to authenticate the installation, type in your
password for your linux user, and then another box opens to accept the
KSplice Uptrack Subscription Agreement. This is a free tool for Ubuntu
Desktop users, so tick the agreement, and click "Forward" as shown
below.  If
any dialogs open up to authenticate, just use the Linux password that
was set for your linux userid when Ubuntu was installed. The KSplice
Uptrack Manager runs as shown below advising of security updates to the
kernel that need to be applied. Click on "Install All Updates" as shown
below.  Type in the user password for your Ubuntu linux account to authenticate KSplice Uptrack Manager as shown below.  Oracle KSplice Uptrack runs and then displays that all security kernel updates have been applied as shown below.  Run
Oracle KSplice Uptrack regularly to check for kernel security updates
to be applied with zero downtime (no reboot needed). This is the
functionality that KSplice Uptrack technology provides: zero-downtime
kernel security updates. Update Fresh Installation (optional)If this is a fresh install, then update Ubuntu software to get all most recent updates using the Software Updater app or "sudo apt-get update". If this is not a fresh install, optionally check for updates using the same steps.Install LXC, OpenvSwitch, and UML-Utilities PackagesNow install other required packages as shown below. gstanden@u1504:~$ sudo apt-get install lxc uml-utilities openvswitch-switch gstanden@u1504:~$ sudo apt-get install bind9 bind9utils isc-dhcp-serverInstall OpenvSwitch Configuration ScriptsThe scripts of the form "crt_ovs_*.sh" where * in {sw1, sw2, sw3, ... sw9, sx1 } in the following directory:/home/gstanden/OpenvSwitch/are the scripts that create and configure the OpenvSwitch switches. For this installation, those scripts are called at boot time after the network has started by the script /etc/network/if-up.d/openvswitch-netand the contents of the openvswitch-net file is shown below. This is the file that calls the files that create the openvswitch for this Oracle RAC installation. This script runs automatically at boot, but after the network has started. This 6-node Oracle RAC will run on this OpenvSwitch which is created by these scripts. gstanden@vmem1:/etc/network/if-up.d$ more openvswitch-net#!/bin/sh# openvswitch-net - Start OpenvSwitch## description "Start OpenvSwitch Switches"# start on net-device-up# task# script/home/gstanden/OpenvSwitch/crt_ovs_sx1.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sx1.log/home/gstanden/OpenvSwitch/crt_ovs_sw1.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw1.log/home/gstanden/OpenvSwitch/crt_ovs_sw2.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw2.log/home/gstanden/OpenvSwitch/crt_ovs_sw3.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw3.log/home/gstanden/OpenvSwitch/crt_ovs_sw4.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw4.log/home/gstanden/OpenvSwitch/crt_ovs_sw5.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw5.log/home/gstanden/OpenvSwitch/crt_ovs_sw6.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw6.log/home/gstanden/OpenvSwitch/crt_ovs_sw7.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw7.log/home/gstanden/OpenvSwitch/crt_ovs_sw8.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw8.log/home/gstanden/OpenvSwitch/crt_ovs_sw9.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw9.log# end scriptgstanden@vmem1:/etc/network/if-up.d$ The scripts above which are called by the openvswitch-net script are shown below. They are also attached at the end of this blog and can be downloaded. These scripts are based on work that was published by Jean-Jacquest Sarton here Advanced Networking Page by Jean-Jacques Sarton . I would like to take the opportunity to thank Mr. Sarton very sincerely for publishing his work, without which the present work to run Oracle ASM RAC Flex Clusters on OpenvSwitch might not have progressed forward. gstanden@vmem1:~/OpenvSwitch$ more crt_ovs_s*.sh::::::::::::::crt_ovs_sw1.sh::::::::::::::#!/bin/bash# Requires use of Upstart Script /etc/init/my-network-up.conf to ensure interfaces are up before running. tunctl -t s1tunctl -t s2tunctl -t s3tunctl -t s4tunctl -t s5#tunctl -t s6ip link set s1 upip link set s2 upip link set s3 upip link set s4 upip link set s5 up#ip link set s6 upovs-vsctl add-br sw1ovs-vsctl add-port sw1 s1ovs-vsctl add-port sw1 s2ovs-vsctl add-port sw1 s3ovs-vsctl add-port sw1 s4ovs-vsctl add-port sw1 s5#ovs-vsctl add-port sw1 s6ip link set up dev sw1ip addr add 10.207.39.1/24 dev sw1ip route replace 10.207.39.0/24 dev sw1ovs-vsctl set port sw1 trunks=10ovs-vsctl set port sw1 tag=10# GLS 20140825 Get active external interface dynamically at boot. Tested & works with {wlan0, eth0, bnep0} on Ubuntu 14.04.1 Desktop x86_64.# GLS 20140825 Interface "bnep0" is for example Blackberry Z30 OS10 Bluetooth Tether, or any smartphone bluetooth tether.### BEGIN Get Active EXTIF Dynamcially. ###function GetInterface{ifconfig | egrep -B1 'inet addr' | egrep -A1 'wlan|eth|bnep' | sed '$!N;s/\n/ /' | sed 's/ */ /g' | cut -f1,7 -d' ' | sed 's/ addr//' | head -1 | cut -f1 -d':'}function GetIP{ifconfig | egrep -B1 'inet addr' | egrep -A1 'wlan|eth|bnep' | sed '$!N;s/\n/ /' | sed 's/ */ /g' | cut -f1,7 -d' ' | sed 's/ addr//' | head -1 | cut -f2 -d':'}### END Get Active EXTIF Dynamically. ###echo ' IP: '$(GetIP)echo 'Interface: '$(GetInterface)INTIF="sw1"EXTIF=$(GetInterface)# EXTIF="wlan0"echo 1 > /proc/sys/net/ipv4/ip_forward# clear existing iptable rules, set a default policyiptables -P INPUT ACCEPTiptables -F INPUT iptables -P OUTPUT ACCEPTiptables -F OUTPUT iptables -P FORWARD DROPiptables -F FORWARD iptables -t nat -F# set forwarding and nat rulesiptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPTiptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPTiptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADEservice isc-dhcp-server startservice bind9 restart::::::::::::::crt_ovs_sw2.sh::::::::::::::#!/bin/bashtunctl -t t1tunctl -t t2tunctl -t t3tunctl -t t4tunctl -t t5ip link set t1 upip link set t2 upip link set t3 upip link set t4 upip link set t5 upovs-vsctl add-br sw2ovs-vsctl add-port sw2 t1ovs-vsctl add-port sw2 t2ovs-vsctl add-port sw2 t3ovs-vsctl add-port sw2 t4ovs-vsctl add-port sw2 t5ip link set up dev sw2ip addr add 10.207.40.1/24 dev sw2ip route replace 10.207.40.0/24 dev sw2ifconfig sw2 10.207.40.1 netmask 255.255.255.0ovs-vsctl set port sw2 tag=80# INTIF="sw2"# EXTIF="wlan0"# echo 1 > /proc/sys/net/ipv4/ip_forward# clear existing iptable rules, set a default policy# iptables -P INPUT ACCEPT# iptables -F INPUT # iptables -P OUTPUT ACCEPT# iptables -F OUTPUT # iptables -P FORWARD DROP# iptables -F FORWARD # iptables -t nat -F# set forwarding and nat rules# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE# service isc-dhcp-server start::::::::::::::crt_ovs_sw3.sh::::::::::::::#!/bin/bashtunctl -t w1tunctl -t w2tunctl -t w3tunctl -t w4tunctl -t w5ip link set w1 upip link set w2 upip link set w3 upip link set w4 upip link set w5 upovs-vsctl add-br sw3ovs-vsctl add-port sw3 w1ovs-vsctl add-port sw3 w2ovs-vsctl add-port sw3 w3ovs-vsctl add-port sw3 w4ovs-vsctl add-port sw3 w5ip link set up dev sw3ip addr add 10.207.41.1/24 dev sw3ip route replace 10.207.41.0/24 dev sw3ifconfig sw3 10.207.41.1 netmask 255.255.255.0 ovs-vsctl set port sw3 tag=90# INTIF="sw3"# EXTIF="wlan0"# echo 1 > /proc/sys/net/ipv4/ip_forward# clear existing iptable rules, set a default policy# iptables -P INPUT ACCEPT# iptables -F INPUT # iptables -P OUTPUT ACCEPT# iptables -F OUTPUT # iptables -P FORWARD DROP# iptables -F FORWARD # iptables -t nat -F# set forwarding and nat rules# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE# service isc-dhcp-server start::::::::::::::crt_ovs_sw4.sh::::::::::::::#!/bin/bashovs-vsctl add-br sw4ip link set up dev sw4ip addr add 192.210.39.1/24 dev sw4ip route replace 192.210.39.0/24 dev sw4ifconfig sw4 192.210.39.1 netmask 255.255.255.0# INTIF="sw3"# EXTIF="wlan0"# echo 1 > /proc/sys/net/ipv4/ip_forward# clear existing iptable rules, set a default policy# iptables -P INPUT ACCEPT# iptables -F INPUT # iptables -P OUTPUT ACCEPT# iptables -F OUTPUT # iptables -P FORWARD DROP# iptables -F FORWARD # iptables -t nat -F# set forwarding and nat rules# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE# service isc-dhcp-server start::::::::::::::crt_ovs_sw5.sh::::::::::::::#!/bin/bashovs-vsctl add-br sw5ip link set up dev sw5ip addr add 192.211.39.1/24 dev sw5ip route replace 192.211.39.0/24 dev sw5ifconfig sw5 192.211.39.1 netmask 255.255.255.0# INTIF="sw3"# EXTIF="wlan0"# echo 1 > /proc/sys/net/ipv4/ip_forward# clear existing iptable rules, set a default policy# iptables -P INPUT ACCEPT# iptables -F INPUT # iptables -P OUTPUT ACCEPT# iptables -F OUTPUT # iptables -P FORWARD DROP# iptables -F FORWARD # iptables -t nat -F# set forwarding and nat rules# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE# service isc-dhcp-server start::::::::::::::crt_ovs_sw6.sh::::::::::::::#!/bin/bashovs-vsctl add-br sw6ip link set up dev sw6ip addr add 192.212.39.1/24 dev sw6ip route replace 192.212.39.0/24 dev sw6ifconfig sw6 192.212.39.1 netmask 255.255.255.0# INTIF="sw3"# EXTIF="wlan0"# echo 1 > /proc/sys/net/ipv4/ip_forward# clear existing iptable rules, set a default policy# iptables -P INPUT ACCEPT# iptables -F INPUT # iptables -P OUTPUT ACCEPT# iptables -F OUTPUT # iptables -P FORWARD DROP# iptables -F FORWARD # iptables -t nat -F# set forwarding and nat rules# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE# service isc-dhcp-server start::::::::::::::crt_ovs_sw7.sh::::::::::::::#!/bin/bashovs-vsctl add-br sw7ip link set up dev sw7ip addr add 192.213.39.1/24 dev sw7ip route replace 192.213.39.0/24 dev sw7ifconfig sw7 192.213.39.1 netmask 255.255.255.0# INTIF="sw3"# EXTIF="wlan0"# echo 1 > /proc/sys/net/ipv4/ip_forward# clear existing iptable rules, set a default policy# iptables -P INPUT ACCEPT# iptables -F INPUT # iptables -P OUTPUT ACCEPT# iptables -F OUTPUT # iptables -P FORWARD DROP# iptables -F FORWARD # iptables -t nat -F# set forwarding and nat rules# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE# service isc-dhcp-server start::::::::::::::crt_ovs_sw8.sh::::::::::::::#!/bin/bashovs-vsctl add-br sw8ip link set up dev sw8ip addr add 172.220.40.1/24 dev sw8ip route replace 172.220.40.0/24 dev sw8ifconfig sw8 172.220.40.1 netmask 255.255.255.0# INTIF="sw3"# EXTIF="wlan0"# echo 1 > /proc/sys/net/ipv4/ip_forward# clear existing iptable rules, set a default policy# iptables -P INPUT ACCEPT# iptables -F INPUT # iptables -P OUTPUT ACCEPT# iptables -F OUTPUT # iptables -P FORWARD DROP# iptables -F FORWARD # iptables -t nat -F# set forwarding and nat rules# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE# service isc-dhcp-server start::::::::::::::crt_ovs_sw9.sh::::::::::::::#!/bin/bashovs-vsctl add-br sw9ip link set up dev sw9ip addr add 172.221.40.1/24 dev sw9ip route replace 172.221.40.0/24 dev sw9ifconfig sw9 172.221.40.1 netmask 255.255.255.0# INTIF="sw3"# EXTIF="wlan0"# echo 1 > /proc/sys/net/ipv4/ip_forward# clear existing iptable rules, set a default policy# iptables -P INPUT ACCEPT# iptables -F INPUT # iptables -P OUTPUT ACCEPT# iptables -F OUTPUT # iptables -P FORWARD DROP# iptables -F FORWARD # iptables -t nat -F# set forwarding and nat rules# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE# service isc-dhcp-server start::::::::::::::crt_ovs_sx1.sh::::::::::::::#!/bin/bash# Requires use of Upstart Script /etc/init/my-network-up.conf to ensure interfaces are up before running.sleep 10 tunctl -t a1tunctl -t a2tunctl -t a3tunctl -t a4tunctl -t a5#tunctl -t a6ip link set a1 upip link set a2 upip link set a3 upip link set a4 upip link set a5 up#ip link set a6 upovs-vsctl add-br sx1ovs-vsctl add-port sx1 a1ovs-vsctl add-port sx1 a2ovs-vsctl add-port sx1 a3ovs-vsctl add-port sx1 a4ovs-vsctl add-port sx1 a5#ovs-vsctl add-port sx1 a6ip link set up dev sx1ip addr add 10.207.29.1/24 dev sx1ip route replace 10.207.29.0/24 dev sx1ovs-vsctl set port sx1 trunks=11ovs-vsctl set port sx1 tag=11# GLS 20140825 Get active external interface dynamically at boot. Tested & works with {wlan0, eth0, bnep0} on Ubuntu 14.04.1 Desktop x86_64.# GLS 20140825 Interface "bnep0" is Blackberry Z30 OS10 Bluetooth Tether.### BEGIN Get Active EXTIF Dynamcially. ###function GetInterface{ifconfig | egrep -B1 'inet addr' | egrep -A1 'wlan|eth|bnep' | sed '$!N;s/\n/ /' | sed 's/ */ /g' | cut -f1,7 -d' ' | sed 's/ addr//' | head -1 | cut -f1 -d':'}function GetIP{ifconfig | egrep -B1 'inet addr' | egrep -A1 'wlan|eth|bnep' | sed '$!N;s/\n/ /' | sed 's/ */ /g' | cut -f1,7 -d' ' | sed 's/ addr//' | head -1 | cut -f2 -d':'}### END Get Active EXTIF Dynamically. ###echo ' IP: '$(GetIP)echo 'Interface: '$(GetInterface)INTIF="sx1"EXTIF=$(GetInterface)# EXTIF="wlan0"echo 1 > /proc/sys/net/ipv4/ip_forward# clear existing iptable rules, set a default policyiptables -P INPUT ACCEPTiptables -F INPUT iptables -P OUTPUT ACCEPTiptables -F OUTPUT iptables -P FORWARD DROPiptables -F FORWARD iptables -t nat -F# set forwarding and nat rulesiptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPTiptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPTiptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE# service isc-dhcp-server restart# service bind9 restartgstanden@vmem1:~/OpenvSwitch$ There are also scripts in the directories below which are needed when the LXC containers start, and when the LXC containers stop, to configure the Openvswitches for the LXC containers. Those scripts are located in the following directories as shown below. /etc/network/if-up.d/openvswitch/ /etc/network/if-down.d/openvswitch All these scripts are attached to this blog (see end of blog) and can be downloaded directly. They will need to be installed in their correct directories. These scripts are also listed below. Only the scripts for lxcora01 LXC container are shown, because the scripts for additional LXC containers can be generated from these scripts using another script (see below). These scripts shown below are the startup scripts. gstanden@vmem1:/etc/network/if-up.d/openvswitch$ more lxcora01-*::::::::::::::lxcora01-asm1-ifup-sw8::::::::::::::#!/bin/bashovsBr='sw8'sudo ovs-vsctl add-port ${ovsBr} $5sudo ovs-vsctl set port $5 tag=60sudo ovs-vsctl set port ${ovsBr} tag=60sudo ovs-vsctl set port ${ovsBr} trunks=60,70::::::::::::::lxcora01-asm2-ifup-sw9::::::::::::::#!/bin/bashovsBr='sw9'sudo ovs-vsctl add-port ${ovsBr} $5sudo ovs-vsctl set port $5 tag=70sudo ovs-vsctl set port ${ovsBr} tag=70sudo ovs-vsctl set port ${ovsBr} trunks=60,70::::::::::::::lxcora01-priv1-ifup-sw4::::::::::::::#!/bin/bashovsBr='sw4'sudo ovs-vsctl add-port ${ovsBr} $5sudo ovs-vsctl set port $5 tag=20sudo ovs-vsctl set port ${ovsBr} tag=20sudo ovs-vsctl set port ${ovsBr} trunks=20,30,40,50::::::::::::::lxcora01-priv2-ifup-sw5::::::::::::::#!/bin/bashovsBr='sw5'sudo ovs-vsctl add-port ${ovsBr} $5sudo ovs-vsctl set port $5 tag=30sudo ovs-vsctl set port ${ovsBr} tag=30sudo ovs-vsctl set port ${ovsBr} trunks=20,30,40,50::::::::::::::lxcora01-priv3-ifup-sw6::::::::::::::#!/bin/bashovsBr='sw6'sudo ovs-vsctl add-port ${ovsBr} $5sudo ovs-vsctl set port $5 tag=40sudo ovs-vsctl set port ${ovsBr} tag=40sudo ovs-vsctl set port ${ovsBr} trunks=20,30,40,50::::::::::::::lxcora01-priv4-ifup-sw7::::::::::::::#!/bin/bashovsBr='sw7'sudo ovs-vsctl add-port ${ovsBr} $5sudo ovs-vsctl set port $5 tag=50sudo ovs-vsctl set port ${ovsBr} tag=50sudo ovs-vsctl set port ${ovsBr} trunks=20,30,40,50::::::::::::::lxcora01-pub-ifup-sw1::::::::::::::#!/bin/bashovsBr='sw1'ovs-vsctl add-port ${ovsBr} $5ovs-vsctl set port $5 tag=10gstanden@vmem1:/etc/network/if-up.d/openvswitch$ There are also LXC container shutdown scripts which clean up the ports on the OpenvSwitches that the LXC containers create dynamically when they boot up. Those ports need to be removed, and the scripts which do that are shown below. Again, only the scripts for lxcora01 are shown. The scripts for additional LXC containers are generated by the script. Before running the script to create the additional scripts, be sure that the startup and shutdown scripts have first been installed to /etc/network/if-up.d/openvswitch/ /etc/network/if-down.d/openvswitch/ and then run the script to create the additional scripts (see below). gstanden@vmem1:/etc/network/if-down.d/openvswitch$ more lxcora01*::::::::::::::lxcora01-asm1-ifdown-sw8::::::::::::::#!/bin/bashovsBr='sw8'sudo ovs-vsctl del-port ${ovsBr} $5function CheckPortExists{sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'}PortExists=$(CheckPortExists)if [ "$PortExists" = "$5" ]thensudo ovs-vsctl del-port ${ovsBr} $5fi::::::::::::::lxcora01-asm2-ifdown-sw9::::::::::::::#!/bin/bashovsBr='sw9'sudo ovs-vsctl del-port ${ovsBr} $5function CheckPortExists{sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'}PortExists=$(CheckPortExists)if [ "$PortExists" = "$5" ]thensudo ovs-vsctl del-port ${ovsBr} $5fi::::::::::::::lxcora01-priv1-ifdown-sw4::::::::::::::#!/bin/bashovsBr='sw4'sudo ovs-vsctl del-port ${ovsBr} $5function CheckPortExists{sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'}PortExists=$(CheckPortExists)if [ "$PortExists" = "$5" ]thensudo ovs-vsctl del-port ${ovsBr} $5fi::::::::::::::lxcora01-priv2-ifdown-sw5::::::::::::::#!/bin/bashovsBr='sw5'sudo ovs-vsctl del-port ${ovsBr} $5function CheckPortExists{sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'}PortExists=$(CheckPortExists)if [ "$PortExists" = "$5" ]thensudo ovs-vsctl del-port ${ovsBr} $5fi::::::::::::::lxcora01-priv3-ifdown-sw6::::::::::::::#!/bin/bashovsBr='sw6'sudo ovs-vsctl del-port ${ovsBr} $5function CheckPortExists{sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'}PortExists=$(CheckPortExists)if [ "$PortExists" = "$5" ]thensudo ovs-vsctl del-port ${ovsBr} $5fi::::::::::::::lxcora01-priv4-ifdown-sw7::::::::::::::#!/bin/bashovsBr='sw7'sudo ovs-vsctl del-port ${ovsBr} $5function CheckPortExists{sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'}PortExists=$(CheckPortExists)if [ "$PortExists" = "$5" ]thensudo ovs-vsctl del-port ${ovsBr} $5fi::::::::::::::lxcora01-pub-ifdown-sw1::::::::::::::#!/bin/bashovsBr='sw1'sudo ovs-vsctl del-port ${ovsBr} $5function CheckPortExists{sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'}PortExists=$(CheckPortExists)if [ "$PortExists" = "$5" ]thensudo ovs-vsctl del-port ${ovsBr} $5figstanden@vmem1:/etc/network/if-down.d/openvswitch$Here is the script for generating the required scripts for additional LXC containers. To use this script, simply decide how many LXC containers will be in the RAC cluster (in this blog six LXC container nodes will comprise the 6-node RAC). The usage of the script is highlighted in bold below. The "prefix name" of the containers is "lxcora0" and the number of containers is a total of "6". This script will generate all the rest of the similar scripts. Alternatively, they can all be downloaded from the end of this blog. gstanden@vmem1:/etc/network/if-up.d/openvswitch$ more create-ovs-sw-files-v2.sh #! /bin/bash# Usage# sudo ./create-ovs-sw-files.sh ContainerBaseName ContainerCount# Example# sudo ./create-ovs-sw-files.sh lxcora0 6# Above example would create networking files for 6 LXC containers with names of the form {lxcora01, lxcora02, lxcora03,lxcora04 lxcora05,lxcora06} # Notes# The ContainerBaseName must be passed in at the command line and is the prefix of the LXC container names# The ContainerCount is the total number of LXC containers to be created# Run this script in the directory where the ovs files are located (e.g. /etc/network/if-up.d/openvswitch) export DirectoryName='/etc/network/if-up.d/openvswitch /etc/network/if-down.d/openvswitch'export ContainerBaseName=$1export ContainerCount=$2for j in $DirectoryNamedocd $jfunction GetFileNames {ls "$ContainerBaseName"1* | more | grep "$1" | sed 's/$/ /' | tr -d '\n'}FileName=$(GetFileNames)let n=2 for i in $FileName do while [ $n -le $ContainerCount ] do function NewFileNames { echo $i | sed "s/"$ContainerBaseName"1/$ContainerBaseName$n/" } NewFile=$(NewFileNames) cp $i $NewFile ((n=n+1)) done let n=2 donedonechmod 755 /etc/network/if-up.d/openvswitch/lxc*chmod 755 /etc/network/if-down.d/openvswitch/lxc*gstanden@vmem1:/etc/network/if-up.d/openvswitch$ Setting Up Multipath StorageAn /etc/multipath.conf file will be needed for the multipathing of the SCST LUNs on the Ubuntu host for the Oracle 6-node RAC. The SCST LUNs are presented to the Ubuntu host from the KVM SCST guest. Once the LUNs are configured on the Ubuntu host, they are presented to the LXC guests using a configuration in the config file of the LXC container. Two openswitches are used to create the multipathing so that actual multipathing (two paths) is achieved. This is shown further down below using the "multipath -ll -v2" command. The sessions connected to the Ubuntu host are shown below. gstanden@vmem1:~/Networking$ sudo iscsiadm -m sessiontcp: [10] 10.207.41.74:3260,1 iqn.2014-10.org.vmem1:oracle651.san.asm.lunstcp: [11] 10.207.40.74:3260,1 iqn.2014-10.org.vmem1:oracle651.san.asm.luns2tcp: [12] 10.207.41.74:3260,1 iqn.2014-10.org.vmem1:oracle651.san.asm.luns2tcp: [7] 10.207.40.74:3260,1 iqn.2014-08.org.vmem:oracle651.san.asm.lunstcp: [8] 10.207.41.74:3260,1 iqn.2014-08.org.vmem:oracle651.san.asm.lunstcp: [9] 10.207.40.74:3260,1 iqn.2014-10.org.vmem1:oracle651.san.asm.lunsgstanden@vmem1:~/Networking$ There are two storage networks defined on the Openvswitches {10.207.40.1 and 10.207.41.1} as indicated above and as shown below. Because there are two OpenvSwitches, there are two multipaths for the KVM guest SCST LUNS. gstanden@vmem1:~/Networking$ ifconfig sw2sw2 Link encap:Ethernet HWaddr 8a:2d:67:d9:e2:42 inet addr:10.207.40.1 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::882d:67ff:fed9:e242/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:13107477 errors:0 dropped:0 overruns:0 frame:0 TX packets:17591412 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:9309165790 (9.3 GB) TX bytes:17331031641 (17.3 GB)gstanden@vmem1:~/Networking$ ifconfig sw3sw3 Link encap:Ethernet HWaddr 46:e6:7e:58:14:4b inet addr:10.207.41.1 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::44e6:7eff:fe58:144b/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:6753516 errors:0 dropped:0 overruns:0 frame:0 TX packets:9382228 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:6196524340 (6.1 GB) TX bytes:8724771816 (8.7 GB)The initiatorname on the Ubuntu host can be customized. It comes with a default name. For this project it has been customized as shown below. This value matches the value in the /etc/scst.conf file as shown below. root@vmem1:/etc/iscsi# cat /etc/iscsi/initiatorname.iscsi ## DO NOT EDIT OR REMOVE THIS FILE!## If you remove this file, the iSCSI daemon will not start.## If you change the InitiatorName, existing access control lists## may reject this initiator. The InitiatorName must be unique## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.# InitiatorName=iqn.1993-08.org.debian:01:ba4381d4cce2InitiatorName=iqn.2014-09.org.vmem1:oracle.asm.lunsroot@vmem1:/etc/iscsi# The /etc/scst.conf file on the KVM SCST guest is shown below. Note how the initiatorname above matches in the below scst.conf file to the value in the /etc/iscsi/initiatorname.iscsi file above. This setting means that onlly that iscsi initiator can login to the groups associated with those iscsi targets and prevents the initiator from logging into unnecessary targets. [root@oracle651 ~]# cat /etc/scst.conf# Automatically generated by SCST Configurator v2.2.1.HANDLER vdisk_fileio { DEVICE AsmDat631_01 { filename /scst_oracle631/AsmDat631_01.img } DEVICE AsmDat631_02 { filename /scst_oracle631/AsmDat631_02.img } DEVICE AsmDat631_03 { filename /scst_oracle631/AsmDat631_03.img } DEVICE AsmDat631_04 { filename /scst_oracle631/AsmDat631_04.img } DEVICE AsmDat631_05 { filename /scst_oracle632/AsmDat631_05.img } DEVICE AsmDat631_06 { filename /scst_oracle632/AsmDat631_06.img } DEVICE AsmDat631_07 { filename /scst_oracle632/AsmDat631_07.img } DEVICE AsmDat631_08 { filename /scst_oracle632/AsmDat631_08.img } DEVICE AsmDat_01 { filename /scst_vmem1/AsmDat_01.img } DEVICE AsmDat_02 { filename /scst_vmem1/AsmDat_02.img } DEVICE AsmDat_03 { filename /scst_vmem1/AsmDat_03.img } DEVICE AsmFra_01 { filename /scst_vmem1/AsmFra_01.img } DEVICE AsmFra_02 { filename /scst_vmem1/AsmFra_02.img } DEVICE AsmFra_03 { filename /scst_vmem1/AsmFra_03.img } DEVICE AsmSlb_01 { filename /scst_vmem1/AsmSlb_01.img } DEVICE AsmSlb_02 { filename /scst_vmem1/AsmSlb_02.img } DEVICE AsmSlb_03 { filename /scst_vmem1/AsmSlb_03.img } DEVICE AsmSys_01 { filename /scst_vmem1/AsmSys_01.img } DEVICE AsmSys_02 { filename /scst_vmem1/AsmSys_02.img } DEVICE AsmSys_03 { filename /scst_vmem1/AsmSys_03.img } DEVICE AsmTst_01 { filename /scst_vmem1/AsmTst_01.img } DEVICE AsmTst_02 { filename /scst_vmem1/AsmTst_02.img } DEVICE AsmTst_03 { filename /scst_vmem1/AsmTst_03.img } DEVICE AsmUnd_01 { filename /scst_vmem1/AsmUnd_01.img } DEVICE AsmUnd_02 { filename /scst_vmem1/AsmUnd_02.img } DEVICE AsmUnd_03 { filename /scst_vmem1/AsmUnd_03.img }}TARGET_DRIVER iscsi { enabled 1 TARGET iqn.2014-10.org.vmem1:oracle651.san.asm.luns {# cpu_mask ff enabled 1 rel_tgt_id 1 GROUP vmem1 { LUN 0 AsmDat_02 LUN 1 AsmUnd_02 LUN 2 AsmFra_02 LUN 3 AsmSys_02 LUN 4 AsmSlb_02 INITIATOR iqn.2014-09.org.vmem1:oracle.asm.luns cpu_mask ff } } TARGET iqn.2014-10.org.vmem1:oracle651.san.asm.luns2 {# cpu_mask ff enabled 1 rel_tgt_id 2 GROUP vmem1 { LUN 0 AsmUnd_03 LUN 1 AsmSlb_03 LUN 2 AsmFra_03 LUN 3 AsmSys_03 LUN 4 AsmDat_03 INITIATOR iqn.2014-09.org.vmem1:oracle.asm.luns cpu_mask ff } } TARGET iqn.2014-08.org.vmem:oracle651.san.asm.luns {# cpu_mask ff enabled 1 rel_tgt_id 3 GROUP oracle631 { LUN 0 AsmDat631_01 LUN 1 AsmDat631_02 LUN 2 AsmDat631_03 LUN 3 AsmDat631_04 LUN 4 AsmDat631_05 LUN 5 AsmDat631_06 LUN 6 AsmDat631_07 LUN 7 AsmDat631_08 INITIATOR iqn.2014-08.org.vmem:oracle631.asm INITIATOR iqn.2014-09.org.vmem:lxcora3.asm cpu_mask ff } GROUP vmem1 { LUN 0 AsmUnd_01 LUN 1 AsmFra_01 LUN 2 AsmSlb_01 LUN 3 AsmSys_01 LUN 4 AsmDat_01 INITIATOR iqn.2014-09.org.vmem1:oracle.asm.luns cpu_mask ff } }}[root@oracle651 ~]# Looking at the KVM SCST guest shows that there are two targets and each target has associated groups. [root@oracle651 ~]# scstadmin -list_groupCollecting current configuration: done.Driver: iscsiTarget: iqn.2014-08.org.vmem:oracle651.san.asm.lunsDriver/target 'iscsi/iqn.2014-08.org.vmem:oracle651.san.asm.luns' has no associated LUNs.Group: oracle631Assigned LUNs: LUN Device ----------------- 0 AsmDat631_01 1 AsmDat631_02 2 AsmDat631_03 3 AsmDat631_04 4 AsmDat631_05 5 AsmDat631_06 6 AsmDat631_07 7 AsmDat631_08Assigned Initiators: Initiator ---------------------------------- iqn.2014-08.org.vmem:oracle631.asm iqn.2014-09.org.vmem:lxcora3.asmGroup: vmem1Assigned LUNs: LUN Device -------------- 0 AsmUnd_01 1 AsmFra_01 2 AsmSlb_01 3 AsmSys_01 4 AsmDat_01Assigned Initiators: Initiator ------------------------------------- iqn.2014-09.org.vmem1:oracle.asm.lunsDriver: iscsiTarget: iqn.2014-10.org.vmem1:oracle651.san.asm.lunsDriver/target 'iscsi/iqn.2014-10.org.vmem1:oracle651.san.asm.luns' has no associated LUNs.Group: vmem1Assigned LUNs: LUN Device -------------- 0 AsmDat_02 1 AsmUnd_02 2 AsmFra_02 3 AsmSys_02 4 AsmSlb_02Assigned Initiators: Initiator ------------------------------------- iqn.2014-09.org.vmem1:oracle.asm.lunsDriver: iscsiTarget: iqn.2014-10.org.vmem1:oracle651.san.asm.luns2Driver/target 'iscsi/iqn.2014-10.org.vmem1:oracle651.san.asm.luns2' has no associated LUNs.Group: vmem1Assigned LUNs: LUN Device -------------- 0 AsmUnd_03 1 AsmSlb_03 2 AsmFra_03 3 AsmSys_03 4 AsmDat_03Assigned Initiators: Initiator ------------------------------------- iqn.2014-09.org.vmem1:oracle.asm.lunsAll done.[root@oracle651 ~]# Once the SCST Linux SAN has been setup and the LUNs presented to the Ubuntu host, it is necessary to get the WWNs of the LUNs. There are several ways to do this as shown below. These are all ways to get the WWN, which is needed for editing the /etc/multipath.conf file. gstanden@vmem1:~/Networking$ sudo /lib/udev/scsi_id --whitelisted --device=/dev/dm-423832373561623531gstanden@vmem1:~/Networking$ sudo /lib/udev/scsi_id -g -u -d /dev/mapper/asm_disk_lxc1_systemdg_0123832373561623531gstanden@vmem1:~/Networking$ sudo /lib/udev/scsi_id --whitelisted --device=/dev/mapper/asm_disk_lxc1_systemdg_0123832373561623531gstanden@vmem1:~/Networking$ ssh root@lxcora01Last login: Sun Aug 2 12:42:53 2015 from 10.207.39.1[root@lxcora01 ~]# scsi_id -g -u -d /dev/mapper/asm_disk_lxc1_systemdg_0123832373561623531[root@lxcora01 ~]# exitlogoutConnection to lxcora01 closed.gstanden@vmem1:~/Networking$ The /etc/multipath.conf file for the Oracle 6-node RAC on LXC Linux Containers on Ubuntu host is shown below. gstanden@vmem1:~$ cat /etc/multipath.conf# Configuration file for device mapper multipath.# For a complete list of the default configuration values:# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults# For a list of configuration options with descriptions:# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotated# Begin specific Instructions for the Oracle 6-node RAC on LXC OEL 6.5 Linux Containers on Ubuntu Host OS Project# 1. Blacklist Setup# Run the following command to get the "device" blacklist information for LXC Ubuntu host OS harddrive.# udevadm info -a -p $(udevadm info -q path -n /dev/sda) | egrep 'ATTRS{rev}|ATTRS{model}|ATTRS{vendor}' | grep -v 0x# Output of udevadm command for OS hard drive is shown below. Note that "spaces matter ".# ATTRS{rev}=="SM30" <-- blacklist "revision"# ATTRS{model}=="ST1000LM014-1EJ1" <-- blacklist "product"# ATTRS{vendor}=="ATA " <-- blacklist "vendor"# Run the following command to get the "device" blacklist information for the Wintec 128Gb Flash ExpressCard.# udevadm info -a -p $(udevadm info -q path -n /dev/sdag) | egrep 'ATTRS{rev}|ATTRS{model}|ATTRS{vendor}' | grep -v 0x# Output of udevadm command for Wintec Flash Express Card is shown below. Note that "spaces matter ".# ATTRS{rev}=="18 " <-- blacklist "revision"# ATTRS{model}=="Wintec JM612 128" <-- blacklist "product"# ATTRS{vendor}=="ATA " <-- blacklist "vendor"# Run the following command to get the "wwid" blacklist information for your Ubuntu host hard drive.# sudo /lib/udev/scsi_id -g -u -d /dev/sda# Output of scsi_id command for OS harddrive is shown below.# "35000c500828b9d7e" <-- blacklist "wwid"# Output of scsi_id command for Wintec Flash Express Card is shown below.# sudo /lib/udev/scsi_id --whitelisted --device=/dev/sdag# Output of scsi_id command for Wintec Flash Express Card is shown below. Note that "spaces matter ".# "1ATA Wintec JM612 128G ExpressCard SSD WT612-128G2300000424" <-- blacklist "wwid"# Add the above blacklists below to blacklist these devices from multipath as shown below.# Handle any other devices to be blacklisted using similar commands.# Both devices are blacklisted using two methods: (1) by devices, and; (2) by wwid's.# 2. Additional Devices Setup# On the Lenovo W520 laptop being used here, for example, there is a Wintec JM612 128Gb Express Card flash device.# http://www.wintecind.com/OEM/SSD/MicroSATA/JM612.html# If there are external or plugin devices you can configure them in the multipath.conf also as shown below.# Output of below udevadm command is shown below for the "Wintec JM612 128Gb Express Card".# udevadm info -a -p $(udevadm info -q path -n /dev/sdag) | egrep 'ATTRS{rev}|ATTRS{model}|ATTRS{vendor}' | grep -v 0x# ATTRS{rev}=="18 " <-- device "revision"# ATTRS{model}=="Wintec JM612 128" <-- device "product"# ATTRS{vendor}=="ATA " <-- device "vendor"# Although Wintec Flash Express Card is blacklisted, it could be used as a multipath device, thus included as a device.# 3. SCST_FIO# Configure the SCST_FIO device as shown below using the stanza as shown.# The SCST_FIO is the file device driver that is used for file-backed SCST LUNs.# The file-backed SCST LUNs is used for the RAC LUNs in this project.# 4. SCST_BIO# Configure the SCST_BIO device as shown below using the stanza as shown.# Device-backed devices are not used in the Oracle RAC project but they could be used alternatively.# 5. WWID# There are various commands which can be used. Examples are shown below of ways to get the information needed.# Note that on Ubuntu the command location is "/lib/udev/scsi_id" as shown below.# All these commands show how the wwid values of the LUNs can be obtained. Note that all methods give the same result# 23832373561623531# sudo /lib/udev/scsi_id -g -u -d /dev/mapper/asm_disk_lxc1_systemdg_01# 23832373561623531# sudo /lib/udev/scsi_id --whitelisted --device=/dev/mapper/asm_disk_lxc1_systemdg_01# 23832373561623531# Once the LXC Linux container is configured the wwid can also be checked from the container as shown below.# gstanden@vmem1:~/Networking$ ssh root@lxcora01# Last login: Sun Aug 2 12:42:53 2015 from 10.207.39.1# [root@lxcora01 ~]# scsi_id -g -u -d /dev/mapper/asm_disk_lxc1_systemdg_01# 23832373561623531# Once all the wwid and device friendly names are decided (being sure to put correctly sized LUNs with the desired devices) complete the "multipaths" section.# End specific Instructions for the Oracle 6-node RAC on LXC OEL 6.5 Linux Containers on Ubuntu Host OS Project.# Begin multipath.conf fileblacklist { wwid "35000c500828b9d7e" wwid "1ATA Wintec JM612 128G ExpressCard SSD WT612-128G2300000424" devnode "sd[a]$" device { vendor "ATA" product "ST1000LM014-1EJ1" revision "SM30" } device { vendor "ATA" product "Wintec JM612 128" revision "18" }}defaults { user_friendly_names yes}devices { device { vendor "ATA" product "Wintec JM612 128" revision "18" path_grouping_policy group_by_serial getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n" path_checker tur path_selector "queue-length 0" retain_attached_hw_handler yes detect_prio yes failback immediate rr_weight uniform rr_min_io_rq 3 no_path_retry fail } device { vendor "SCST_FIO" product "asm*" revision "310" path_grouping_policy group_by_serial getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n" path_checker tur path_selector "queue-length 0" retain_attached_hw_handler yes detect_prio yes failback immediate rr_weight uniform rr_min_io_rq 3 no_path_retry fail } device { vendor "SCST_BIO" product "asm*" revision "221" path_grouping_policy group_by_serial getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n" path_checker tur path_selector "queue-length 0" retain_attached_hw_handler yes detect_prio yes failback immediate rr_weight uniform rr_min_io_rq 1000 no_path_retry fail }}multipaths { multipath { wwid "23936383239343134" alias asm_disk_lxc1_undo_01 } multipath { wwid "26132363533633864" alias asm_disk_lxc1_undo_02 } multipath { wwid "26231633761346661" alias asm_disk_lxc1_undo_03 } multipath { wwid "23832373561623531" alias asm_disk_lxc1_systemdg_01 } multipath { wwid "26135333039626266" alias asm_disk_lxc1_systemdg_02 } multipath { wwid "26236393230336338" alias asm_disk_lxc1_systemdg_03 } multipath { wwid "2626237633731622d" alias asm_disk_lxc1_data_01 } multipath { wwid "23366353036663832" alias asm_disk_lxc1_data_02 } multipath { wwid "23263663266376635" alias asm_disk_lxc1_data_03 } multipath { wwid "26138386663393765" alias asm_disk_lxc1_slob_01 } multipath { wwid "23963363836316537" alias asm_disk_lxc1_slob_02 } multipath { wwid "23866636166393930" alias asm_disk_lxc1_slob_03 } multipath { wwid "23738373764356130" alias asm_disk_lxc1_fra_01 } multipath { wwid "23463393037643339" alias asm_disk_lxc1_fra_02 } multipath { wwid "23566333265353465" alias asm_disk_lxc1_fra_03 } multipath { wwid "23863366361643430" alias asm_disk_lxc1_test_01 } multipath { wwid "26238386230356439" alias asm_disk_lxc1_test_02 } multipath { wwid "26162323939646165" alias asm_disk_lxc1_test_03 }}gstanden@vmem1:~$ Below, the new /etc/multipath.conf file is tested. Note that on this system the Oracle 6-node RAC is already running, so the output of the command "multipath -F" shows the Oracle multipath LUNS "in use" and does not affect them. Note that the blacklists are working as expected and that the both "/dev/mapper" and also "multipath -ll -v2" commands show only the Oracle RAC LUNS as intended. gstanden@vmem1:~$ sudo multipath -FAug 02 16:21:41 | asm_disk_lxc1_fra_03: map in useAug 02 16:21:41 | asm_disk_lxc1_undo_03: map in useAug 02 16:21:41 | asm_disk_lxc1_fra_02: map in useAug 02 16:21:41 | asm_disk_lxc1_slob_03: map in useAug 02 16:21:41 | asm_disk_lxc1_undo_02: map in useAug 02 16:21:41 | asm_disk_lxc1_fra_01: map in useAug 02 16:21:41 | asm_disk_lxc1_slob_02: map in useAug 02 16:21:41 | asm_disk_lxc1_data_03: map in useAug 02 16:21:41 | asm_disk_lxc1_undo_01: map in useAug 02 16:21:41 | asm_disk_lxc1_systemdg_03: map in useAug 02 16:21:41 | asm_disk_lxc1_slob_01: map in useAug 02 16:21:41 | asm_disk_lxc1_data_02: map in useAug 02 16:21:41 | asm_disk_lxc1_systemdg_02: map in useAug 02 16:21:41 | asm_disk_lxc1_data_01: map in useAug 02 16:21:41 | asm_disk_lxc1_systemdg_01: map in usegstanden@vmem1:~$ ls -lrt /dev/mappertotal 0crw------- 1 root root 10, 236 Jul 31 20:59 controlbrw-rw---- 1 1098 1100 252, 0 Jul 31 21:04 asm_disk_lxc1_fra_01brw-rw---- 1 1098 1100 252, 1 Jul 31 21:04 asm_disk_lxc1_undo_01brw-rw---- 1 1098 1100 252, 3 Jul 31 21:04 asm_disk_lxc1_data_01brw-rw---- 1 1098 1100 252, 2 Jul 31 21:04 asm_disk_lxc1_slob_01brw-rw---- 1 1098 1100 252, 5 Jul 31 21:04 asm_disk_lxc1_data_02brw-rw---- 1 1098 1100 252, 7 Jul 31 21:04 asm_disk_lxc1_slob_02brw-rw---- 1 1098 1100 252, 6 Jul 31 21:04 asm_disk_lxc1_fra_02brw-rw---- 1 1098 1100 252, 8 Jul 31 21:04 asm_disk_lxc1_undo_02brw-rw---- 1 1098 1100 252, 10 Jul 31 21:04 asm_disk_lxc1_fra_03brw-rw---- 1 1098 1100 252, 11 Jul 31 21:04 asm_disk_lxc1_slob_03brw-rw---- 1 1098 1100 252, 14 Jul 31 21:04 asm_disk_lxc1_data_03brw-rw---- 1 1098 1100 252, 13 Jul 31 21:04 asm_disk_lxc1_undo_03brw-rw---- 1 1098 1100 252, 9 Aug 2 16:21 asm_disk_lxc1_systemdg_03brw-rw---- 1 1098 1100 252, 12 Aug 2 16:21 asm_disk_lxc1_systemdg_02brw-rw---- 1 1098 1100 252, 4 Aug 2 16:21 asm_disk_lxc1_systemdg_01gstanden@vmem1:~$ sudo service multipath-tools restartgstanden@vmem1:~$ ls -lrt /dev/mappertotal 0crw------- 1 root root 10, 236 Jul 31 20:59 controlbrw-rw---- 1 1098 1100 252, 0 Jul 31 21:04 asm_disk_lxc1_fra_01brw-rw---- 1 1098 1100 252, 1 Jul 31 21:04 asm_disk_lxc1_undo_01brw-rw---- 1 1098 1100 252, 3 Jul 31 21:04 asm_disk_lxc1_data_01brw-rw---- 1 1098 1100 252, 2 Jul 31 21:04 asm_disk_lxc1_slob_01brw-rw---- 1 1098 1100 252, 5 Jul 31 21:04 asm_disk_lxc1_data_02brw-rw---- 1 1098 1100 252, 7 Jul 31 21:04 asm_disk_lxc1_slob_02brw-rw---- 1 1098 1100 252, 6 Jul 31 21:04 asm_disk_lxc1_fra_02brw-rw---- 1 1098 1100 252, 8 Jul 31 21:04 asm_disk_lxc1_undo_02brw-rw---- 1 1098 1100 252, 10 Jul 31 21:04 asm_disk_lxc1_fra_03brw-rw---- 1 1098 1100 252, 11 Jul 31 21:04 asm_disk_lxc1_slob_03brw-rw---- 1 1098 1100 252, 14 Jul 31 21:04 asm_disk_lxc1_data_03brw-rw---- 1 1098 1100 252, 13 Jul 31 21:04 asm_disk_lxc1_undo_03brw-rw---- 1 1098 1100 252, 9 Aug 2 16:22 asm_disk_lxc1_systemdg_03brw-rw---- 1 1098 1100 252, 4 Aug 2 16:22 asm_disk_lxc1_systemdg_01brw-rw---- 1 1098 1100 252, 12 Aug 2 16:22 asm_disk_lxc1_systemdg_02gstanden@vmem1:~$ sudo multipath -ll -v2asm_disk_lxc1_fra_03 (23566333265353465) dm-10 SCST_FIO,AsmFra_03size=10G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 16:0:0:2 sdx 65:112 active ready running`-+- policy='round-robin 0' prio=1 status=enabled `- 17:0:0:2 sdac 65:192 active ready runningasm_disk_lxc1_undo_03 (26231633761346661) dm-13 SCST_FIO,AsmUnd_03size=30G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 16:0:0:0 sdv 65:80 active ready running`-+- policy='round-robin 0' prio=1 status=enabled `- 17:0:0:0 sdaa 65:160 active ready runningasm_disk_lxc1_fra_02 (23463393037643339) dm-6 SCST_FIO,AsmFra_02size=10G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 15:0:0:2 sds 65:32 active ready running`-+- policy='round-robin 0' prio=1 status=enabled `- 14:0:0:2 sdn 8:208 active ready runningasm_disk_lxc1_slob_03 (23866636166393930) dm-11 SCST_FIO,AsmSlb_03size=30G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 16:0:0:1 sdw 65:96 active ready running`-+- policy='round-robin 0' prio=1 status=enabled `- 17:0:0:1 sdab 65:176 active ready runningasm_disk_lxc1_undo_02 (26132363533633864) dm-8 SCST_FIO,AsmUnd_02size=30G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 14:0:0:1 sdl 8:176 active ready running`-+- policy='round-robin 0' prio=1 status=enabled `- 15:0:0:1 sdr 65:16 active ready runningasm_disk_lxc1_fra_01 (23738373764356130) dm-0 SCST_FIO,AsmFra_01size=10G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 13:0:0:1 sde 8:64 active ready running`-+- policy='round-robin 0' prio=1 status=enabled `- 12:0:0:1 sdc 8:32 active ready runningasm_disk_lxc1_slob_02 (23963363836316537) dm-7 SCST_FIO,AsmSlb_02size=30G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 15:0:0:4 sdu 65:64 active ready running`-+- policy='round-robin 0' prio=1 status=enabled `- 14:0:0:4 sdp 8:240 active ready runningasm_disk_lxc1_data_03 (23263663266376635) dm-14 SCST_FIO,AsmDat_03size=30G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 16:0:0:4 sdz 65:144 active ready running`-+- policy='round-robin 0' prio=1 status=enabled `- 17:0:0:4 sdae 65:224 active ready runningasm_disk_lxc1_undo_01 (23936383239343134) dm-1 SCST_FIO,AsmUnd_01size=30G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 13:0:0:0 sdd 8:48 active ready running`-+- policy='round-robin 0' prio=1 status=enabled `- 12:0:0:0 sdb 8:16 active ready runningasm_disk_lxc1_systemdg_03 (26236393230336338) dm-9 SCST_FIO,AsmSys_02size=5.0G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 14:0:0:3 sdo 8:224 active ready running`-+- policy='round-robin 0' prio=1 status=enabled `- 15:0:0:3 sdt 65:48 active ready runningasm_disk_lxc1_slob_01 (26138386663393765) dm-2 SCST_FIO,AsmSlb_01size=30G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 13:0:0:2 sdf 8:80 active ready running`-+- policy='round-robin 0' prio=1 status=enabled `- 12:0:0:2 sdg 8:96 active ready runningasm_disk_lxc1_data_02 (23366353036663832) dm-5 SCST_FIO,AsmDat_02size=30G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 15:0:0:0 sdq 65:0 active ready running`-+- policy='round-robin 0' prio=1 status=enabled `- 14:0:0:0 sdk 8:160 active ready runningasm_disk_lxc1_systemdg_02 (26135333039626266) dm-12 SCST_FIO,AsmSys_03size=5.0G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 16:0:0:3 sdy 65:128 active ready running`-+- policy='round-robin 0' prio=1 status=enabled `- 17:0:0:3 sdad 65:208 active ready runningasm_disk_lxc1_data_01 (2626237633731622d) dm-3 SCST_FIO,AsmDat_01size=30G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 13:0:0:4 sdj 8:144 active ready running`-+- policy='round-robin 0' prio=1 status=enabled `- 12:0:0:4 sdm 8:192 active ready runningasm_disk_lxc1_systemdg_01 (23832373561623531) dm-4 SCST_FIO,AsmSys_01size=5.0G features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 13:0:0:3 sdi 8:128 active ready running`-+- policy='round-robin 0' prio=1 status=enabled `- 12:0:0:3 sdh 8:112 active ready runninggstanden@vmem1:~$ |
