This blog post is intended to be a complete, comprehensive step-by-step how-to for installing a 6-node Oracle 12c RAC GNS ASM Flex Cluster on Ubuntu 15.04 using Oracle Enterprise Linux 6.5 LXC Linux Containers and OpenvSwitch.
The guide that is being created here now is based on a fresh install of Ubuntu 15.04 Vivid Vervet 64-bit desktop edition running on a Lenovo W520 mobile workstation.
The way I did the LUN presentation to Oracle running in the LXC containers previously was to build a KVM Oracle Enterprise Linux (OEL) 6.5 guest and then compile SCST into the kernel of that KVM guest. Since SCST has to be compiled into the kernel, installing it on the base Ubuntu 15.04 OS presents the problem that the kernel will have to be recompiled every time there is an update, and with Ubuntu that is typically going to be very frequently. The downside is that KVM and SCST do not provide all the great of IO relatively speaking to what one can get in a Linux Container, and so all Oracle IO to the LUNs on the KVM guest running SCST has to go through KVM which will degrade IO performance to the Oracle DB measurably. A less than optimal solution from that point of view. I haven't got a particularly good solution at this point for a better mousetrap for presenting the LUNs other than using a KVM guest, because, the nice thing about KVM is that I can set the logical and physical sector size on the LUNs with KVM, and SCST also has a manual setting for 4K physical sector size LUNs, so this KVM-SCST combo allows this lab system to present both 512/512 and 4096/4096 and 512/4096 LUN configurations to Oracle so that the setup can be used to play with Oracle using 4K native LUNs and/or 4K native LUNs with 512 emulation mode.
This guide is also a de-facto recipe for creating an OpenvSwitch-based Ubuntu Linux [ desktop | laptop ] networking environment that accomplishes a number of key goals as enumerated below.
This set of procedures has been tested and built on a fresh install of the following Ubuntu distributions. Installation on matching Ubuntu installations which have been running for a few months or years and have been customized could possibly introduce variations of existing configuration that could cause the steps described in this blog to have results different from the desired and expected results, YMMV. This blog to reiterate assumes a fresh install of the following distributions, and has been built and tested successfully on all of the following distributions of Ubuntu Linux.
Ubuntu 14.04 64-bit desktop edition
Ubuntu 14.10 64-bit desktop edition
Ubuntu 15.04 64-bit desktop edition
Note, if installing Ubuntu 14.04 or 14.10 or 15.04 fresh for this work, do NOT accept the "download updates during install" option. The reason is because these updates contain a number of CVE kernel security updates, and this procedure uses the post install Ubuntu CVE updates to check and verify correct operation of Oracle Ksplice Uptrack, so, in other words, Oracle Ksplice will handle the post-install CVE kernel security updates. Once KSplice operation has been so verified, Ubuntu Software Updater can be used for updates going forward as normal. Oracle KSplice can be used for urgent CVE kernel security updates with zero downtime as needed.
This particular blog is for Ubuntu 15.04 64-bit desktop Vivid Vervet.
DEU
Dense elastici unit. What formerly was called a "VM" but which now, with the advent of Linux Containers, requires imho a new term which includes density and elasticity solutions such as LXC which do not use hardware virtualization and as such are not "machines" at all. So in this blog, a "DEU" will be used as the all-inclusive term to describe a single LXC Linux Container".
VM
A type of DEU which uses a hypervisor to virtualize (emulate using software) the hardware layers of an actual physical machine and provides what is often called a "guest" operating system.
LXC
The term LXC will be used interchangeably according to context to either mean the software called LXC which is a Linux Container software in the same category as OpenVZ and others, and in other contexts, LXC may be used to denote a specific single LXC DEU which is running on LXC software.
No changes to Ubuntu dnsmasq-base default network, i.e is a pure add-on networking overlay;
Does not require any changes to Ubuntu NetworkManager or to default Ubuntu networking;
Does not require any removal of Linux Bridge software;
No physical interfaces directly to the OpenvSwitch (iptables/NAT/masq for external address resolution);
Allows external interface switching on LXC containers are running with NO loss of www DNS resolution;
Automatic internet-connected interface (eth0, wlan0, bnep0) detection and connection to OpenvSwitch;
Uses OpenvSwitch as the networking solution for LXC, KVM and VirtualBox DEUs;
Uses bind9 for OpenvSwitch DNS;
Uses isc-dhcp-server for OpenvSwitch DHCP services;
Integrates DNS and DHCP to update DNS automatically when new DEUs are added;
Uses the built-in Ubuntu dnsmasq-base for Ubuntu default networking.
Because it's LXC, VT-d and VT-x are NOT needed so older laptops and desktops without VT-d/x should be able to run LXC
The installation of bind9 is constructed so as not to interfere with the default Ubuntu desktop; that is, both bind9 and dnsmasq-base coexist peacefully and successfully so that even with the OpenvSwitch and bind9 installed, all the features of Ubuntu NetworkManager are preserved, including use of wifi network manager, VPN manager, bluetooth, and in general ALL default and configurable features of Ubuntu Network Manager.
Synaptic Package Manager is not required, but has some very nice features and since old-timers to Ubuntu sometimes like it, so it can be installed. It was the GUI for package management in Ubuntu Linux desktop prior to the introduction of the "Ubuntu Software Manager" and so some folks like to have it in addtion to the Software Manager. Both tools can coexist, and synaptic can sometimes be very useful for some tasks.
gstanden@u1504:~$ sudo apt-get install synaptic
[sudo] password for gstanden:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
docbook-xml libept1.4.12 librarian0 libvte-2.90-9 libvte-2.90-common rarian-compat sgml-data
Suggested packages:
docbook docbook-dsssl docbook-xsl docbook-defguide perlsgml w3-recs opensp libxml2-utils dwww menu deborphan tasksel
The following NEW packages will be installed:
docbook-xml libept1.4.12 librarian0 libvte-2.90-9 libvte-2.90-common rarian-compat sgml-data synaptic
0 upgraded, 8 newly installed, 0 to remove and 53 not upgraded.
Need to get 2,413 kB of archives.
After this operation, 14.2 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://us.archive.ubuntu.com/ubuntu/ vivid/main sgml-data all 2.0.10 [173 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ vivid/main docbook-xml all 4.5-7.2 [336 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ vivid/main libept1.4.12 amd64 1.0.12.1 [125 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu/ vivid/main libvte-2.90-common all 1:0.36.3-1ubuntu2 [24.5 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu/ vivid/main libvte-2.90-9 amd64 1:0.36.3-1ubuntu2 [278 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu/ vivid/main librarian0 amd64 0.8.1-6 [50.4 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu/ vivid/main rarian-compat amd64 0.8.1-6 [55.7 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu/ vivid/universe synaptic amd64 0.81.3 [1,370 kB]
Fetched 2,413 kB in 6s (355 kB/s)
Selecting previously unselected package sgml-data.
(Reading database ... 173095 files and directories currently installed.)
Preparing to unpack .../sgml-data_2.0.10_all.deb ...
Unpacking sgml-data (2.0.10) ...
Selecting previously unselected package docbook-xml.
Preparing to unpack .../docbook-xml_4.5-7.2_all.deb ...
Unpacking docbook-xml (4.5-7.2) ...
Selecting previously unselected package libept1.4.12:amd64.
Preparing to unpack .../libept1.4.12_1.0.12.1_amd64.deb ...
Unpacking libept1.4.12:amd64 (1.0.12.1) ...
Selecting previously unselected package libvte-2.90-common.
Preparing to unpack .../libvte-2.90-common_1%3a0.36.3-1ubuntu2_all.deb ...
Unpacking libvte-2.90-common (1:0.36.3-1ubuntu2) ...
Selecting previously unselected package libvte-2.90-9.
Preparing to unpack .../libvte-2.90-9_1%3a0.36.3-1ubuntu2_amd64.deb ...
Unpacking libvte-2.90-9 (1:0.36.3-1ubuntu2) ...
Selecting previously unselected package librarian0.
Preparing to unpack .../librarian0_0.8.1-6_amd64.deb ...
Unpacking librarian0 (0.8.1-6) ...
Selecting previously unselected package rarian-compat.
Preparing to unpack .../rarian-compat_0.8.1-6_amd64.deb ...
Unpacking rarian-compat (0.8.1-6) ...
Selecting previously unselected package synaptic.
Preparing to unpack .../synaptic_0.81.3_amd64.deb ...
Unpacking synaptic (0.81.3) ...
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
Processing triggers for doc-base (0.10.6) ...
Scrollkeeper was installed, forcing re-registration of all documents.
Unregistering 35 doc-base files, re-registering 35 doc-base files...
Registering documents with scrollkeeper...
Processing triggers for gnome-menus (3.10.1-0ubuntu5) ...
Processing triggers for desktop-file-utils (0.22-1ubuntu3) ...
Processing triggers for bamfdaemon (0.5.1+15.04.20150202-0ubuntu1) ...
Rebuilding /usr/share/applications/bamf-2.index...
Processing triggers for mime-support (3.58ubuntu1) ...
Processing triggers for hicolor-icon-theme (0.14-0ubuntu1) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up sgml-data (2.0.10) ...
Setting up libept1.4.12:amd64 (1.0.12.1) ...
Setting up libvte-2.90-common (1:0.36.3-1ubuntu2) ...
Setting up libvte-2.90-9 (1:0.36.3-1ubuntu2) ...
Setting up librarian0 (0.8.1-6) ...
Setting up synaptic (0.81.3) ...
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
Setting up docbook-xml (4.5-7.2) ...
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
Setting up rarian-compat (0.8.1-6) ...
Processing triggers for libc-bin (2.21-0ubuntu4) ...
gstanden@u1504:~$
Oracle Ksplice is a free product for Ubuntu Linux which allows updates of kernel software with no downtime. Oracle Ksplice is not required for this project, but as Wim Coaekerts has pointed out here, Oracle Ksplice is likely to play a big role for organizations using Linux Containers, because Ksplice allows zero downtime kernel security patching. So it is included in this blog as part of this build, especially since Ksplice is free to install and use the service for Ubuntu Linux.
Oracle Ksplice will be a key part of any LXC Linux Container deployment because it allows all LXC Containers to continue running during security updates to the single kernel used by all LXC containers on the system.
Download Oracle KSplice Uptrack
The Oracle KSplice website already has support for Ubuntu 15.04 64-bit deskop Vivid Vervet. Not only that, but the install instructions at the KSplice website worked perfectly and were much simpler than what had to be done (in my experience anyway) before when I did this install on 14.04 and 14.10. The instructions for installing Oracle KSplice onto Ubuntu 15.04 are here at Oracle KSplice Website and I followed the instructions there verbatim and the install went perfectly according to those instructions no glitches. Click on the red text "Show Ubuntu Installation Instructions" and just follow them exactly.
My install screen shots for Oracle KSplice were as shown below. After clicking on the "Download" red button in the screenshot above, you get the download dialog on the Ubuntu desktop. Choose the "open with Ubuntu Software Center (default)" option as shown below.
The "Ubuntu Software Center" opens. Click the "Install" button at the upper right hand side of the screen below the black bar. The install of KSplice begins as shown below.
A dialog box opens to authenticate the installation, type in your password for your linux user, and then another box opens to accept the KSplice Uptrack Subscription Agreement. This is a free tool for Ubuntu Desktop users, so tick the agreement, and click "Forward" as shown below.
If any dialogs open up to authenticate, just use the Linux password that was set for your linux userid when Ubuntu was installed. The KSplice Uptrack Manager runs as shown below advising of security updates to the kernel that need to be applied. Click on "Install All Updates" as shown below.
Type in the user password for your Ubuntu linux account to authenticate KSplice Uptrack Manager as shown below.
Oracle KSplice Uptrack runs and then displays that all security kernel updates have been applied as shown below.
Run Oracle KSplice Uptrack regularly to check for kernel security updates to be applied with zero downtime (no reboot needed). This is the functionality that KSplice Uptrack technology provides: zero-downtime kernel security updates.
If this is a fresh install, then update Ubuntu software to get all most recent updates using the Software Updater app or "sudo apt-get update". If this is not a fresh install, optionally check for updates using the same steps.
Now install other required packages as shown below.
gstanden@u1504:~$ sudo apt-get install lxc uml-utilities openvswitch-switch
[sudo] password for gstanden:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
bridge-utils cloud-image-utils debootstrap distro-info distro-info-data euca2ools libaio1 libboost-thread1.55.0 liblxc1 librados2 librbd1 libseccomp2 lxc-templates
lxcfs openvswitch-common python-ndg-httpsclient python-requestbuilder python-requests python-setuptools python-urllib3 python3-lxc qemu-utils sharutils uidmap
Suggested packages:
shunit2 btrfs-tools lxctl qemu-user-static bsd-mailx mailx user-mode-linux
The following NEW packages will be installed:
bridge-utils cloud-image-utils debootstrap distro-info distro-info-data euca2ools libaio1 libboost-thread1.55.0 liblxc1 librados2 librbd1 libseccomp2 lxc
lxc-templates lxcfs openvswitch-common openvswitch-switch python-ndg-httpsclient python-requestbuilder python-requests python-setuptools python-urllib3 python3-lxc
qemu-utils sharutils uidmap uml-utilities
0 upgraded, 27 newly installed, 0 to remove and 0 not upgraded.
Need to get 7,180 kB of archives.
After this operation, 29.9 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://us.archive.ubuntu.com/ubuntu/ vivid/main libaio1 amd64 0.3.110-1 [6,454 B]
Get:2 http://us.archive.ubuntu.com/ubuntu/ vivid/main libboost-thread1.55.0 amd64 1.55.0+dfsg-3ubuntu2 [31.5 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ vivid/main libseccomp2 amd64 2.1.1-1 [26.3 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu/ vivid/main liblxc1 amd64 1.1.2-0ubuntu3 [175 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu/ vivid/main librados2 amd64 0.94.1-0ubuntu1 [1,550 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu/ vivid/main librbd1 amd64 0.94.1-0ubuntu1 [1,684 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu/ vivid/main python-urllib3 all 1.9.1-3 [49.1 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu/ vivid/main python-requests all 2.4.3-6 [193 kB]
Get:9 http://us.archive.ubuntu.com/ubuntu/ vivid/main python-requestbuilder all 0.2.3-1 [27.2 kB]
Get:10 http://us.archive.ubuntu.com/ubuntu/ vivid/main bridge-utils amd64 1.5-7ubuntu1 [29.1 kB]
Get:11 http://us.archive.ubuntu.com/ubuntu/ vivid-updates/main distro-info-data all 0.26ubuntu0.1 [4,056 B]
Get:12 http://us.archive.ubuntu.com/ubuntu/ vivid/main distro-info amd64 0.14 [20.1 kB]
Get:13 http://us.archive.ubuntu.com/ubuntu/ vivid/main python-setuptools all 12.2-1 [197 kB]
Get:14 http://us.archive.ubuntu.com/ubuntu/ vivid/main euca2ools all 3.1.0-1 [270 kB] Get:15 http://us.archive.ubuntu.com/ubuntu/ vivid/main python3-lxc amd64 1.1.2-0ubuntu3 [20.5 kB] Get:16 http://us.archive.ubuntu.com/ubuntu/ vivid/main lxc amd64 1.1.2-0ubuntu3 [520 kB] Get:17 http://us.archive.ubuntu.com/ubuntu/ vivid/main lxc-templates amd64 1.1.2-0ubuntu3 [63.8 kB] Get:18 http://us.archive.ubuntu.com/ubuntu/ vivid/main lxcfs amd64 0.7-0ubuntu4 [22.5 kB] Get:19 http://us.archive.ubuntu.com/ubuntu/ vivid/main python-ndg-httpsclient all 0.3.2-1ubuntu4 [20.8 kB] Get:20 http://us.archive.ubuntu.com/ubuntu/ vivid-updates/main qemu-utils amd64 1:2.2+dfsg-5expubuntu9.1 [445 kB] Get:21 http://us.archive.ubuntu.com/ubuntu/ vivid/main sharutils amd64 1:4.14-2 [146 kB] Get:22 http://us.archive.ubuntu.com/ubuntu/ vivid/main uidmap amd64 1:4.1.5.1-1.1ubuntu4 [63.3 kB] Get:23 http://us.archive.ubuntu.com/ubuntu/ vivid/main cloud-image-utils all 0.27-0ubuntu15 [25.5 kB] Get:24 http://us.archive.ubuntu.com/ubuntu/ vivid-updates/main debootstrap all 1.0.67ubuntu0.1 [35.5 kB] Get:25 http://us.archive.ubuntu.com/ubuntu/ vivid/main openvswitch-common amd64 2.3.1-0ubuntu1 [520 kB] Get:26 http://us.archive.ubuntu.com/ubuntu/ vivid/main openvswitch-switch amd64 2.3.1-0ubuntu1 [985 kB] Get:27 http://us.archive.ubuntu.com/ubuntu/ vivid/universe uml-utilities amd64 20070815-1.4 [49.4 kB] Fetched 7,180 kB in 9s (749 kB/s) Selecting previously unselected package libaio1:amd64.
(Reading database ... 203959 files and directories currently installed.)
Preparing to unpack .../libaio1_0.3.110-1_amd64.deb ...
Unpacking libaio1:amd64 (0.3.110-1) ...
Selecting previously unselected package libboost-thread1.55.0:amd64.
Preparing to unpack .../libboost-thread1.55.0_1.55.0+dfsg-3ubuntu2_amd64.deb ...
Unpacking libboost-thread1.55.0:amd64 (1.55.0+dfsg-3ubuntu2) ...
Selecting previously unselected package libseccomp2:amd64.
Preparing to unpack .../libseccomp2_2.1.1-1_amd64.deb ...
Unpacking libseccomp2:amd64 (2.1.1-1) ...
Selecting previously unselected package liblxc1.
Preparing to unpack .../liblxc1_1.1.2-0ubuntu3_amd64.deb ...
Unpacking liblxc1 (1.1.2-0ubuntu3) ...
Selecting previously unselected package librados2.
Preparing to unpack .../librados2_0.94.1-0ubuntu1_amd64.deb ...
Unpacking librados2 (0.94.1-0ubuntu1) ...
Selecting previously unselected package librbd1.
Preparing to unpack .../librbd1_0.94.1-0ubuntu1_amd64.deb ...
Unpacking librbd1 (0.94.1-0ubuntu1) ...
Selecting previously unselected package python-urllib3.
Preparing to unpack .../python-urllib3_1.9.1-3_all.deb ...
Unpacking python-urllib3 (1.9.1-3) ...
Selecting previously unselected package python-requests.
Preparing to unpack .../python-requests_2.4.3-6_all.deb ...
Unpacking python-requests (2.4.3-6) ...
Selecting previously unselected package python-requestbuilder.
Preparing to unpack .../python-requestbuilder_0.2.3-1_all.deb ...
Unpacking python-requestbuilder (0.2.3-1) ...
Selecting previously unselected package bridge-utils.
Preparing to unpack .../bridge-utils_1.5-7ubuntu1_amd64.deb ...
Unpacking bridge-utils (1.5-7ubuntu1) ...
Selecting previously unselected package distro-info-data.
Preparing to unpack .../distro-info-data_0.26ubuntu0.1_all.deb ...
Unpacking distro-info-data (0.26ubuntu0.1) ...
Selecting previously unselected package distro-info.
Preparing to unpack .../distro-info_0.14_amd64.deb ...
Unpacking distro-info (0.14) ...
Selecting previously unselected package python-setuptools.
Preparing to unpack .../python-setuptools_12.2-1_all.deb ...
Unpacking python-setuptools (12.2-1) ...
Selecting previously unselected package euca2ools.
Preparing to unpack .../euca2ools_3.1.0-1_all.deb ...
Unpacking euca2ools (3.1.0-1) ...
Selecting previously unselected package python3-lxc.
Preparing to unpack .../python3-lxc_1.1.2-0ubuntu3_amd64.deb ...
Unpacking python3-lxc (1.1.2-0ubuntu3) ...
Selecting previously unselected package lxc.
Preparing to unpack .../lxc_1.1.2-0ubuntu3_amd64.deb ...
Unpacking lxc (1.1.2-0ubuntu3) ...
Selecting previously unselected package lxc-templates.
Preparing to unpack .../lxc-templates_1.1.2-0ubuntu3_amd64.deb ...
Unpacking lxc-templates (1.1.2-0ubuntu3) ...
Selecting previously unselected package lxcfs.
Preparing to unpack .../lxcfs_0.7-0ubuntu4_amd64.deb ...
Unpacking lxcfs (0.7-0ubuntu4) ...
Selecting previously unselected package python-ndg-httpsclient.
Preparing to unpack .../python-ndg-httpsclient_0.3.2-1ubuntu4_all.deb ...
Unpacking python-ndg-httpsclient (0.3.2-1ubuntu4) ...
Selecting previously unselected package qemu-utils.
Preparing to unpack .../qemu-utils_1%3a2.2+dfsg-5expubuntu9.1_amd64.deb ...
Unpacking qemu-utils (1:2.2+dfsg-5expubuntu9.1) ...
Selecting previously unselected package sharutils.
Preparing to unpack .../sharutils_1%3a4.14-2_amd64.deb ...
Unpacking sharutils (1:4.14-2) ...
Selecting previously unselected package uidmap.
Preparing to unpack .../uidmap_1%3a4.1.5.1-1.1ubuntu4_amd64.deb ...
Unpacking uidmap (1:4.1.5.1-1.1ubuntu4) ...
Selecting previously unselected package cloud-image-utils.
Preparing to unpack .../cloud-image-utils_0.27-0ubuntu15_all.deb ...
Unpacking cloud-image-utils (0.27-0ubuntu15) ...
Selecting previously unselected package debootstrap.
Preparing to unpack .../debootstrap_1.0.67ubuntu0.1_all.deb ...
Unpacking debootstrap (1.0.67ubuntu0.1) ...
Selecting previously unselected package openvswitch-common.
Preparing to unpack .../openvswitch-common_2.3.1-0ubuntu1_amd64.deb ...
Unpacking openvswitch-common (2.3.1-0ubuntu1) ...
Selecting previously unselected package openvswitch-switch.
Preparing to unpack .../openvswitch-switch_2.3.1-0ubuntu1_amd64.deb ...
Unpacking openvswitch-switch (2.3.1-0ubuntu1) ...
Selecting previously unselected package uml-utilities.
Preparing to unpack .../uml-utilities_20070815-1.4_amd64.deb ...
Unpacking uml-utilities (20070815-1.4) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for ureadahead (0.100.0-19) ...
ureadahead will be reprofiled on next reboot
Processing triggers for systemd (219-7ubuntu5) ...
Processing triggers for install-info (5.2.0.dfsg.1-6) ...
Setting up libaio1:amd64 (0.3.110-1) ...
Setting up libboost-thread1.55.0:amd64 (1.55.0+dfsg-3ubuntu2) ...
Setting up libseccomp2:amd64 (2.1.1-1) ...
Setting up liblxc1 (1.1.2-0ubuntu3) ...
Setting up librados2 (0.94.1-0ubuntu1) ...
Setting up librbd1 (0.94.1-0ubuntu1) ...
Setting up python-urllib3 (1.9.1-3) ...
Setting up python-requests (2.4.3-6) ...
Setting up python-requestbuilder (0.2.3-1) ...
Setting up bridge-utils (1.5-7ubuntu1) ...
Setting up distro-info-data (0.26ubuntu0.1) ...
Setting up distro-info (0.14) ...
Setting up python-setuptools (12.2-1) ...
Setting up euca2ools (3.1.0-1) ...
Setting up python3-lxc (1.1.2-0ubuntu3) ...
Setting up lxc (1.1.2-0ubuntu3) ...
Setting up lxc dnsmasq configuration.
Setting up lxc-templates (1.1.2-0ubuntu3) ...
Setting up lxcfs (0.7-0ubuntu4) ...
Setting up python-ndg-httpsclient (0.3.2-1ubuntu4) ...
Setting up qemu-utils (1:2.2+dfsg-5expubuntu9.1) ...
Setting up sharutils (1:4.14-2) ...
Setting up uidmap (1:4.1.5.1-1.1ubuntu4) ...
Setting up cloud-image-utils (0.27-0ubuntu15) ...
Setting up debootstrap (1.0.67ubuntu0.1) ...
Setting up openvswitch-common (2.3.1-0ubuntu1) ...
Setting up openvswitch-switch (2.3.1-0ubuntu1) ...
openvswitch-nonetwork.service is a disabled or a static unit, not starting it.
Setting up uml-utilities (20070815-1.4) ...
Processing triggers for libc-bin (2.21-0ubuntu4) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (219-7ubuntu5) ...
gstanden@u1504:~$
Next install bind9 and isc-dhcp-utils as shown below.
gstanden@u1504:~$ sudo apt-get install bind9 bind9utils isc-dhcp-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
bind9-doc isc-dhcp-server-ldap
The following NEW packages will be installed:
bind9 bind9utils isc-dhcp-server
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 804 kB of archives.
After this operation, 2,663 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ vivid/main bind9utils amd64 1:9.9.5.dfsg-9 [147 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ vivid/main bind9 amd64 1:9.9.5.dfsg-9 [294 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ vivid-updates/main isc-dhcp-server amd64 4.3.1-5ubuntu2.1 [363 kB]
Fetched 804 kB in 2s (335 kB/s)
Preconfiguring packages ...
Selecting previously unselected package bind9utils.
(Reading database ... 205485 files and directories currently installed.)
Preparing to unpack .../bind9utils_1%3a9.9.5.dfsg-9_amd64.deb ...
Unpacking bind9utils (1:9.9.5.dfsg-9) ...
Selecting previously unselected package bind9.
Preparing to unpack .../bind9_1%3a9.9.5.dfsg-9_amd64.deb ...
Unpacking bind9 (1:9.9.5.dfsg-9) ...
Selecting previously unselected package isc-dhcp-server.
Preparing to unpack .../isc-dhcp-server_4.3.1-5ubuntu2.1_amd64.deb ...
Unpacking isc-dhcp-server (4.3.1-5ubuntu2.1) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for ufw (0.34~rc-0ubuntu5) ...
Processing triggers for systemd (219-7ubuntu5) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up bind9utils (1:9.9.5.dfsg-9) ...
Setting up bind9 (1:9.9.5.dfsg-9) ...
Adding group `bind' (GID 134) ...
Done.
Adding system user `bind' (UID 122) ...
Adding new user `bind' (UID 122) with group `bind' ...
Not creating home directory `/var/cache/bind'.
wrote key file "/etc/bind/rndc.key"
#
Setting up isc-dhcp-server (4.3.1-5ubuntu2.1) ...
Generating /etc/default/isc-dhcp-server...
Processing triggers for systemd (219-7ubuntu5) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for ufw (0.34~rc-0ubuntu5) ...
gstanden@u1504:~$
The scripts of the form "crt_ovs_*.sh" where * in {sw1, sw2, sw3, ... sw9, sx1 } in the following directory:
/home/gstanden/OpenvSwitch/
are the scripts that create and configure the OpenvSwitch switches. For this installation, those scripts are called at boot time after the network has started by the script
/etc/network/if-up.d/openvswitch-net
and the contents of the openvswitch-net file is shown below. This is the file that calls the files that create the openvswitch for this Oracle RAC installation. This script runs automatically at boot, but after the network has started. This 6-node Oracle RAC will run on this OpenvSwitch which is created by these scripts.
gstanden@vmem1:/etc/network/if-up.d$ more openvswitch-net
#!/bin/sh
# openvswitch-net - Start OpenvSwitch
#
# description "Start OpenvSwitch Switches"
# start on net-device-up
# task
# script
/home/gstanden/OpenvSwitch/crt_ovs_sx1.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sx1.log
/home/gstanden/OpenvSwitch/crt_ovs_sw1.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw1.log
/home/gstanden/OpenvSwitch/crt_ovs_sw2.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw2.log
/home/gstanden/OpenvSwitch/crt_ovs_sw3.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw3.log
/home/gstanden/OpenvSwitch/crt_ovs_sw4.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw4.log
/home/gstanden/OpenvSwitch/crt_ovs_sw5.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw5.log
/home/gstanden/OpenvSwitch/crt_ovs_sw6.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw6.log
/home/gstanden/OpenvSwitch/crt_ovs_sw7.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw7.log
/home/gstanden/OpenvSwitch/crt_ovs_sw8.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw8.log
/home/gstanden/OpenvSwitch/crt_ovs_sw9.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw9.log
# end script
gstanden@vmem1:/etc/network/if-up.d$
The scripts above which are called by the openvswitch-net script are shown below. They are also attached at the end of this blog and can be downloaded. These scripts are based on work that was published by Jean-Jacquest Sarton here Advanced Networking Page by Jean-Jacques Sarton . I would like to take the opportunity to thank Mr. Sarton very sincerely for publishing his work, without which the present work to run Oracle ASM RAC Flex Clusters on OpenvSwitch might not have progressed forward.
gstanden@vmem1:~/OpenvSwitch$ more crt_ovs_s*.sh
::::::::::::::
crt_ovs_sw1.sh
::::::::::::::
#!/bin/bash
# Requires use of Upstart Script /etc/init/my-network-up.conf to ensure interfaces are up before running.
tunctl -t s1
tunctl -t s2
tunctl -t s3
tunctl -t s4
tunctl -t s5
#tunctl -t s6
ip link set s1 up
ip link set s2 up
ip link set s3 up
ip link set s4 up
ip link set s5 up
#ip link set s6 up
ovs-vsctl add-br sw1
ovs-vsctl add-port sw1 s1
ovs-vsctl add-port sw1 s2
ovs-vsctl add-port sw1 s3
ovs-vsctl add-port sw1 s4
ovs-vsctl add-port sw1 s5
#ovs-vsctl add-port sw1 s6
ip link set up dev sw1
ip addr add 10.207.39.1/24 dev sw1
ip route replace 10.207.39.0/24 dev sw1
ovs-vsctl set port sw1 trunks=10
ovs-vsctl set port sw1 tag=10
# GLS 20140825 Get active external interface dynamically at boot. Tested & works with {wlan0, eth0, bnep0} on Ubuntu 14.04.1 Desktop x86_64.
# GLS 20140825 Interface "bnep0" is for example Blackberry Z30 OS10 Bluetooth Tether, or any smartphone bluetooth tether.
### BEGIN Get Active EXTIF Dynamcially. ###
function GetInterface
{
ifconfig | egrep -B1 'inet addr' | egrep -A1 'wlan|eth|bnep' | sed '$!N;s/\n/ /' | sed 's/ */ /g' | cut -f1,7 -d' ' | sed 's/ addr//' | head -1 | cut -f1 -d':'
}
function GetIP
{
ifconfig | egrep -B1 'inet addr' | egrep -A1 'wlan|eth|bnep' | sed '$!N;s/\n/ /' | sed 's/ */ /g' | cut -f1,7 -d' ' | sed 's/ addr//' | head -1 | cut -f2 -d':'
}
### END Get Active EXTIF Dynamically. ###
echo ' IP: '$(GetIP)
echo 'Interface: '$(GetInterface)
INTIF="sw1"
EXTIF=$(GetInterface)
# EXTIF="wlan0"
echo 1 > /proc/sys/net/ipv4/ip_forward
# clear existing iptable rules, set a default policy
iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -t nat -F
# set forwarding and nat rules
iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
service isc-dhcp-server start
service bind9 restart
::::::::::::::
crt_ovs_sw2.sh
::::::::::::::
#!/bin/bash
tunctl -t t1
tunctl -t t2
tunctl -t t3
tunctl -t t4
tunctl -t t5
ip link set t1 up
ip link set t2 up
ip link set t3 up
ip link set t4 up
ip link set t5 up
ovs-vsctl add-br sw2
ovs-vsctl add-port sw2 t1
ovs-vsctl add-port sw2 t2
ovs-vsctl add-port sw2 t3
ovs-vsctl add-port sw2 t4
ovs-vsctl add-port sw2 t5
ip link set up dev sw2
ip addr add 10.207.40.1/24 dev sw2
ip route replace 10.207.40.0/24 dev sw2
ifconfig sw2 10.207.40.1 netmask 255.255.255.0
ovs-vsctl set port sw2 tag=80
# INTIF="sw2"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward
# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F
# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
# service isc-dhcp-server start
::::::::::::::
crt_ovs_sw3.sh
::::::::::::::
#!/bin/bash
tunctl -t w1
tunctl -t w2
tunctl -t w3
tunctl -t w4
tunctl -t w5
ip link set w1 up
ip link set w2 up
ip link set w3 up
ip link set w4 up
ip link set w5 up
ovs-vsctl add-br sw3
ovs-vsctl add-port sw3 w1
ovs-vsctl add-port sw3 w2
ovs-vsctl add-port sw3 w3
ovs-vsctl add-port sw3 w4
ovs-vsctl add-port sw3 w5
ip link set up dev sw3
ip addr add 10.207.41.1/24 dev sw3
ip route replace 10.207.41.0/24 dev sw3
ifconfig sw3 10.207.41.1 netmask 255.255.255.0
ovs-vsctl set port sw3 tag=90
# INTIF="sw3"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward
# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F
# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
# service isc-dhcp-server start
::::::::::::::
crt_ovs_sw4.sh
::::::::::::::
#!/bin/bash
ovs-vsctl add-br sw4
ip link set up dev sw4
ip addr add 192.210.39.1/24 dev sw4
ip route replace 192.210.39.0/24 dev sw4
ifconfig sw4 192.210.39.1 netmask 255.255.255.0
# INTIF="sw3"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward
# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F
# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
# service isc-dhcp-server start
::::::::::::::
crt_ovs_sw5.sh
::::::::::::::
#!/bin/bash
ovs-vsctl add-br sw5
ip link set up dev sw5
ip addr add 192.211.39.1/24 dev sw5
ip route replace 192.211.39.0/24 dev sw5
ifconfig sw5 192.211.39.1 netmask 255.255.255.0
# INTIF="sw3"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward
# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F
# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
# service isc-dhcp-server start
::::::::::::::
crt_ovs_sw6.sh
::::::::::::::
#!/bin/bash
ovs-vsctl add-br sw6
ip link set up dev sw6
ip addr add 192.212.39.1/24 dev sw6
ip route replace 192.212.39.0/24 dev sw6
ifconfig sw6 192.212.39.1 netmask 255.255.255.0
# INTIF="sw3"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward
# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F
# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
# service isc-dhcp-server start
::::::::::::::
crt_ovs_sw7.sh
::::::::::::::
#!/bin/bash
ovs-vsctl add-br sw7
ip link set up dev sw7
ip addr add 192.213.39.1/24 dev sw7
ip route replace 192.213.39.0/24 dev sw7
ifconfig sw7 192.213.39.1 netmask 255.255.255.0
# INTIF="sw3"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward
# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F
# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
# service isc-dhcp-server start
::::::::::::::
crt_ovs_sw8.sh
::::::::::::::
#!/bin/bash
ovs-vsctl add-br sw8
ip link set up dev sw8
ip addr add 172.220.40.1/24 dev sw8
ip route replace 172.220.40.0/24 dev sw8
ifconfig sw8 172.220.40.1 netmask 255.255.255.0
# INTIF="sw3"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward
# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F
# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
# service isc-dhcp-server start
::::::::::::::
crt_ovs_sw9.sh
::::::::::::::
#!/bin/bash
ovs-vsctl add-br sw9
ip link set up dev sw9
ip addr add 172.221.40.1/24 dev sw9
ip route replace 172.221.40.0/24 dev sw9
ifconfig sw9 172.221.40.1 netmask 255.255.255.0
# INTIF="sw3"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward
# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F
# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
# service isc-dhcp-server start
::::::::::::::
crt_ovs_sx1.sh
::::::::::::::
#!/bin/bash
# Requires use of Upstart Script /etc/init/my-network-up.conf to ensure interfaces are up before running.
sleep 10
tunctl -t a1
tunctl -t a2
tunctl -t a3
tunctl -t a4
tunctl -t a5
#tunctl -t a6
ip link set a1 up
ip link set a2 up
ip link set a3 up
ip link set a4 up
ip link set a5 up
#ip link set a6 up
ovs-vsctl add-br sx1
ovs-vsctl add-port sx1 a1
ovs-vsctl add-port sx1 a2
ovs-vsctl add-port sx1 a3
ovs-vsctl add-port sx1 a4
ovs-vsctl add-port sx1 a5
#ovs-vsctl add-port sx1 a6
ip link set up dev sx1
ip addr add 10.207.29.1/24 dev sx1
ip route replace 10.207.29.0/24 dev sx1
ovs-vsctl set port sx1 trunks=11
ovs-vsctl set port sx1 tag=11
# GLS 20140825 Get active external interface dynamically at boot. Tested & works with {wlan0, eth0, bnep0} on Ubuntu 14.04.1 Desktop x86_64.
# GLS 20140825 Interface "bnep0" is Blackberry Z30 OS10 Bluetooth Tether.
### BEGIN Get Active EXTIF Dynamcially. ###
function GetInterface
{
ifconfig | egrep -B1 'inet addr' | egrep -A1 'wlan|eth|bnep' | sed '$!N;s/\n/ /' | sed 's/ */ /g' | cut -f1,7 -d' ' | sed 's/ addr//' | head -1 | cut -f1 -d':'
}
function GetIP
{
ifconfig | egrep -B1 'inet addr' | egrep -A1 'wlan|eth|bnep' | sed '$!N;s/\n/ /' | sed 's/ */ /g' | cut -f1,7 -d' ' | sed 's/ addr//' | head -1 | cut -f2 -d':'
}
### END Get Active EXTIF Dynamically. ###
echo ' IP: '$(GetIP)
echo 'Interface: '$(GetInterface)
INTIF="sx1"
EXTIF=$(GetInterface)
# EXTIF="wlan0"
echo 1 > /proc/sys/net/ipv4/ip_forward
# clear existing iptable rules, set a default policy
iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -t nat -F
# set forwarding and nat rules
iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
# service isc-dhcp-server restart
# service bind9 restart
gstanden@vmem1:~/OpenvSwitch$
There are also scripts in the directories below which are needed when the LXC containers start, and when the LXC containers stop, to configure the Openvswitches for the LXC containers. Those scripts are located in the following directories as shown below.
/etc/network/if-up.d/openvswitch/
/etc/network/if-down.d/openvswitch
All these scripts are attached to this blog (see end of blog) and can be downloaded directly. They will need to be installed in their correct directories.
These scripts are also listed below. Only the scripts for lxcora01 LXC container are shown, because the scripts for additional LXC containers can be generated from these scripts using another script (see below). These scripts shown below are the startup scripts.
gstanden@vmem1:/etc/network/if-up.d/openvswitch$ more lxcora01-*
::::::::::::::
lxcora01-asm1-ifup-sw8
::::::::::::::
#!/bin/bash
ovsBr='sw8'
sudo ovs-vsctl add-port ${ovsBr} $5
sudo ovs-vsctl set port $5 tag=60
sudo ovs-vsctl set port ${ovsBr} tag=60
sudo ovs-vsctl set port ${ovsBr} trunks=60,70
::::::::::::::
lxcora01-asm2-ifup-sw9
::::::::::::::
#!/bin/bash
ovsBr='sw9'
sudo ovs-vsctl add-port ${ovsBr} $5
sudo ovs-vsctl set port $5 tag=70
sudo ovs-vsctl set port ${ovsBr} tag=70
sudo ovs-vsctl set port ${ovsBr} trunks=60,70
::::::::::::::
lxcora01-priv1-ifup-sw4
::::::::::::::
#!/bin/bash
ovsBr='sw4'
sudo ovs-vsctl add-port ${ovsBr} $5
sudo ovs-vsctl set port $5 tag=20
sudo ovs-vsctl set port ${ovsBr} tag=20
sudo ovs-vsctl set port ${ovsBr} trunks=20,30,40,50
::::::::::::::
lxcora01-priv2-ifup-sw5
::::::::::::::
#!/bin/bash
ovsBr='sw5'
sudo ovs-vsctl add-port ${ovsBr} $5
sudo ovs-vsctl set port $5 tag=30
sudo ovs-vsctl set port ${ovsBr} tag=30
sudo ovs-vsctl set port ${ovsBr} trunks=20,30,40,50
::::::::::::::
lxcora01-priv3-ifup-sw6
::::::::::::::
#!/bin/bash
ovsBr='sw6'
sudo ovs-vsctl add-port ${ovsBr} $5
sudo ovs-vsctl set port $5 tag=40
sudo ovs-vsctl set port ${ovsBr} tag=40
sudo ovs-vsctl set port ${ovsBr} trunks=20,30,40,50
::::::::::::::
lxcora01-priv4-ifup-sw7
::::::::::::::
#!/bin/bash
ovsBr='sw7'
sudo ovs-vsctl add-port ${ovsBr} $5
sudo ovs-vsctl set port $5 tag=50
sudo ovs-vsctl set port ${ovsBr} tag=50
sudo ovs-vsctl set port ${ovsBr} trunks=20,30,40,50
::::::::::::::
lxcora01-pub-ifup-sw1
::::::::::::::
#!/bin/bash
ovsBr='sw1'
ovs-vsctl add-port ${ovsBr} $5
ovs-vsctl set port $5 tag=10
gstanden@vmem1:/etc/network/if-up.d/openvswitch$
There are also LXC container shutdown scripts which clean up the ports on the OpenvSwitches that the LXC containers create dynamically when they boot up. Those ports need to be removed, and the scripts which do that are shown below. Again, only the scripts for lxcora01 are shown. The scripts for additional LXC containers are generated by the script.
Before running the script to create the additional scripts, be sure that the startup and shutdown scripts have first been installed to
/etc/network/if-up.d/openvswitch/
/etc/network/if-down.d/openvswitch/
and then run the script to create the additional scripts (see below).
gstanden@vmem1:/etc/network/if-down.d/openvswitch$ more lxcora01*
::::::::::::::
lxcora01-asm1-ifdown-sw8
::::::::::::::
#!/bin/bash
ovsBr='sw8'
sudo ovs-vsctl del-port ${ovsBr} $5
function CheckPortExists
{
sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'
}
PortExists=$(CheckPortExists)
if [ "$PortExists" = "$5" ]
then
sudo ovs-vsctl del-port ${ovsBr} $5
fi
::::::::::::::
lxcora01-asm2-ifdown-sw9
::::::::::::::
#!/bin/bash
ovsBr='sw9'
sudo ovs-vsctl del-port ${ovsBr} $5
function CheckPortExists
{
sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'
}
PortExists=$(CheckPortExists)
if [ "$PortExists" = "$5" ]
then
sudo ovs-vsctl del-port ${ovsBr} $5
fi
::::::::::::::
lxcora01-priv1-ifdown-sw4
::::::::::::::
#!/bin/bash
ovsBr='sw4'
sudo ovs-vsctl del-port ${ovsBr} $5
function CheckPortExists
{
sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'
}
PortExists=$(CheckPortExists)
if [ "$PortExists" = "$5" ]
then
sudo ovs-vsctl del-port ${ovsBr} $5
fi
::::::::::::::
lxcora01-priv2-ifdown-sw5
::::::::::::::
#!/bin/bash
ovsBr='sw5'
sudo ovs-vsctl del-port ${ovsBr} $5
function CheckPortExists
{
sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'
}
PortExists=$(CheckPortExists)
if [ "$PortExists" = "$5" ]
then
sudo ovs-vsctl del-port ${ovsBr} $5
fi
::::::::::::::
lxcora01-priv3-ifdown-sw6
::::::::::::::
#!/bin/bash
ovsBr='sw6'
sudo ovs-vsctl del-port ${ovsBr} $5
function CheckPortExists
{
sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'
}
PortExists=$(CheckPortExists)
if [ "$PortExists" = "$5" ]
then
sudo ovs-vsctl del-port ${ovsBr} $5
fi
::::::::::::::
lxcora01-priv4-ifdown-sw7
::::::::::::::
#!/bin/bash
ovsBr='sw7'
sudo ovs-vsctl del-port ${ovsBr} $5
function CheckPortExists
{
sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'
}
PortExists=$(CheckPortExists)
if [ "$PortExists" = "$5" ]
then
sudo ovs-vsctl del-port ${ovsBr} $5
fi
::::::::::::::
lxcora01-pub-ifdown-sw1
::::::::::::::
#!/bin/bash
ovsBr='sw1'
sudo ovs-vsctl del-port ${ovsBr} $5
function CheckPortExists
{
sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'
}
PortExists=$(CheckPortExists)
if [ "$PortExists" = "$5" ]
then
sudo ovs-vsctl del-port ${ovsBr} $5
fi
gstanden@vmem1:/etc/network/if-down.d/openvswitch$
Here is the script for generating the required scripts for additional LXC containers. To use this script, simply decide how many LXC containers will be in the RAC cluster (in this blog six LXC container nodes will comprise the 6-node RAC). The usage of the script is highlighted in bold below. The "prefix name" of the containers is "lxcora0" and the number of containers is a total of "6". This script will generate all the rest of the similar scripts. Alternatively, they can all be downloaded from the end of this blog.
gstanden@vmem1:/etc/network/if-up.d/openvswitch$ more create-ovs-sw-files-v2.sh
#! /bin/bash
# Usage
# sudo ./create-ovs-sw-files.sh ContainerBaseName ContainerCount
# Example
# sudo ./create-ovs-sw-files.sh lxcora0 6
# Above example would create networking files for 6 LXC containers with names of the form {lxcora01, lxcora02, lxcora03,lxcora04 lxcora05,lxcora06}
# Notes
# The ContainerBaseName must be passed in at the command line and is the prefix of the LXC container names
# The ContainerCount is the total number of LXC containers to be created
# Run this script in the directory where the ovs files are located (e.g. /etc/network/if-up.d/openvswitch)
export DirectoryName='/etc/network/if-up.d/openvswitch /etc/network/if-down.d/openvswitch'
export ContainerBaseName=$1
export ContainerCount=$2
for j in $DirectoryName
do
cd $j
function GetFileNames {
ls "$ContainerBaseName"1* | more | grep "$1" | sed 's/$/ /' | tr -d '\n'
}
FileName=$(GetFileNames)
let n=2
for i in $FileName
do
while [ $n -le $ContainerCount ]
do
function NewFileNames {
echo $i | sed "s/"$ContainerBaseName"1/$ContainerBaseName$n/"
}
NewFile=$(NewFileNames)
cp $i $NewFile
((n=n+1))
done
let n=2
done
done
chmod 755 /etc/network/if-up.d/openvswitch/lxc*
chmod 755 /etc/network/if-down.d/openvswitch/lxc*
gstanden@vmem1:/etc/network/if-up.d/openvswitch$
An /etc/multipath.conf file will be needed for the multipathing of the SCST LUNs on the Ubuntu host for the Oracle 6-node RAC. The SCST LUNs are presented to the Ubuntu host from the KVM SCST guest. Once the LUNs are configured on the Ubuntu host, they are presented to the LXC guests using a configuration in the config file of the LXC container. Two openswitches are used to create the multipathing so that actual multipathing (two paths) is achieved. This is shown further down below using the "multipath -ll -v2" command.
The sessions connected to the Ubuntu host are shown below.
gstanden@vmem1:~/Networking$ sudo iscsiadm -m session
tcp: [10] 10.207.41.74:3260,1 iqn.2014-10.org.vmem1:oracle651.san.asm.luns
tcp: [11] 10.207.40.74:3260,1 iqn.2014-10.org.vmem1:oracle651.san.asm.luns2
tcp: [12] 10.207.41.74:3260,1 iqn.2014-10.org.vmem1:oracle651.san.asm.luns2
tcp: [7] 10.207.40.74:3260,1 iqn.2014-08.org.vmem:oracle651.san.asm.luns
tcp: [8] 10.207.41.74:3260,1 iqn.2014-08.org.vmem:oracle651.san.asm.luns
tcp: [9] 10.207.40.74:3260,1 iqn.2014-10.org.vmem1:oracle651.san.asm.luns
gstanden@vmem1:~/Networking$
There are two storage networks defined on the Openvswitches {10.207.40.1 and 10.207.41.1} as indicated above and as shown below. Because there are two OpenvSwitches, there are two multipaths for the KVM guest SCST LUNS.
gstanden@vmem1:~/Networking$ ifconfig sw2
sw2 Link encap:Ethernet HWaddr 8a:2d:67:d9:e2:42
inet addr:10.207.40.1 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::882d:67ff:fed9:e242/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:13107477 errors:0 dropped:0 overruns:0 frame:0
TX packets:17591412 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9309165790 (9.3 GB) TX bytes:17331031641 (17.3 GB)
gstanden@vmem1:~/Networking$ ifconfig sw3
sw3 Link encap:Ethernet HWaddr 46:e6:7e:58:14:4b
inet addr:10.207.41.1 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::44e6:7eff:fe58:144b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6753516 errors:0 dropped:0 overruns:0 frame:0
TX packets:9382228 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:6196524340 (6.1 GB) TX bytes:8724771816 (8.7 GB)
The initiatorname on the Ubuntu host can be customized. It comes with a default name. For this project it has been customized as shown below. This value matches the value in the /etc/scst.conf file as shown below.
root@vmem1:/etc/iscsi# cat /etc/iscsi/initiatorname.iscsi
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator. The InitiatorName must be unique
## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.
# InitiatorName=iqn.1993-08.org.debian:01:ba4381d4cce2
InitiatorName=iqn.2014-09.org.vmem1:oracle.asm.luns
root@vmem1:/etc/iscsi#
The /etc/scst.conf file on the KVM SCST guest is shown below. Note how the initiatorname above matches in the below scst.conf file to the value in the /etc/iscsi/initiatorname.iscsi file above. This setting means that onlly that iscsi initiator can login to the groups associated with those iscsi targets and prevents the initiator from logging into unnecessary targets.
[root@oracle651 ~]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v2.2.1.
HANDLER vdisk_fileio {
DEVICE AsmDat631_01 {
filename /scst_oracle631/AsmDat631_01.img
}
DEVICE AsmDat631_02 {
filename /scst_oracle631/AsmDat631_02.img
}
DEVICE AsmDat631_03 {
filename /scst_oracle631/AsmDat631_03.img
}
DEVICE AsmDat631_04 {
filename /scst_oracle631/AsmDat631_04.img
}
DEVICE AsmDat631_05 {
filename /scst_oracle632/AsmDat631_05.img
}
DEVICE AsmDat631_06 {
filename /scst_oracle632/AsmDat631_06.img
}
DEVICE AsmDat631_07 {
filename /scst_oracle632/AsmDat631_07.img
}
DEVICE AsmDat631_08 {
filename /scst_oracle632/AsmDat631_08.img
}
DEVICE AsmDat_01 {
filename /scst_vmem1/AsmDat_01.img
}
DEVICE AsmDat_02 {
filename /scst_vmem1/AsmDat_02.img
}
DEVICE AsmDat_03 {
filename /scst_vmem1/AsmDat_03.img
}
DEVICE AsmFra_01 {
filename /scst_vmem1/AsmFra_01.img
}
DEVICE AsmFra_02 {
filename /scst_vmem1/AsmFra_02.img
}
DEVICE AsmFra_03 {
filename /scst_vmem1/AsmFra_03.img
}
DEVICE AsmSlb_01 {
filename /scst_vmem1/AsmSlb_01.img
}
DEVICE AsmSlb_02 {
filename /scst_vmem1/AsmSlb_02.img
}
DEVICE AsmSlb_03 {
filename /scst_vmem1/AsmSlb_03.img
}
DEVICE AsmSys_01 {
filename /scst_vmem1/AsmSys_01.img
}
DEVICE AsmSys_02 {
filename /scst_vmem1/AsmSys_02.img
}
DEVICE AsmSys_03 {
filename /scst_vmem1/AsmSys_03.img
}
DEVICE AsmTst_01 {
filename /scst_vmem1/AsmTst_01.img
}
DEVICE AsmTst_02 {
filename /scst_vmem1/AsmTst_02.img
}
DEVICE AsmTst_03 {
filename /scst_vmem1/AsmTst_03.img
}
DEVICE AsmUnd_01 {
filename /scst_vmem1/AsmUnd_01.img
}
DEVICE AsmUnd_02 {
filename /scst_vmem1/AsmUnd_02.img
}
DEVICE AsmUnd_03 {
filename /scst_vmem1/AsmUnd_03.img
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2014-10.org.vmem1:oracle651.san.asm.luns {
# cpu_mask ff
enabled 1
rel_tgt_id 1
GROUP vmem1 {
LUN 0 AsmDat_02
LUN 1 AsmUnd_02
LUN 2 AsmFra_02
LUN 3 AsmSys_02
LUN 4 AsmSlb_02
INITIATOR iqn.2014-09.org.vmem1:oracle.asm.luns
cpu_mask ff
}
}
TARGET iqn.2014-10.org.vmem1:oracle651.san.asm.luns2 {
# cpu_mask ff
enabled 1
rel_tgt_id 2
GROUP vmem1 {
LUN 0 AsmUnd_03
LUN 1 AsmSlb_03
LUN 2 AsmFra_03
LUN 3 AsmSys_03
LUN 4 AsmDat_03
INITIATOR iqn.2014-09.org.vmem1:oracle.asm.luns
cpu_mask ff
}
}
TARGET iqn.2014-08.org.vmem:oracle651.san.asm.luns {
# cpu_mask ff
enabled 1
rel_tgt_id 3
GROUP oracle631 {
LUN 0 AsmDat631_01
LUN 1 AsmDat631_02
LUN 2 AsmDat631_03
LUN 3 AsmDat631_04
LUN 4 AsmDat631_05
LUN 5 AsmDat631_06
LUN 6 AsmDat631_07
LUN 7 AsmDat631_08
INITIATOR iqn.2014-08.org.vmem:oracle631.asm
INITIATOR iqn.2014-09.org.vmem:lxcora3.asm
cpu_mask ff
}
GROUP vmem1 {
LUN 0 AsmUnd_01
LUN 1 AsmFra_01
LUN 2 AsmSlb_01
LUN 3 AsmSys_01
LUN 4 AsmDat_01
INITIATOR iqn.2014-09.org.vmem1:oracle.asm.luns
cpu_mask ff
}
}
}
[root@oracle651 ~]#
Looking at the KVM SCST guest shows that there are two targets and each target has associated groups.
[root@oracle651 ~]# scstadmin -list_group
Collecting current configuration: done.
Driver: iscsi
Target: iqn.2014-08.org.vmem:oracle651.san.asm.luns
Driver/target 'iscsi/iqn.2014-08.org.vmem:oracle651.san.asm.luns' has no associated LUNs.
Group: oracle631
Assigned LUNs:
LUN Device
-----------------
0 AsmDat631_01
1 AsmDat631_02
2 AsmDat631_03
3 AsmDat631_04
4 AsmDat631_05
5 AsmDat631_06
6 AsmDat631_07
7 AsmDat631_08
Assigned Initiators:
Initiator
----------------------------------
iqn.2014-08.org.vmem:oracle631.asm
iqn.2014-09.org.vmem:lxcora3.asm
Group: vmem1
Assigned LUNs:
LUN Device
--------------
0 AsmUnd_01
1 AsmFra_01
2 AsmSlb_01
3 AsmSys_01
4 AsmDat_01
Assigned Initiators:
Initiator
-------------------------------------
iqn.2014-09.org.vmem1:oracle.asm.luns
Driver: iscsi
Target: iqn.2014-10.org.vmem1:oracle651.san.asm.luns
Driver/target 'iscsi/iqn.2014-10.org.vmem1:oracle651.san.asm.luns' has no associated LUNs.
Group: vmem1
Assigned LUNs:
LUN Device
--------------
0 AsmDat_02
1 AsmUnd_02
2 AsmFra_02
3 AsmSys_02
4 AsmSlb_02
Assigned Initiators:
Initiator
-------------------------------------
iqn.2014-09.org.vmem1:oracle.asm.luns
Driver: iscsi
Target: iqn.2014-10.org.vmem1:oracle651.san.asm.luns2
Driver/target 'iscsi/iqn.2014-10.org.vmem1:oracle651.san.asm.luns2' has no associated LUNs.
Group: vmem1
Assigned LUNs:
LUN Device
--------------
0 AsmUnd_03
1 AsmSlb_03
2 AsmFra_03
3 AsmSys_03
4 AsmDat_03
Assigned Initiators:
Initiator
-------------------------------------
iqn.2014-09.org.vmem1:oracle.asm.luns
All done.
[root@oracle651 ~]#
Once the SCST Linux SAN has been setup and the LUNs presented to the Ubuntu host, it is necessary to get the WWNs of the LUNs. There are several ways to do this as shown below. These are all ways to get the WWN, which is needed for editing the /etc/multipath.conf file.
gstanden@vmem1:~/Networking$ sudo /lib/udev/scsi_id --whitelisted --device=/dev/dm-4
23832373561623531
gstanden@vmem1:~/Networking$ sudo /lib/udev/scsi_id -g -u -d /dev/mapper/asm_disk_lxc1_systemdg_01
23832373561623531
gstanden@vmem1:~/Networking$ sudo /lib/udev/scsi_id --whitelisted --device=/dev/mapper/asm_disk_lxc1_systemdg_01
23832373561623531
gstanden@vmem1:~/Networking$ ssh root@lxcora01
Last login: Sun Aug 2 12:42:53 2015 from 10.207.39.1
[root@lxcora01 ~]# scsi_id -g -u -d /dev/mapper/asm_disk_lxc1_systemdg_01
23832373561623531
[root@lxcora01 ~]# exit
logout
Connection to lxcora01 closed.
gstanden@vmem1:~/Networking$
The /etc/multipath.conf file for the Oracle 6-node RAC on LXC Linux Containers on Ubuntu host is shown below.
gstanden@vmem1:~$ cat /etc/multipath.conf
# Configuration file for device mapper multipath.
# For a complete list of the default configuration values:
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults
# For a list of configuration options with descriptions:
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotated
# Begin specific Instructions for the Oracle 6-node RAC on LXC OEL 6.5 Linux Containers on Ubuntu Host OS Project
# 1. Blacklist Setup
# Run the following command to get the "device" blacklist information for LXC Ubuntu host OS harddrive.
# udevadm info -a -p $(udevadm info -q path -n /dev/sda) | egrep 'ATTRS{rev}|ATTRS{model}|ATTRS{vendor}' | grep -v 0x
# Output of udevadm command for OS hard drive is shown below. Note that "spaces matter ".
# ATTRS{rev}=="SM30" <-- blacklist "revision"
# ATTRS{model}=="ST1000LM014-1EJ1" <-- blacklist "product"
# ATTRS{vendor}=="ATA " <-- blacklist "vendor"
# Run the following command to get the "device" blacklist information for the Wintec 128Gb Flash ExpressCard.
# udevadm info -a -p $(udevadm info -q path -n /dev/sdag) | egrep 'ATTRS{rev}|ATTRS{model}|ATTRS{vendor}' | grep -v 0x
# Output of udevadm command for Wintec Flash Express Card is shown below. Note that "spaces matter ".
# ATTRS{rev}=="18 " <-- blacklist "revision"
# ATTRS{model}=="Wintec JM612 128" <-- blacklist "product"
# ATTRS{vendor}=="ATA " <-- blacklist "vendor"
# Run the following command to get the "wwid" blacklist information for your Ubuntu host hard drive.
# sudo /lib/udev/scsi_id -g -u -d /dev/sda
# Output of scsi_id command for OS harddrive is shown below.
# "35000c500828b9d7e" <-- blacklist "wwid"
# Output of scsi_id command for Wintec Flash Express Card is shown below.
# sudo /lib/udev/scsi_id --whitelisted --device=/dev/sdag
# Output of scsi_id command for Wintec Flash Express Card is shown below. Note that "spaces matter ".
# "1ATA Wintec JM612 128G ExpressCard SSD WT612-128G2300000424" <-- blacklist "wwid"
# Add the above blacklists below to blacklist these devices from multipath as shown below.
# Handle any other devices to be blacklisted using similar commands.
# Both devices are blacklisted using two methods: (1) by devices, and; (2) by wwid's.
# 2. Additional Devices Setup
# On the Lenovo W520 laptop being used here, for example, there is a Wintec JM612 128Gb Express Card flash device.
# http://www.wintecind.com/OEM/SSD/MicroSATA/JM612.html
# If there are external or plugin devices you can configure them in the multipath.conf also as shown below.
# Output of below udevadm command is shown below for the "Wintec JM612 128Gb Express Card".
# udevadm info -a -p $(udevadm info -q path -n /dev/sdag) | egrep 'ATTRS{rev}|ATTRS{model}|ATTRS{vendor}' | grep -v 0x
# ATTRS{rev}=="18 " <-- device "revision"
# ATTRS{model}=="Wintec JM612 128" <-- device "product"
# ATTRS{vendor}=="ATA " <-- device "vendor"
# Although Wintec Flash Express Card is blacklisted, it could be used as a multipath device, thus included as a device.
# 3. SCST_FIO
# Configure the SCST_FIO device as shown below using the stanza as shown.
# The SCST_FIO is the file device driver that is used for file-backed SCST LUNs.
# The file-backed SCST LUNs is used for the RAC LUNs in this project.
# 4. SCST_BIO
# Configure the SCST_BIO device as shown below using the stanza as shown.
# The SCST_BIO is the block device driver that is used for device-backed SCST LUNs.
# Device-backed devices are not used in the Oracle RAC project but they could be used alternatively.
# 5. WWID
# The WWID of the Oracle RAC SCST LUNs must be obtained for each LUN to be used and multipath.conf updated with wwid values.
# There are various commands which can be used. Examples are shown below of ways to get the information needed.
# Note that on Ubuntu the command location is "/lib/udev/scsi_id" as shown below.
# All these commands show how the wwid values of the LUNs can be obtained. Note that all methods give the same result
# Output of the scsi_id command for SCST_FIO LUNS is shown below.
# sudo /lib/udev/scsi_id --whitelisted --device=/dev/dm-4
# 23832373561623531
# sudo /lib/udev/scsi_id -g -u -d /dev/mapper/asm_disk_lxc1_systemdg_01
# 23832373561623531
# sudo /lib/udev/scsi_id --whitelisted --device=/dev/mapper/asm_disk_lxc1_systemdg_01
# 23832373561623531
# Once the LXC Linux container is configured the wwid can also be checked from the container as shown below.
# gstanden@vmem1:~/Networking$ ssh root@lxcora01
# Last login: Sun Aug 2 12:42:53 2015 from 10.207.39.1
# [root@lxcora01 ~]# scsi_id -g -u -d /dev/mapper/asm_disk_lxc1_systemdg_01
# 23832373561623531
# Once all the wwid and device friendly names are decided (being sure to put correctly sized LUNs with the desired devices) complete the "multipaths" section.
# End specific Instructions for the Oracle 6-node RAC on LXC OEL 6.5 Linux Containers on Ubuntu Host OS Project.
# Begin multipath.conf file
blacklist {
wwid "35000c500828b9d7e"
wwid "1ATA Wintec JM612 128G ExpressCard SSD WT612-128G2300000424"
devnode "sd[a]$"
device {
vendor "ATA"
product "ST1000LM014-1EJ1"
revision "SM30"
}
device {
vendor "ATA"
product "Wintec JM612 128"
revision "18"
}
}
defaults {
user_friendly_names yes
}
devices {
device {
vendor "ATA"
product "Wintec JM612 128"
revision "18"
path_grouping_policy group_by_serial
getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
path_checker tur
path_selector "queue-length 0"
retain_attached_hw_handler yes
detect_prio yes
failback immediate
rr_weight uniform
rr_min_io_rq 3
no_path_retry fail
}
device {
vendor "SCST_FIO"
product "asm*"
revision "310"
path_grouping_policy group_by_serial
getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
path_checker tur
path_selector "queue-length 0"
retain_attached_hw_handler yes
detect_prio yes
failback immediate
rr_weight uniform
rr_min_io_rq 3
no_path_retry fail
}
device {
vendor "SCST_BIO"
product "asm*"
revision "221"
path_grouping_policy group_by_serial
getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
path_checker tur
path_selector "queue-length 0"
retain_attached_hw_handler yes
detect_prio yes
failback immediate
rr_weight uniform
rr_min_io_rq 1000
no_path_retry fail
}
}
multipaths {
multipath {
wwid "23936383239343134"
alias asm_disk_lxc1_undo_01
}
multipath {
wwid "26132363533633864"
alias asm_disk_lxc1_undo_02
}
multipath {
wwid "26231633761346661"
alias asm_disk_lxc1_undo_03
}
multipath {
wwid "23832373561623531"
alias asm_disk_lxc1_systemdg_01
}
multipath {
wwid "26135333039626266"
alias asm_disk_lxc1_systemdg_02
}
multipath {
wwid "26236393230336338"
alias asm_disk_lxc1_systemdg_03
}
multipath {
wwid "2626237633731622d"
alias asm_disk_lxc1_data_01
}
multipath {
wwid "23366353036663832"
alias asm_disk_lxc1_data_02
}
multipath {
wwid "23263663266376635"
alias asm_disk_lxc1_data_03
}
multipath {
wwid "26138386663393765"
alias asm_disk_lxc1_slob_01
}
multipath {
wwid "23963363836316537"
alias asm_disk_lxc1_slob_02
}
multipath {
wwid "23866636166393930"
alias asm_disk_lxc1_slob_03
}
multipath {
wwid "23738373764356130"
alias asm_disk_lxc1_fra_01
}
multipath {
wwid "23463393037643339"
alias asm_disk_lxc1_fra_02
}
multipath {
wwid "23566333265353465"
alias asm_disk_lxc1_fra_03
}
multipath {
wwid "23863366361643430"
alias asm_disk_lxc1_test_01
}
multipath {
wwid "26238386230356439"
alias asm_disk_lxc1_test_02
}
multipath {
wwid "26162323939646165"
alias asm_disk_lxc1_test_03
}
}
gstanden@vmem1:~$
Below, the new /etc/multipath.conf file is tested. Note that on this system the Oracle 6-node RAC is already running, so the output of the command "multipath -F" shows the Oracle multipath LUNS "in use" and does not affect them. Note that the blacklists are working as expected and that the both "/dev/mapper" and also "multipath -ll -v2" commands show only the Oracle RAC LUNS as intended.
gstanden@vmem1:~$ sudo multipath -F
Aug 02 16:21:41 | asm_disk_lxc1_fra_03: map in use
Aug 02 16:21:41 | asm_disk_lxc1_undo_03: map in use
Aug 02 16:21:41 | asm_disk_lxc1_fra_02: map in use
Aug 02 16:21:41 | asm_disk_lxc1_slob_03: map in use
Aug 02 16:21:41 | asm_disk_lxc1_undo_02: map in use
Aug 02 16:21:41 | asm_disk_lxc1_fra_01: map in use
Aug 02 16:21:41 | asm_disk_lxc1_slob_02: map in use
Aug 02 16:21:41 | asm_disk_lxc1_data_03: map in use
Aug 02 16:21:41 | asm_disk_lxc1_undo_01: map in use
Aug 02 16:21:41 | asm_disk_lxc1_systemdg_03: map in use
Aug 02 16:21:41 | asm_disk_lxc1_slob_01: map in use
Aug 02 16:21:41 | asm_disk_lxc1_data_02: map in use
Aug 02 16:21:41 | asm_disk_lxc1_systemdg_02: map in use
Aug 02 16:21:41 | asm_disk_lxc1_data_01: map in use
Aug 02 16:21:41 | asm_disk_lxc1_systemdg_01: map in use
gstanden@vmem1:~$ ls -lrt /dev/mapper
total 0
crw------- 1 root root 10, 236 Jul 31 20:59 control
brw-rw---- 1 1098 1100 252, 0 Jul 31 21:04 asm_disk_lxc1_fra_01
brw-rw---- 1 1098 1100 252, 1 Jul 31 21:04 asm_disk_lxc1_undo_01
brw-rw---- 1 1098 1100 252, 3 Jul 31 21:04 asm_disk_lxc1_data_01
brw-rw---- 1 1098 1100 252, 2 Jul 31 21:04 asm_disk_lxc1_slob_01
brw-rw---- 1 1098 1100 252, 5 Jul 31 21:04 asm_disk_lxc1_data_02
brw-rw---- 1 1098 1100 252, 7 Jul 31 21:04 asm_disk_lxc1_slob_02
brw-rw---- 1 1098 1100 252, 6 Jul 31 21:04 asm_disk_lxc1_fra_02
brw-rw---- 1 1098 1100 252, 8 Jul 31 21:04 asm_disk_lxc1_undo_02
brw-rw---- 1 1098 1100 252, 10 Jul 31 21:04 asm_disk_lxc1_fra_03
brw-rw---- 1 1098 1100 252, 11 Jul 31 21:04 asm_disk_lxc1_slob_03
brw-rw---- 1 1098 1100 252, 14 Jul 31 21:04 asm_disk_lxc1_data_03
brw-rw---- 1 1098 1100 252, 13 Jul 31 21:04 asm_disk_lxc1_undo_03
brw-rw---- 1 1098 1100 252, 9 Aug 2 16:21 asm_disk_lxc1_systemdg_03
brw-rw---- 1 1098 1100 252, 12 Aug 2 16:21 asm_disk_lxc1_systemdg_02
brw-rw---- 1 1098 1100 252, 4 Aug 2 16:21 asm_disk_lxc1_systemdg_01
gstanden@vmem1:~$ sudo service multipath-tools restart
gstanden@vmem1:~$ ls -lrt /dev/mapper
total 0
crw------- 1 root root 10, 236 Jul 31 20:59 control
brw-rw---- 1 1098 1100 252, 0 Jul 31 21:04 asm_disk_lxc1_fra_01
brw-rw---- 1 1098 1100 252, 1 Jul 31 21:04 asm_disk_lxc1_undo_01
brw-rw---- 1 1098 1100 252, 3 Jul 31 21:04 asm_disk_lxc1_data_01
brw-rw---- 1 1098 1100 252, 2 Jul 31 21:04 asm_disk_lxc1_slob_01
brw-rw---- 1 1098 1100 252, 5 Jul 31 21:04 asm_disk_lxc1_data_02
brw-rw---- 1 1098 1100 252, 7 Jul 31 21:04 asm_disk_lxc1_slob_02
brw-rw---- 1 1098 1100 252, 6 Jul 31 21:04 asm_disk_lxc1_fra_02
brw-rw---- 1 1098 1100 252, 8 Jul 31 21:04 asm_disk_lxc1_undo_02
brw-rw---- 1 1098 1100 252, 10 Jul 31 21:04 asm_disk_lxc1_fra_03
brw-rw---- 1 1098 1100 252, 11 Jul 31 21:04 asm_disk_lxc1_slob_03
brw-rw---- 1 1098 1100 252, 14 Jul 31 21:04 asm_disk_lxc1_data_03
brw-rw---- 1 1098 1100 252, 13 Jul 31 21:04 asm_disk_lxc1_undo_03
brw-rw---- 1 1098 1100 252, 9 Aug 2 16:22 asm_disk_lxc1_systemdg_03
brw-rw---- 1 1098 1100 252, 4 Aug 2 16:22 asm_disk_lxc1_systemdg_01
brw-rw---- 1 1098 1100 252, 12 Aug 2 16:22 asm_disk_lxc1_systemdg_02
gstanden@vmem1:~$ sudo multipath -ll -v2
asm_disk_lxc1_fra_03 (23566333265353465) dm-10 SCST_FIO,AsmFra_03
size=10G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 16:0:0:2 sdx 65:112 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 17:0:0:2 sdac 65:192 active ready running
asm_disk_lxc1_undo_03 (26231633761346661) dm-13 SCST_FIO,AsmUnd_03
size=30G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 16:0:0:0 sdv 65:80 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 17:0:0:0 sdaa 65:160 active ready running
asm_disk_lxc1_fra_02 (23463393037643339) dm-6 SCST_FIO,AsmFra_02
size=10G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 15:0:0:2 sds 65:32 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 14:0:0:2 sdn 8:208 active ready running
asm_disk_lxc1_slob_03 (23866636166393930) dm-11 SCST_FIO,AsmSlb_03
size=30G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 16:0:0:1 sdw 65:96 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 17:0:0:1 sdab 65:176 active ready running
asm_disk_lxc1_undo_02 (26132363533633864) dm-8 SCST_FIO,AsmUnd_02
size=30G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 14:0:0:1 sdl 8:176 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 15:0:0:1 sdr 65:16 active ready running
asm_disk_lxc1_fra_01 (23738373764356130) dm-0 SCST_FIO,AsmFra_01
size=10G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 13:0:0:1 sde 8:64 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 12:0:0:1 sdc 8:32 active ready running
asm_disk_lxc1_slob_02 (23963363836316537) dm-7 SCST_FIO,AsmSlb_02
size=30G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 15:0:0:4 sdu 65:64 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 14:0:0:4 sdp 8:240 active ready running
asm_disk_lxc1_data_03 (23263663266376635) dm-14 SCST_FIO,AsmDat_03
size=30G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 16:0:0:4 sdz 65:144 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 17:0:0:4 sdae 65:224 active ready running
asm_disk_lxc1_undo_01 (23936383239343134) dm-1 SCST_FIO,AsmUnd_01
size=30G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 13:0:0:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 12:0:0:0 sdb 8:16 active ready running
asm_disk_lxc1_systemdg_03 (26236393230336338) dm-9 SCST_FIO,AsmSys_02
size=5.0G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 14:0:0:3 sdo 8:224 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 15:0:0:3 sdt 65:48 active ready running
asm_disk_lxc1_slob_01 (26138386663393765) dm-2 SCST_FIO,AsmSlb_01
size=30G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 13:0:0:2 sdf 8:80 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 12:0:0:2 sdg 8:96 active ready running
asm_disk_lxc1_data_02 (23366353036663832) dm-5 SCST_FIO,AsmDat_02
size=30G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 15:0:0:0 sdq 65:0 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 14:0:0:0 sdk 8:160 active ready running
asm_disk_lxc1_systemdg_02 (26135333039626266) dm-12 SCST_FIO,AsmSys_03
size=5.0G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 16:0:0:3 sdy 65:128 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 17:0:0:3 sdad 65:208 active ready running
asm_disk_lxc1_data_01 (2626237633731622d) dm-3 SCST_FIO,AsmDat_01
size=30G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 13:0:0:4 sdj 8:144 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 12:0:0:4 sdm 8:192 active ready running
asm_disk_lxc1_systemdg_01 (23832373561623531) dm-4 SCST_FIO,AsmSys_01
size=5.0G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 13:0:0:3 sdi 8:128 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 12:0:0:3 sdh 8:112 active ready running
gstanden@vmem1:~$