Oracle 12c RAC ASM Flex Cluster on LXC Linux Containers Ubuntu 15.04

Contents

  1. 1 Updated Begin for Ubuntu 15.04 2015-05-16
    1. 1.1 Update End for Ubuntu 15.04 2015-05-16
  2. 2 Design Assumptions
    1. 2.1 Definitions
    2. 2.2 Design Features
  3. 3 Install Synaptic Package Manager (optional)
  4. 4 Install Oracle Ksplice (optional)
    1. 4.1 Download Oracle KSplice Uptrack
  5. 5 Update Fresh Installation (optional)
  6. 6 Install LXC, OpenvSwitch, and UML-Utilities Packages
  7. 7 Install DNS and DHCP packages
  8. 8 Install OpenvSwitch Configuration Scripts
    1. 8.1 Update Ubuntu 15.04 2015-05-16
    2. 8.2 Update End Ubuntu 15.04 2015-05-16
  9. 9 Setup DNS and DHCP for System
    1. 9.1 Backup DNS Default Configuration Files and Install Downloaded Versions
    2. 9.2 Backup DHCP Default Configuration Files
    3. 9.3 Update Begin Ubuntu 15.04 2015-05-16
    4. 9.4 Install Downloaded DHCP Files
    5. 9.5 Set Ownership and Permissions of DHCP Configuration Files
    6. 9.6 Install DNS Forward and Reverse Zone Configuration Files
    7. 9.7 Edit DNS Forward and Reverse Zone Files
    8. 9.8 Configure dnsmasq-base
    9. 9.9 Configure /etc/network/interfaces
  10. 10 Configure /etc/sysctl.conf
  11. 11 Install and Configure Required Apparmor Package
  12. 12 Install and configure openssh-server
  13. 13 Add Settings to /etc/sysctl.conf File
  14. 14 Restart DNS and DHCP to Verify Configuration
  15. 15 Create Oracle Enteprise Linux 6.5 LXC Linux Container
    1. 15.1 Install Ubuntu rpm and yum Packages
    2. 15.2 Create LXC Container
    3. 15.3 Configure LXC Container for OpenvSwitch Networking
    4. 15.4 Edit LXC Container config File for OpenvSwitch
    5. 15.5 Create Additional Required Networking Files
    6. 15.6 Configure dhclient.conf File
    7. 15.7 Verify Container Operation on DHCP and DNS with OpenvSwitch
    8. 15.8 Verify Container on OpenvSwitch Network
    9. 15.9 Verify Container /etc/resolv.conf Correct Dynamic Formatting
    10. 15.10 Install Package "bind-utils" into LXC Container
    11. 15.11 Run nslookup Tests in LXC Container to Verify DNS Resolutions
    12. 15.12 Run Various Status Commands on Container
    13. 15.13 Create File Management Links
    14. 15.14 Add HAIP and ASM Networks to Container Configuation File
  16. 16 Disable Transparent HugePages on Ubuntu Linux
  17. 17 Configure Users and Packages In LXC Container
  18. 18 Add the Following Lines to the end of /etc/security/limits.conf File
  19. 19 Add Oracle Configurations to /etc/sysctl.conf
  20. 20 Apply Updates to "/etc/sysctl.conf"
  21. 21 Set umask for "oracle" and "grid" Users
  22. 22 Set Password for "grid" Linux User
  23. 23 Edit "/etc/rc.local" File
  24. 24 Configure "/etc/sysconfig/ntpd" File
  25. 25 Install NFS Server on Ubuntu Base OS Host
  26. 26 Create ASM LUNs for Oracle Grid Infrastructure
  27. 27 Configure "/etc/exports" File on Ubuntu Base OS Host
  28. 28 Edit "/etc/apparmor.d/lxc/lxc-default" File for NFS
  29. 29 Configure /dev/shm in LXC Containers
  30. 30 Edit "/etc/fstab" File
  31. 31 Install cvuqdisk RPM
  32. 32 Configure Additional Linux Groups for Oracle 12c
  33. 33 Configure ZEROCONF
  34. 34 Verify SELINUX
  35. 35 Clone the Container to Create Second RAC Node
  36. 36 Note on Startup and Shutdown of LXC Oracle Linux 6.5 Containers
  37. 37 Run Cluster Verification Check
    1. 37.1 Explanation of Cluster Verification Exceptions
      1. 37.1.1 The /boot Mount Exception
      2. 37.1.2 The NTP Exception
      3. 37.1.3 The ASM Filter Driver Warning Exception
      4. 37.1.4 The ASM Shared Devices Exception
      5. 37.1.5 The rmem and wmem Kernel Parameter Exceptions
  38. 38 Create Additional LXC Containers for n-Node Oracle RAC
  39. 39 Create "if-up.d" and "if-down.d" Files for Cloned Nodes
  40. 40 SCST Discovery Issue
    1. 40.1 [root@oracle651 ~]# iscsiadm -m discovery -t sendtargets -p 10.207.40.76 10.207.40.76:3260,1 iqn.2015-07.org.vmem:oracle651.san.asm.luns10.207.39.76:3260,1 iqn.2015-07.org.vmem:oracle651.san.asm.luns10.207.41.76:3260,1 iqn.2015-07.org.vmem:oracle651.san.asm.luns[root@oracle651 ~]#
    2. 40.2 Run orainstRoot.sh on All Nodes
    3. 40.3 Create the /etc/network/if-up.d/openvswitch-net file
This blog post is intended to be a complete, comprehensive step-by-step how-to for installing Oracle 12c RAC GNS ASM Flex Cluster on Ubuntu 15.04 using Oracle Enterprise Linux 6.5 LXC Linux Containers and OpenvSwitch.  This is an update to the similar guide for Ubuntu 14.10 here.  The guide for 14.10 was copied to this blog page, and then the procedure for doing the whole setup for OpenvSwitch and Oracle RAC on Ubuntu 15.04 was documented in this blog by editing and updating the steps where necessary.

Updated Begin for Ubuntu 15.04 2015-05-16

I recently upgraded my laptop to Ubuntu 15.04 and there were a few issues that had to be resolved which had to do with SCST.  The cpu_mask issue is a reported bug in SCST and was probably the cause.  Bart Van Assche responded back to my SCST list server enquiry with the following information. 

Until June 2014 there was a bug in the SCST code for parsing cpu masks. You may have hit that bug.  Updating to SCST trunk r5596 or later or to the latest version of the  SCST 3.0 branch should resolve the issue related to parsing cpu masks.

The guide that is being created here now is based on a fresh install of Ubuntu 15.04 Vivid Vervet 64-bit desktop edition running on a Lenovo W520 mobile workstation.  Below there is a note about SCST.  The way I did the LUN presentation to Oracle previously was to build a KVM Oracle Enterprise Linux (OEL) 6.5 guest and then compile SCST into the kernel of that KVM guest.  Since SCST has to be compiled into the kernel, installing it on the base Ubuntu 15.04 OS presents the problem that the kernel will have to be recompiled every time there is an update, and with Ubuntu that is typically going to be very frequently.  The downside is that KVM and SCST do not provide all the great of IO relatively speaking to what one can get in a Linux Container, and so all Oracle IO to the LUNs on the KVM guest running SCST has to go through KVM which will degrade IO performance to the Oracle DB measurably.  A less than optimal solution from that point of view.  I haven't got a particularly good solution at this point for a better mousetrap for presenting the LUNs other than using a KVM guest, because, the nice thing about KVM is that I can set the logical and physical sector size on the LUNs with KVM, and SCST also has a manual setting for 4K physical sector size LUNs, so this KVM-SCST combo allows this lab system to present both 512/512 and 4096/4096 and 512/4096 LUN configurations to Oracle so that the setup can be used to play with Oracle using 4K native LUNs and/or 4K native LUNs with 512 emulation mode.

Update End for Ubuntu 15.04 2015-05-16

This guide is also a de-facto recipe for creating an OpenvSwitch-based Ubuntu Linux [ desktop | laptop ] networking environment that accomplishes a number of key goals as enumerated below.

Design Assumptions

This set of procedures has been tested and built on a fresh install of the following Ubuntu distributions.  Installation on matching Ubuntu installations which have been running for a few months or years and have been customized could possibly introduce variations of existing configuration that could cause the steps described in this blog to have results different from the desired and expected results, YMMV.   This blog to reiterate assumes a fresh install of the following distributions, and has been built and tested successfully on all of the following distributions of Ubuntu Linux.
  • Ubuntu 14.04 64-bit desktop edition
  • Ubuntu 14.10 64-bit desktop edition
  • Ubuntu 15.04 64-bit desktop edition

Note, if installing Ubuntu 14.04 or 14.10 or 15.04 fresh for this work, do NOT accept the "download updates during install" option.  The reason is because these updates contain a number of CVE kernel security updates, and this procedure uses the post install Ubuntu CVE updates to check and verify correct operation of Oracle Ksplice Uptrack, so, in other words, Oracle Ksplice will handle the post-install CVE kernel security updates.  Once KSplice operation has been so verified, Ubuntu Software Updater can be used for updates going forward as normal.  Oracle KSplice can be used for urgent CVE kernel security updates with zero downtime as needed. 

This particular blog is for Ubuntu 15.04 64-bit desktop Vivid Vervet.

Definitions

DEU   

Density and elasticity unit.  What formerly was called a "VM" but which now, with the advent of Linux Containers, requires imho a new term which includes density and elasticity solutions such as LXC which do not use hardware virtualization and as such are not "machines" at all.  So in this blog, a "DEU" will be used as the all-inclusive term to describe a single "VM" or "LXC Linux Container".

VM

A type of DEU which uses a hypervisor to virtualize (emulate using software) the hardware layers of an actual physical machine and provides what is often called a "guest" operating system.

LXC

The term LXC will be used interchangeably according to context to either mean the software called LXC which is a Linux Container software in the same category as OpenVZ and others, and in other contexts, LXC may be used to denote a specific single LXC DEU which is running on LXC software.

Design Features

  1. No changes to Ubuntu dnsmasq-base default network, i.e is a pure add-on networking overlay;
  2. Does not require any changes to Ubuntu NetworkManager or to default Ubuntu networking;
  3. Does not require any removal of Linux Bridge software;
  4. No physical interfaces directly to the OpenvSwitch (iptables/NAT/masq for external address resolution);
  5. Allows external interface switching on LXC containers are running with NO loss of www DNS resolution;
  6. Automatic internet-connected interface (eth0, wlan0, bnep0) detection and connection to OpenvSwitch;
  7. Uses OpenvSwitch as the networking solution for LXC, KVM and VirtualBox DEUs;
  8. Uses bind9 for OpenvSwitch DNS;
  9. Uses isc-dhcp-server for OpenvSwitch DHCP services;
  10. Integrates DNS and DHCP to update DNS automatically when new DEUs are added;
  11. Uses the built-in Ubuntu dnsmasq-base for Ubuntu default networking.
  12. Because it's LXC, VT-d and VT-x are NOT needed so older laptops and desktops without VT-d/x should be able to run LXC
The installation of bind9 is constructed so as not to interfere with the default Ubuntu desktop; that is, both bind9 and dnsmasq-base coexist peacefully and successfully so that even with the OpenvSwitch and bind9 installed, all the features of Ubuntu NetworkManager are preserved, including use of wifi network manager, VPN manager, bluetooth, and in general ALL default and configurable features of Ubuntu Network Manager. 

Install Synaptic Package Manager (optional)

Synaptic Package Manager is not required, but has some very nice features and since old-timers to Ubuntu sometimes like it, so it can be installed.  It was the GUI for package management in Ubuntu Linux desktop prior to the introduction of the "Ubuntu Software Manager" and so some folks like to have it in addtion to the Software Manager.  Both tools can coexist, and synaptic can sometimes be very useful for some tasks.

gstanden@u1504:~$ sudo apt-get install synaptic
[sudo] password for gstanden:
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  docbook-xml libept1.4.12 librarian0 libvte-2.90-9 libvte-2.90-common rarian-compat sgml-data
Suggested packages:
  docbook docbook-dsssl docbook-xsl docbook-defguide perlsgml w3-recs opensp libxml2-utils dwww menu deborphan tasksel
The following NEW packages will be installed:
  docbook-xml libept1.4.12 librarian0 libvte-2.90-9 libvte-2.90-common rarian-compat sgml-data synaptic
0 upgraded, 8 newly installed, 0 to remove and 53 not upgraded.
Need to get 2,413 kB of archives.
After this operation, 14.2 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://us.archive.ubuntu.com/ubuntu/ vivid/main sgml-data all 2.0.10 [173 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ vivid/main docbook-xml all 4.5-7.2 [336 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ vivid/main libept1.4.12 amd64 1.0.12.1 [125 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu/ vivid/main libvte-2.90-common all 1:0.36.3-1ubuntu2 [24.5 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu/ vivid/main libvte-2.90-9 amd64 1:0.36.3-1ubuntu2 [278 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu/ vivid/main librarian0 amd64 0.8.1-6 [50.4 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu/ vivid/main rarian-compat amd64 0.8.1-6 [55.7 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu/ vivid/universe synaptic amd64 0.81.3 [1,370 kB]
Fetched 2,413 kB in 6s (355 kB/s)                                                                                                                                        
Selecting previously unselected package sgml-data.
(Reading database ... 173095 files and directories currently installed.)
Preparing to unpack .../sgml-data_2.0.10_all.deb ...
Unpacking sgml-data (2.0.10) ...
Selecting previously unselected package docbook-xml.
Preparing to unpack .../docbook-xml_4.5-7.2_all.deb ...
Unpacking docbook-xml (4.5-7.2) ...
Selecting previously unselected package libept1.4.12:amd64.
Preparing to unpack .../libept1.4.12_1.0.12.1_amd64.deb ...
Unpacking libept1.4.12:amd64 (1.0.12.1) ...
Selecting previously unselected package libvte-2.90-common.
Preparing to unpack .../libvte-2.90-common_1%3a0.36.3-1ubuntu2_all.deb ...
Unpacking libvte-2.90-common (1:0.36.3-1ubuntu2) ...
Selecting previously unselected package libvte-2.90-9.
Preparing to unpack .../libvte-2.90-9_1%3a0.36.3-1ubuntu2_amd64.deb ...
Unpacking libvte-2.90-9 (1:0.36.3-1ubuntu2) ...
Selecting previously unselected package librarian0.
Preparing to unpack .../librarian0_0.8.1-6_amd64.deb ...
Unpacking librarian0 (0.8.1-6) ...
Selecting previously unselected package rarian-compat.
Preparing to unpack .../rarian-compat_0.8.1-6_amd64.deb ...
Unpacking rarian-compat (0.8.1-6) ...
Selecting previously unselected package synaptic.
Preparing to unpack .../synaptic_0.81.3_amd64.deb ...
Unpacking synaptic (0.81.3) ...
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
Processing triggers for doc-base (0.10.6) ...
Scrollkeeper was installed, forcing re-registration of all documents.
Unregistering 35 doc-base files, re-registering 35 doc-base files...
Registering documents with scrollkeeper...
Processing triggers for gnome-menus (3.10.1-0ubuntu5) ...
Processing triggers for desktop-file-utils (0.22-1ubuntu3) ...
Processing triggers for bamfdaemon (0.5.1+15.04.20150202-0ubuntu1) ...
Rebuilding /usr/share/applications/bamf-2.index...
Processing triggers for mime-support (3.58ubuntu1) ...
Processing triggers for hicolor-icon-theme (0.14-0ubuntu1) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up sgml-data (2.0.10) ...
Setting up libept1.4.12:amd64 (1.0.12.1) ...
Setting up libvte-2.90-common (1:0.36.3-1ubuntu2) ...
Setting up libvte-2.90-9 (1:0.36.3-1ubuntu2) ...
Setting up librarian0 (0.8.1-6) ...
Setting up synaptic (0.81.3) ...
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
Setting up docbook-xml (4.5-7.2) ...
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
Setting up rarian-compat (0.8.1-6) ...
Processing triggers for libc-bin (2.21-0ubuntu4) ...
gstanden@u1504:~$

Install Oracle Ksplice (optional)

Oracle Ksplice is a free product for Ubuntu Linux which allows updates of kernel software with no downtime.  Oracle Ksplice is not required for this project, but as Wim Coaekerts has pointed out here, Oracle Ksplice is likely to play a big role for organizations using Linux Containers, because Ksplice allows zero downtime kernel security patching.  So it is included in this blog as part of this build, especially since Ksplice is free to install and use the service for Ubuntu Linux.

Oracle Ksplice will be a key part of any LXC Linux Container deployment because it allows all LXC Containers to continue running during security updates to the single kernel used by all LXC containers on the system.

Download Oracle KSplice Uptrack

The Oracle KSplice website already has support for Ubuntu 15.04 64-bit deskop Vivid Vervet.  Not only that, but the install instructions at the KSplice website worked perfectly and were much simpler than what had to be done (in my experience anyway) before when I did this install on 14.04 and 14.10.  The instructions for installing Oracle KSplice onto Ubuntu 15.04 are here at Oracle KSplice Website and I followed the instructions there verbatim and the install went perfectly according to those instructions no glitches.  Click on the red text "Show Ubuntu Installation Instructions" and just follow them exactly.


My install screen shots for Oracle KSplice were as shown below.  After clicking on the "Download" red button in the screenshot above, you get the download dialog on the Ubuntu desktop.  Choose the "open with Ubuntu Software Center (default)" option as shown below.


The "Ubuntu Software Center" opens.  Click the "Install" button at the upper right hand side of the screen below the black bar.  The install of KSplice begins as shown below.


A dialog box opens to authenticate the installation, type in your password for your linux user, and then another box opens to accept the KSplice Uptrack Subscription Agreement.  This is a free tool for Ubuntu Desktop users, so tick the agreement, and click "Forward" as shown below.


If any dialogs open up to authenticate, just use the Linux password that was set for your linux userid when Ubuntu was installed.  The KSplice Uptrack Manager runs as shown below advising of security updates to the kernel that need to be applied. Click on "Install All Updates" as shown below.


Type in the user password for your Ubuntu linux account to authenticate KSplice Uptrack Manager as shown below.


Oracle KSplice Uptrack runs and then displays that all security kernel updates have been applied as shown below.


Run Oracle KSplice Uptrack regularly to check for kernel security updates to be applied with zero downtime (no reboot needed).  This is the functionality that KSplice Uptrack technology provides:  zero-downtime kernel security updates.

Update Fresh Installation (optional)

If this is a fresh install, then update Ubuntu software to get all most recent updates using the Software Updater app or "sudo apt-get update".  If this is not a fresh install, optionally check for updates using the same steps.

Install LXC, OpenvSwitch, and UML-Utilities Packages

Now install other required packages as shown below.

gstanden@u1504:~$ sudo apt-get install lxc uml-utilities openvswitch-switch
[sudo] password for gstanden:
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following extra packages will be installed:
  bridge-utils cloud-image-utils debootstrap distro-info distro-info-data euca2ools libaio1 libboost-thread1.55.0 liblxc1 librados2 librbd1 libseccomp2 lxc-templates
  lxcfs openvswitch-common python-ndg-httpsclient python-requestbuilder python-requests python-setuptools python-urllib3 python3-lxc qemu-utils sharutils uidmap
Suggested packages:
  shunit2 btrfs-tools lxctl qemu-user-static bsd-mailx mailx user-mode-linux
The following NEW packages will be installed:
  bridge-utils cloud-image-utils debootstrap distro-info distro-info-data euca2ools libaio1 libboost-thread1.55.0 liblxc1 librados2 librbd1 libseccomp2 lxc
  lxc-templates lxcfs openvswitch-common openvswitch-switch python-ndg-httpsclient python-requestbuilder python-requests python-setuptools python-urllib3 python3-lxc
  qemu-utils sharutils uidmap uml-utilities
0 upgraded, 27 newly installed, 0 to remove and 0 not upgraded.
Need to get 7,180 kB of archives.
After this operation, 29.9 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://us.archive.ubuntu.com/ubuntu/ vivid/main libaio1 amd64 0.3.110-1 [6,454 B]
Get:2 http://us.archive.ubuntu.com/ubuntu/ vivid/main libboost-thread1.55.0 amd64 1.55.0+dfsg-3ubuntu2 [31.5 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ vivid/main libseccomp2 amd64 2.1.1-1 [26.3 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu/ vivid/main liblxc1 amd64 1.1.2-0ubuntu3 [175 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu/ vivid/main librados2 amd64 0.94.1-0ubuntu1 [1,550 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu/ vivid/main librbd1 amd64 0.94.1-0ubuntu1 [1,684 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu/ vivid/main python-urllib3 all 1.9.1-3 [49.1 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu/ vivid/main python-requests all 2.4.3-6 [193 kB]
Get:9 http://us.archive.ubuntu.com/ubuntu/ vivid/main python-requestbuilder all 0.2.3-1 [27.2 kB]
Get:10 http://us.archive.ubuntu.com/ubuntu/ vivid/main bridge-utils amd64 1.5-7ubuntu1 [29.1 kB]
Get:11 http://us.archive.ubuntu.com/ubuntu/ vivid-updates/main distro-info-data all 0.26ubuntu0.1 [4,056 B]
Get:12 http://us.archive.ubuntu.com/ubuntu/ vivid/main distro-info amd64 0.14 [20.1 kB]
Get:13 http://us.archive.ubuntu.com/ubuntu/ vivid/main python-setuptools all 12.2-1 [197 kB]
Get:14 http://us.archive.ubuntu.com/ubuntu/ vivid/main euca2ools all 3.1.0-1 [270 kB]                                           Get:15 http://us.archive.ubuntu.com/ubuntu/ vivid/main python3-lxc amd64 1.1.2-0ubuntu3 [20.5 kB]                               Get:16 http://us.archive.ubuntu.com/ubuntu/ vivid/main lxc amd64 1.1.2-0ubuntu3 [520 kB]                                        Get:17 http://us.archive.ubuntu.com/ubuntu/ vivid/main lxc-templates amd64 1.1.2-0ubuntu3 [63.8 kB]                             Get:18 http://us.archive.ubuntu.com/ubuntu/ vivid/main lxcfs amd64 0.7-0ubuntu4 [22.5 kB]                                       Get:19 http://us.archive.ubuntu.com/ubuntu/ vivid/main python-ndg-httpsclient all 0.3.2-1ubuntu4 [20.8 kB]                 Get:20 http://us.archive.ubuntu.com/ubuntu/ vivid-updates/main qemu-utils amd64 1:2.2+dfsg-5expubuntu9.1 [445 kB]                Get:21 http://us.archive.ubuntu.com/ubuntu/ vivid/main sharutils amd64 1:4.14-2 [146 kB]                                   Get:22 http://us.archive.ubuntu.com/ubuntu/ vivid/main uidmap amd64 1:4.1.5.1-1.1ubuntu4 [63.3 kB]                               Get:23 http://us.archive.ubuntu.com/ubuntu/ vivid/main cloud-image-utils all 0.27-0ubuntu15 [25.5 kB]                            Get:24 http://us.archive.ubuntu.com/ubuntu/ vivid-updates/main debootstrap all 1.0.67ubuntu0.1 [35.5 kB]                         Get:25 http://us.archive.ubuntu.com/ubuntu/ vivid/main openvswitch-common amd64 2.3.1-0ubuntu1 [520 kB]                          Get:26 http://us.archive.ubuntu.com/ubuntu/ vivid/main openvswitch-switch amd64 2.3.1-0ubuntu1 [985 kB]                          Get:27 http://us.archive.ubuntu.com/ubuntu/ vivid/universe uml-utilities amd64 20070815-1.4 [49.4 kB]                            Fetched 7,180 kB in 9s (749 kB/s)                                                                                                                         Selecting previously unselected package libaio1:amd64.
(Reading database ... 203959 files and directories currently installed.)
Preparing to unpack .../libaio1_0.3.110-1_amd64.deb ...
Unpacking libaio1:amd64 (0.3.110-1) ...
Selecting previously unselected package libboost-thread1.55.0:amd64.
Preparing to unpack .../libboost-thread1.55.0_1.55.0+dfsg-3ubuntu2_amd64.deb ...
Unpacking libboost-thread1.55.0:amd64 (1.55.0+dfsg-3ubuntu2) ...
Selecting previously unselected package libseccomp2:amd64.
Preparing to unpack .../libseccomp2_2.1.1-1_amd64.deb ...
Unpacking libseccomp2:amd64 (2.1.1-1) ...
Selecting previously unselected package liblxc1.
Preparing to unpack .../liblxc1_1.1.2-0ubuntu3_amd64.deb ...
Unpacking liblxc1 (1.1.2-0ubuntu3) ...
Selecting previously unselected package librados2.
Preparing to unpack .../librados2_0.94.1-0ubuntu1_amd64.deb ...
Unpacking librados2 (0.94.1-0ubuntu1) ...
Selecting previously unselected package librbd1.
Preparing to unpack .../librbd1_0.94.1-0ubuntu1_amd64.deb ...
Unpacking librbd1 (0.94.1-0ubuntu1) ...
Selecting previously unselected package python-urllib3.
Preparing to unpack .../python-urllib3_1.9.1-3_all.deb ...
Unpacking python-urllib3 (1.9.1-3) ...
Selecting previously unselected package python-requests.
Preparing to unpack .../python-requests_2.4.3-6_all.deb ...
Unpacking python-requests (2.4.3-6) ...
Selecting previously unselected package python-requestbuilder.
Preparing to unpack .../python-requestbuilder_0.2.3-1_all.deb ...
Unpacking python-requestbuilder (0.2.3-1) ...
Selecting previously unselected package bridge-utils.
Preparing to unpack .../bridge-utils_1.5-7ubuntu1_amd64.deb ...
Unpacking bridge-utils (1.5-7ubuntu1) ...
Selecting previously unselected package distro-info-data.
Preparing to unpack .../distro-info-data_0.26ubuntu0.1_all.deb ...
Unpacking distro-info-data (0.26ubuntu0.1) ...
Selecting previously unselected package distro-info.
Preparing to unpack .../distro-info_0.14_amd64.deb ...
Unpacking distro-info (0.14) ...
Selecting previously unselected package python-setuptools.
Preparing to unpack .../python-setuptools_12.2-1_all.deb ...
Unpacking python-setuptools (12.2-1) ...
Selecting previously unselected package euca2ools.
Preparing to unpack .../euca2ools_3.1.0-1_all.deb ...
Unpacking euca2ools (3.1.0-1) ...
Selecting previously unselected package python3-lxc.
Preparing to unpack .../python3-lxc_1.1.2-0ubuntu3_amd64.deb ...
Unpacking python3-lxc (1.1.2-0ubuntu3) ...
Selecting previously unselected package lxc.
Preparing to unpack .../lxc_1.1.2-0ubuntu3_amd64.deb ...
Unpacking lxc (1.1.2-0ubuntu3) ...
Selecting previously unselected package lxc-templates.
Preparing to unpack .../lxc-templates_1.1.2-0ubuntu3_amd64.deb ...
Unpacking lxc-templates (1.1.2-0ubuntu3) ...
Selecting previously unselected package lxcfs.
Preparing to unpack .../lxcfs_0.7-0ubuntu4_amd64.deb ...
Unpacking lxcfs (0.7-0ubuntu4) ...
Selecting previously unselected package python-ndg-httpsclient.
Preparing to unpack .../python-ndg-httpsclient_0.3.2-1ubuntu4_all.deb ...
Unpacking python-ndg-httpsclient (0.3.2-1ubuntu4) ...
Selecting previously unselected package qemu-utils.
Preparing to unpack .../qemu-utils_1%3a2.2+dfsg-5expubuntu9.1_amd64.deb ...
Unpacking qemu-utils (1:2.2+dfsg-5expubuntu9.1) ...
Selecting previously unselected package sharutils.
Preparing to unpack .../sharutils_1%3a4.14-2_amd64.deb ...
Unpacking sharutils (1:4.14-2) ...
Selecting previously unselected package uidmap.
Preparing to unpack .../uidmap_1%3a4.1.5.1-1.1ubuntu4_amd64.deb ...
Unpacking uidmap (1:4.1.5.1-1.1ubuntu4) ...
Selecting previously unselected package cloud-image-utils.
Preparing to unpack .../cloud-image-utils_0.27-0ubuntu15_all.deb ...
Unpacking cloud-image-utils (0.27-0ubuntu15) ...
Selecting previously unselected package debootstrap.
Preparing to unpack .../debootstrap_1.0.67ubuntu0.1_all.deb ...
Unpacking debootstrap (1.0.67ubuntu0.1) ...
Selecting previously unselected package openvswitch-common.
Preparing to unpack .../openvswitch-common_2.3.1-0ubuntu1_amd64.deb ...
Unpacking openvswitch-common (2.3.1-0ubuntu1) ...
Selecting previously unselected package openvswitch-switch.
Preparing to unpack .../openvswitch-switch_2.3.1-0ubuntu1_amd64.deb ...
Unpacking openvswitch-switch (2.3.1-0ubuntu1) ...
Selecting previously unselected package uml-utilities.
Preparing to unpack .../uml-utilities_20070815-1.4_amd64.deb ...
Unpacking uml-utilities (20070815-1.4) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for ureadahead (0.100.0-19) ...
ureadahead will be reprofiled on next reboot
Processing triggers for systemd (219-7ubuntu5) ...
Processing triggers for install-info (5.2.0.dfsg.1-6) ...
Setting up libaio1:amd64 (0.3.110-1) ...
Setting up libboost-thread1.55.0:amd64 (1.55.0+dfsg-3ubuntu2) ...
Setting up libseccomp2:amd64 (2.1.1-1) ...
Setting up liblxc1 (1.1.2-0ubuntu3) ...
Setting up librados2 (0.94.1-0ubuntu1) ...
Setting up librbd1 (0.94.1-0ubuntu1) ...
Setting up python-urllib3 (1.9.1-3) ...
Setting up python-requests (2.4.3-6) ...
Setting up python-requestbuilder (0.2.3-1) ...
Setting up bridge-utils (1.5-7ubuntu1) ...
Setting up distro-info-data (0.26ubuntu0.1) ...
Setting up distro-info (0.14) ...
Setting up python-setuptools (12.2-1) ...
Setting up euca2ools (3.1.0-1) ...
Setting up python3-lxc (1.1.2-0ubuntu3) ...
Setting up lxc (1.1.2-0ubuntu3) ...
Setting up lxc dnsmasq configuration.
Setting up lxc-templates (1.1.2-0ubuntu3) ...
Setting up lxcfs (0.7-0ubuntu4) ...
Setting up python-ndg-httpsclient (0.3.2-1ubuntu4) ...
Setting up qemu-utils (1:2.2+dfsg-5expubuntu9.1) ...
Setting up sharutils (1:4.14-2) ...
Setting up uidmap (1:4.1.5.1-1.1ubuntu4) ...
Setting up cloud-image-utils (0.27-0ubuntu15) ...
Setting up debootstrap (1.0.67ubuntu0.1) ...
Setting up openvswitch-common (2.3.1-0ubuntu1) ...
Setting up openvswitch-switch (2.3.1-0ubuntu1) ...
openvswitch-nonetwork.service is a disabled or a static unit, not starting it.
Setting up uml-utilities (20070815-1.4) ...
Processing triggers for libc-bin (2.21-0ubuntu4) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (219-7ubuntu5) ...
gstanden@u1504:~$

Install DNS and DHCP packages

Next install bind9 and isc-dhcp-utils as shown below.


gstanden@u1504:~$ sudo apt-get install bind9 bind9utils isc-dhcp-server
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  bind9-doc isc-dhcp-server-ldap
The following NEW packages will be installed:
  bind9 bind9utils isc-dhcp-server
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 804 kB of archives.
After this operation, 2,663 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ vivid/main bind9utils amd64 1:9.9.5.dfsg-9 [147 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ vivid/main bind9 amd64 1:9.9.5.dfsg-9 [294 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ vivid-updates/main isc-dhcp-server amd64 4.3.1-5ubuntu2.1 [363 kB]
Fetched 804 kB in 2s (335 kB/s)           
Preconfiguring packages ...
Selecting previously unselected package bind9utils.
(Reading database ... 205485 files and directories currently installed.)
Preparing to unpack .../bind9utils_1%3a9.9.5.dfsg-9_amd64.deb ...
Unpacking bind9utils (1:9.9.5.dfsg-9) ...
Selecting previously unselected package bind9.
Preparing to unpack .../bind9_1%3a9.9.5.dfsg-9_amd64.deb ...
Unpacking bind9 (1:9.9.5.dfsg-9) ...
Selecting previously unselected package isc-dhcp-server.
Preparing to unpack .../isc-dhcp-server_4.3.1-5ubuntu2.1_amd64.deb ...
Unpacking isc-dhcp-server (4.3.1-5ubuntu2.1) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for ufw (0.34~rc-0ubuntu5) ...
Processing triggers for systemd (219-7ubuntu5) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up bind9utils (1:9.9.5.dfsg-9) ...
Setting up bind9 (1:9.9.5.dfsg-9) ...
Adding group `bind' (GID 134) ...
Done.
Adding system user `bind' (UID 122) ...
Adding new user `bind' (UID 122) with group `bind' ...
Not creating home directory `/var/cache/bind'.
wrote key file "/etc/bind/rndc.key"
#
Setting up isc-dhcp-server (4.3.1-5ubuntu2.1) ...
Generating /etc/default/isc-dhcp-server...
Processing triggers for systemd (219-7ubuntu5) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for ufw (0.34~rc-0ubuntu5) ...
gstanden@u1504:~$ 

Install OpenvSwitch Configuration Scripts

These scripts create and configure the OpenvSwitch switches.  The "crt_ovs_sw*.sh" scripts and the /etc/init/my-network-up script and reboot to verify openvswitch is creating switches and ports correctly. The scripts are attached to this blog (see end of blog) and can be downloaded directly.  They will be downloaded by default to "/home/username/Downloads" directory.

Once the scripts are downloaded, create a directory "/home/username/OpenvSwitch" and install the "crt_ovs_sw*.sh" scripts to that directory and set correct permissions as shown below.

gstanden@u1504:~$ pwd
/home/gstanden
gstanden@u1504:~$ mkdir OpenvSwitch
gstanden@u1504:~$ cd Downloads
gstanden@u1504:~/Downloads$ ls -lrt
total 40
-rw-rw-r-- 1 gstanden gstanden 2153 May 16 21:05 crt_ovs_sw1.sh
-rw-rw-r-- 1 gstanden gstanden 1016 May 16 21:05 crt_ovs_sw2.sh
-rw-rw-r-- 1 gstanden gstanden 1019 May 16 21:05 crt_ovs_sw3.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:05 crt_ovs_sw4.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:05 crt_ovs_sw5.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:05 crt_ovs_sw6.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:06 crt_ovs_sw7.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:06 crt_ovs_sw8.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:06 crt_ovs_sw9.sh
-rw-rw-r-- 1 gstanden gstanden 1935 May 16 21:06 crt_ovs_sx1.sh
gstanden@u1504:~/Downloads$ mv crt_ovs_s*.sh ../OpenvSwitch/.
gstanden@u1504:~/Downloads$ ls -lrt
total 0
gstanden@u1504:~/Downloads$ cd ..
gstanden@u1504:~$ cd OpenvSwitch/
gstanden@u1504:~/OpenvSwitch$ ls -lrt
total 40
-rw-rw-r-- 1 gstanden gstanden 2153 May 16 21:05 crt_ovs_sw1.sh
-rw-rw-r-- 1 gstanden gstanden 1016 May 16 21:05 crt_ovs_sw2.sh
-rw-rw-r-- 1 gstanden gstanden 1019 May 16 21:05 crt_ovs_sw3.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:05 crt_ovs_sw4.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:05 crt_ovs_sw5.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:05 crt_ovs_sw6.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:06 crt_ovs_sw7.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:06 crt_ovs_sw8.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:06 crt_ovs_sw9.sh
-rw-rw-r-- 1 gstanden gstanden 1935 May 16 21:06 crt_ovs_sx1.sh
gstanden@u1504:~/OpenvSwitch$

Also download the "my-network-up.sh" script from the end of this blog.  This script is used to ensure that the OpenvSwitch "crt_ovs_sw*.sh" scripts run only after networking has started.  The idea of using the "my-network-up.sh" script for starting up the OpenvSwitches at boot after the network interfaces are up is thanks to Cheesehead here.


The "my-network-up.sh" script will be downloaded to /home/username/Downloads".  Move the script to "/etc/init" as shown below and ensure it has correct permissions as shown below.  After moving the file into "/etc/init" create a soft link to the file in the "/home/username/OpenvSwitch" directory to reference where this file is located for future configuration updates to find the file easily as shown below.

gstanden@u1504:~$ cd Downloads
gstanden@u1504:~/Downloads$ ls -lrt
total 4
-rw-rw-r-- 1 gstanden gstanden 1119 May 16 21:08 my-network-up.sh
gstanden@u1504:~/Downloads$ cd ..
gstanden@u1504:~$ cd Downloads
gstanden@u1504:~/Downloads$ ls -lrt
total 4
-rw-rw-r-- 1 gstanden gstanden 1119 May 16 21:08 my-network-up.sh

gstanden@u1504:~/Downloads$ more my-network-up.sh
# 'my-network-up.conf' - My custom upstart events
#
# These are the scripts that run when a network appears.

description "My custom upstart events"

start on net-device-up

task

script
/home/gstanden/OpenvSwitch/crt_ovs_sw1.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw1.log
/home/gstanden/OpenvSwitch/crt_ovs_sx1.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sx1.log
/home/gstanden/OpenvSwitch/crt_ovs_sw2.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw2.log
/home/gstanden/OpenvSwitch/crt_ovs_sw3.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw3.log
/home/gstanden/OpenvSwitch/crt_ovs_sw4.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw4.log
/home/gstanden/OpenvSwitch/crt_ovs_sw5.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw5.log
/home/gstanden/OpenvSwitch/crt_ovs_sw6.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw6.log
/home/gstanden/OpenvSwitch/crt_ovs_sw7.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw7.log
/home/gstanden/OpenvSwitch/crt_ovs_sw8.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw8.log
/home/gstanden/OpenvSwitch/crt_ovs_sw9.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw9.log
end script

gstanden@u1504:~/Downloads$ sudo mv my-network-up.sh /etc/init/.
gstanden@u1504:~/Downloads$ cd ..
gstanden@u1504:~$ cd OpenvSwitch

gstanden@u1504:~/OpenvSwitch$ sudo ln -s /etc/init/my-network-up.sh .

gstanden@u1504:~/OpenvSwitch$ ls -lrt

total 40
-rw-rw-r-- 1 gstanden gstanden 2153 May 16 21:05 crt_ovs_sw1.sh
-rw-rw-r-- 1 gstanden gstanden 1016 May 16 21:05 crt_ovs_sw2.sh
-rw-rw-r-- 1 gstanden gstanden 1019 May 16 21:05 crt_ovs_sw3.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:05 crt_ovs_sw4.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:05 crt_ovs_sw5.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:05 crt_ovs_sw6.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:06 crt_ovs_sw7.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:06 crt_ovs_sw8.sh
-rw-rw-r-- 1 gstanden gstanden  704 May 16 21:06 crt_ovs_sw9.sh
-rw-rw-r-- 1 gstanden gstanden 1935 May 16 21:06 crt_ovs_sx1.sh
lrwxrwxrwx 1 root     root       26 May 16 21:11 my-network-up.sh -> /etc/init/my-network-up.sh
gstanden@u1504:~/OpenvSwitch$
gstanden@u1504:~/OpenvSwitch$ sudo chmod 755 /etc/init/my-network-up.sh

Update Ubuntu 15.04 2015-05-16

The my-network-up.sh script doesn't seem to be working anymore in Ubuntu 15.04 but it's (possibly ?) some mistake on my part.  In any case, in the meantime, I had to add entries to /etc/rc.local script to get my OpenvSwitch network to startup ok at boot, so currently I still have the /etc/init/my-network-up.sh script in place, but also now the following in /etc/rc.local as well as shown below.

gstanden@u1504:~$ cat /etc/rc.local

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

/home/gstanden/OpenvSwitch/crt_ovs_sx1.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sx1.log
/home/gstanden/OpenvSwitch/crt_ovs_sw1.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw1.log
/home/gstanden/OpenvSwitch/crt_ovs_sw2.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw2.log
/home/gstanden/OpenvSwitch/crt_ovs_sw3.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw3.log
/home/gstanden/OpenvSwitch/crt_ovs_sw4.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw4.log
/home/gstanden/OpenvSwitch/crt_ovs_sw5.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw5.log
/home/gstanden/OpenvSwitch/crt_ovs_sw6.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw6.log
/home/gstanden/OpenvSwitch/crt_ovs_sw7.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw7.log
/home/gstanden/OpenvSwitch/crt_ovs_sw8.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw8.log
/home/gstanden/OpenvSwitch/crt_ovs_sw9.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw9.log

exit 0


gstanden@u1504:~$

Note also that crt_ovs_sx1.sh has been added.  This is the addition of a new network subnet 10.207.29.1 and comes with a new set of bind9 and dhcp configurations.  Basically, this update to the guide here shows how you add additional networks to the DNS-DHCP bind9/isc-dhcp-server setup on the laptop.

Update End Ubuntu 15.04 2015-05-16

Once the "crt_ovs_sw*.sh" scripts are downloaded, create a directory for them as shown below, and move the scripts to the "/home/username/OpenvSwitch" directory as shown below. Ensure that the "crt_ovs_sw*.sh" scripts have the correct ownership, permissions, and are installed in the directory as shown below.  The log files do not need to be created.  They are written automatically during each bootup of the laptop or desktop when the OpenvSwitch switches are created during bootup.

Create a soft link in the "/home/username/OpenvSwitch" directory as a reminder of where this important script is installed for future maintenance or reference.  The idea is that all scripts directly relevant to the OpenvSwitch configuration are linked in the "/home/username/OpenvSwitch" directory for quick reference and accessability.

gstanden@u1504:~/OpenvSwitch$ chmod 755 crt_ovs_s*.sh

gstanden@u1504:~/OpenvSwitch$ ls -lrt

total 40
-rwxr-xr-x 1 gstanden gstanden 2153 May 16 21:05 crt_ovs_sw1.sh
-rwxr-xr-x 1 gstanden gstanden 1016 May 16 21:05 crt_ovs_sw2.sh
-rwxr-xr-x 1 gstanden gstanden 1019 May 16 21:05 crt_ovs_sw3.sh
-rwxr-xr-x 1 gstanden gstanden  704 May 16 21:05 crt_ovs_sw4.sh
-rwxr-xr-x 1 gstanden gstanden  704 May 16 21:05 crt_ovs_sw5.sh
-rwxr-xr-x 1 gstanden gstanden  704 May 16 21:05 crt_ovs_sw6.sh
-rwxr-xr-x 1 gstanden gstanden  704 May 16 21:06 crt_ovs_sw7.sh
-rwxr-xr-x 1 gstanden gstanden  704 May 16 21:06 crt_ovs_sw8.sh
-rwxr-xr-x 1 gstanden gstanden  704 May 16 21:06 crt_ovs_sw9.sh
-rwxr-xr-x 1 gstanden gstanden 1935 May 16 21:06 crt_ovs_sx1.sh
lrwxrwxrwx 1 root     root       26 May 16 21:11 my-network-up.sh -> /etc/init/my-network-up.sh

gstanden@u1504:~/OpenvSwitch$

The OpenvSwitch switch scripts are also listed below for reference.

gstanden@u1504:~/OpenvSwitch$ more crt_ovs_s*.sh
::::::::::::::
crt_ovs_sw1.sh
::::::::::::::
#!/bin/bash
# Requires use of Upstart Script /etc/init/my-network-up.conf to ensure interfaces are up before running.
 
tunctl -t s1
tunctl -t s2
tunctl -t s3
tunctl -t s4
tunctl -t s5
#tunctl -t s6
ip link set s1 up
ip link set s2 up
ip link set s3 up
ip link set s4 up
ip link set s5 up
#ip link set s6 up
ovs-vsctl add-br sw1
ovs-vsctl add-port sw1 s1
ovs-vsctl add-port sw1 s2
ovs-vsctl add-port sw1 s3
ovs-vsctl add-port sw1 s4
ovs-vsctl add-port sw1 s5
#ovs-vsctl add-port sw1 s6
ip link set up dev sw1
ip addr add 10.207.39.1/24 dev sw1
ip route replace 10.207.39.0/24 dev sw1

ovs-vsctl set port sw1 trunks=10
ovs-vsctl set port sw1 tag=10

# GLS 20140825 Get active external interface dynamically at boot.  Tested & works with {wlan0, eth0, bnep0} on Ubuntu 14.04.1 Desktop x86_64.
# GLS 20140825 Interface "bnep0" is Blackberry Z30 OS10 Bluetooth Tether.

### BEGIN Get Active EXTIF Dynamcially. ###
function GetInterface
{
ifconfig | egrep -B1 'inet addr' | egrep -A1 'wlan|eth|bnep' | sed '$!N;s/\n/ /' | sed 's/  */ /g' | cut -f1,7 -d' ' | sed 's/ addr//' | head -1 | cut -f1 -d':'
}
function GetIP
{
ifconfig | egrep -B1 'inet addr' | egrep -A1 'wlan|eth|bnep' | sed '$!N;s/\n/ /' | sed 's/  */ /g' | cut -f1,7 -d' ' | sed 's/ addr//' | head -1 | cut -f2 -d':'
}
### END Get Active EXTIF Dynamically. ###

echo '       IP: '$(GetIP)
echo 'Interface: '$(GetInterface)

INTIF="sw1"
EXTIF=$(GetInterface)
# EXTIF="wlan0"

echo 1 > /proc/sys/net/ipv4/ip_forward

# clear existing iptable rules, set a default policy
iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -t nat -F

# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
iptables -A FORWARD -s 10.207.39.0/24 -o $EXTIF -j ACCEPT
iptables -A FORWARD -d 10.207.39.0/24 -m state --state ESTABLISHED,RELATED -i $EXTIF -j ACCEPT
iptables -t nat -A POSTROUTING -s 10.207.39.0/24 -o $EXTIF -j MASQUERADE

service isc-dhcp-server start
service bind9 restart
::::::::::::::
crt_ovs_sw2.sh
::::::::::::::
#!/bin/bash

tunctl -t t1
tunctl -t t2
tunctl -t t3
tunctl -t t4
tunctl -t t5
ip link set t1 up
ip link set t2 up
ip link set t3 up
ip link set t4 up
ip link set t5 up
ovs-vsctl add-br sw2
ovs-vsctl add-port sw2 t1
ovs-vsctl add-port sw2 t2
ovs-vsctl add-port sw2 t3
ovs-vsctl add-port sw2 t4
ovs-vsctl add-port sw2 t5
ip link set up dev sw2
ip addr add 10.207.40.1/24 dev sw2
ip route replace 10.207.40.0/24 dev sw2
ifconfig sw2 10.207.40.1 netmask 255.255.255.0

ovs-vsctl set port sw2 tag=80

# INTIF="sw2"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward

# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F

# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

# service isc-dhcp-server start
::::::::::::::
crt_ovs_sw3.sh
::::::::::::::
#!/bin/bash

tunctl -t w1
tunctl -t w2
tunctl -t w3
tunctl -t w4
tunctl -t w5
ip link set w1 up
ip link set w2 up
ip link set w3 up
ip link set w4 up
ip link set w5 up
ovs-vsctl add-br sw3
ovs-vsctl add-port sw3 w1
ovs-vsctl add-port sw3 w2
ovs-vsctl add-port sw3 w3
ovs-vsctl add-port sw3 w4
ovs-vsctl add-port sw3 w5

ip link set up dev sw3
ip addr add 10.207.41.1/24 dev sw3
ip route replace 10.207.41.0/24 dev sw3
ifconfig sw3 10.207.41.1 netmask 255.255.255.0

 ovs-vsctl set port sw3 tag=90

# INTIF="sw3"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward

# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F

# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

# service isc-dhcp-server start

::::::::::::::
crt_ovs_sw4.sh
::::::::::::::
#!/bin/bash

ovs-vsctl add-br sw4
ip link set up dev sw4
ip addr add 192.210.39.1/24 dev sw4
ip route replace 192.210.39.0/24 dev sw4
ifconfig sw4 192.210.39.1 netmask 255.255.255.0

# INTIF="sw3"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward

# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F

# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

# service isc-dhcp-server start

::::::::::::::
crt_ovs_sw5.sh
::::::::::::::
#!/bin/bash

ovs-vsctl add-br sw5
ip link set up dev sw5
ip addr add 192.211.39.1/24 dev sw5
ip route replace 192.211.39.0/24 dev sw5
ifconfig sw5 192.211.39.1 netmask 255.255.255.0

# INTIF="sw3"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward

# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F

# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

# service isc-dhcp-server start

::::::::::::::
crt_ovs_sw6.sh
::::::::::::::
#!/bin/bash

ovs-vsctl add-br sw6
ip link set up dev sw6
ip addr add 192.212.39.1/24 dev sw6
ip route replace 192.212.39.0/24 dev sw6
ifconfig sw6 192.212.39.1 netmask 255.255.255.0

# INTIF="sw3"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward

# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F

# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

# service isc-dhcp-server start

::::::::::::::
crt_ovs_sw7.sh
::::::::::::::
#!/bin/bash

ovs-vsctl add-br sw7
ip link set up dev sw7
ip addr add 192.213.39.1/24 dev sw7
ip route replace 192.213.39.0/24 dev sw7
ifconfig sw7 192.213.39.1 netmask 255.255.255.0

# INTIF="sw3"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward

# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F

# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

# service isc-dhcp-server start

::::::::::::::
crt_ovs_sw8.sh
::::::::::::::
#!/bin/bash

ovs-vsctl add-br sw8
ip link set up dev sw8
ip addr add 172.220.40.1/24 dev sw8
ip route replace 172.220.40.0/24 dev sw8
ifconfig sw8 172.220.40.1 netmask 255.255.255.0

# INTIF="sw3"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward

# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F

# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

# service isc-dhcp-server start

::::::::::::::
crt_ovs_sw9.sh
::::::::::::::
#!/bin/bash

ovs-vsctl add-br sw9
ip link set up dev sw9
ip addr add 172.221.40.1/24 dev sw9
ip route replace 172.221.40.0/24 dev sw9
ifconfig sw9 172.221.40.1 netmask 255.255.255.0

# INTIF="sw3"
# EXTIF="wlan0"
# echo 1 > /proc/sys/net/ipv4/ip_forward

# clear existing iptable rules, set a default policy
# iptables -P INPUT ACCEPT
# iptables -F INPUT
# iptables -P OUTPUT ACCEPT
# iptables -F OUTPUT
# iptables -P FORWARD DROP
# iptables -F FORWARD
# iptables -t nat -F

# set forwarding and nat rules
# iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
# iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
# iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

# service isc-dhcp-server start

::::::::::::::
crt_ovs_sx1.sh
::::::::::::::
#!/bin/bash
# Requires use of Upstart Script /etc/init/my-network-up.conf to ensure interfaces are up before running.
sleep 10
tunctl -t a1
tunctl -t a2
tunctl -t a3
tunctl -t a4
tunctl -t a5
#tunctl -t a6
ip link set a1 up
ip link set a2 up
ip link set a3 up
ip link set a4 up
ip link set a5 up
#ip link set a6 up
ovs-vsctl add-br sx1
ovs-vsctl add-port sx1 a1
ovs-vsctl add-port sx1 a2
ovs-vsctl add-port sx1 a3
ovs-vsctl add-port sx1 a4
ovs-vsctl add-port sx1 a5
#ovs-vsctl add-port sx1 a6
ip link set up dev sx1
ip addr add 10.207.29.1/24 dev sx1
ip route replace 10.207.29.0/24 dev sx1

ovs-vsctl set port sx1 trunks=10
ovs-vsctl set port sx1 tag=10

# GLS 20140825 Get active external interface dynamically at boot.  Tested & works with {wlan0, eth0, bnep0} on Ubuntu 14.04.1 Desktop x86_64.
# GLS 20140825 Interface "bnep0" is Blackberry Z30 OS10 Bluetooth Tether.

### BEGIN Get Active EXTIF Dynamcially. ###
function GetInterface
{
ifconfig | egrep -B1 'inet addr' | egrep -A1 'wlan|eth|bnep' | sed '$!N;s/\n/ /' | sed 's/  */ /g' | cut -f1,7 -d' ' | sed 's/ addr//' | head -1 | cut -f1 -d':'
}
function GetIP
{
ifconfig | egrep -B1 'inet addr' | egrep -A1 'wlan|eth|bnep' | sed '$!N;s/\n/ /' | sed 's/  */ /g' | cut -f1,7 -d' ' | sed 's/ addr//' | head -1 | cut -f2 -d':'
}
### END Get Active EXTIF Dynamically. ###

echo '       IP: '$(GetIP)
echo 'Interface: '$(GetInterface)

INTIF="sx1"
EXTIF=$(GetInterface)
# EXTIF="wlan0"

echo 1 > /proc/sys/net/ipv4/ip_forward

# clear existing iptable rules, set a default policy
iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -t nat -F

# set forwarding and nat rules
iptables -A FORWARD -i $EXTIF -o $INTIF -j ACCEPT
iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

# service isc-dhcp-server restart
# service bind9 restart
gstanden@u1504:~/OpenvSwitch$

Setup DNS and DHCP for System

The DNS and DHCP are setup in this blog so that DHCP can provide addresses to LXC containers and VMs on the OpenvSwitch, and also add those DHCP assignments to DNS automatically so that as containers and KVM guests are created, they get handed out IP addresses, and those IP addresses get automatically added to the fwd and reverse lookup files without having to edit them manually at all.

Backup DNS Default Configuration Files and Install Downloaded Versions

First backup the default configuration files as shown below.  Then download from the end of this blog this files "named.conf.options" and "named.conf.local" and move to "/etc/bind" directory as shown below.

gstanden@u1504:~$ sudo su -

root@u1504:~# cd /etc/bind

root@u1504:/etc/bind# cp -p named.conf.local named.conf.local.original.install.bak
root@u1504:/etc/bind# cp -p named.conf.options named.conf.options.original.install.bak
root@u1504:/etc/bind# cp -p rndc.key rndc.key.original.install.bak

root@u1504:/etc/bind# ls -lrt

total 64
-rw-r--r-- 1 root root 1317 Feb 19 11:07 zones.rfc1918
-rw-r--r-- 1 root bind  165 Feb 19 11:07 named.conf.local.original.install.bak
-rw-r--r-- 1 root bind  165 Feb 19 11:07 named.conf.local
-rw-r--r-- 1 root bind  490 Feb 19 11:07 named.conf.default-zones
-rw-r--r-- 1 root bind  463 Feb 19 11:07 named.conf
-rw-r--r-- 1 root root 3048 Feb 19 11:07 db.root
-rw-r--r-- 1 root root  270 Feb 19 11:07 db.local
-rw-r--r-- 1 root root  353 Feb 19 11:07 db.empty
-rw-r--r-- 1 root root  237 Feb 19 11:07 db.255
-rw-r--r-- 1 root root  271 Feb 19 11:07 db.127
-rw-r--r-- 1 root root  237 Feb 19 11:07 db.0
-rw-r--r-- 1 root root 2389 Feb 19 11:07 bind.keys
-rw-r----- 1 bind bind   77 May 16 21:03 rndc.key.original.install.bak
-rw-r----- 1 bind bind   77 May 16 21:03 rndc.key
-rw-r--r-- 1 root bind  890 May 16 21:03 named.conf.options.original.install.bak
-rw-r--r-- 1 root bind  890 May 16 21:03 named.conf.options

root@u1504:/etc/bind# cd

root@u1504:~# exit
logout

gstanden@u1504:~$ cd Downloads

gstanden@u1504:~/Downloads$ ls -lrt
total 8

-rw-rw-r-- 1 gstanden gstanden 435 May 16 21:33 named.conf.local
-rw-rw-r-- 1 gstanden gstanden 299 May 16 21:33 named.conf.options

gstanden@u1504:~/Downloads$ sudo mv named.* /etc/bind/.

gstanden@u1504:~/Downloads$

The new named.conf.local file is shown below.

gstanden@u1504:~/Downloads$ more named.conf.local

//
// Do any local configuration here
//

// Consider adding the 1918 zones here, if they are not used in your
// organization

include "/etc/bind/zones.rfc1918";
include "/etc/bind/rndc.key";

zone "vmem.org" {
    type master;
    file "/var/lib/bind/fwd.vmem.org";
    allow-update { key rndc-key; };
};

zone "mccc.org" {
    type master;
    file "/var/lib/bind/fwd.mccc.org";
    allow-update { key rndc-key; };
};

zone "39.207.10.in-addr.arpa" {
    type master;
    file "/var/lib/bind/rev.vmem.org";
    allow-update { key rndc-key; };
};

zone "29.207.10.in-addr.arpa" {
    type master;
    file "/var/lib/bind/rev.mccc.org";
    allow-update { key rndc-key; };
};
gstanden@u1504:~/Downloads$

The named.conf.options file is shown below.  Download it from this blog and move it to /etc/bind directory.  The dhcp files are also shown here.

gstanden@u1504:~/Downloads$ sudo mv named.conf.local /etc/bind/.

gstanden@u1504:~/Downloads$ sudo mv named.conf.options /etc/bind/.

gstanden@u1504:~/Downloads$ cat /etc/dhcp/dhclient.conf

# Configuration file for /sbin/dhclient, which is included in Debian's
#    dhcp3-client package.
#
# This is a sample configuration file for dhclient. See dhclient.conf's
#    man page for more information about the syntax of this file
#    and a more comprehensive list of the parameters understood by
#    dhclient.
#
# Normally, if the DHCP server provides reasonable information and does
#    not leave anything out (like the domain name, for example), then
#    few changes must be made to this file, if any.
#

option rfc3442-classless-static-routes code 121 = array of unsigned integer 8;

send host-name = gethostname();
prepend domain-name-servers 127.0.0.1;

# Oracle GNS and additional domains
append domain-name " mccc.org";

request subnet-mask, broadcast-address, time-offset, routers,
    domain-name, domain-name-servers, domain-search, host-name,
    dhcp6.name-servers, dhcp6.domain-search,
    netbios-name-servers, netbios-scope, interface-mtu,
    rfc3442-classless-static-routes, ntp-servers,
    dhcp6.fqdn, dhcp6.sntp-servers;

gstanden@u1504:~/Downloads$ cat /etc/dhcp/dhcpd.conf

#
# Configuration file for ISC dhcpd for Ubuntu 14.04
# Configuration file for ISC dhcpd for Ubuntu 15.04 GLS Tested on 15.04 2015.05.02
# Added a second subnet configuration (10.207.29.1/24) for mediacomcorp.org domain
#
ddns-updates on;  
ddns-update-style interim;  
update-static-leases on;  
authoritative;  
key rndc-key { algorithm hmac-md5; secret "5rcWKMkWCP6RBUeu9tjIZg==";}  
allow unknown-clients;  
use-host-decl-names on;  
default-lease-time 1814400;
max-lease-time 1814400;
log-facility local7;  

zone vmem.org. {
    primary 10.207.39.1;
    key rndc-key;
}
zone mccc.org. {
    primary 10.207.29.1;
    key rndc-key;
}
zone 39.207.10.in-addr.arpa. {
    primary 10.207.39.1;
    key rndc-key;
}
zone 29.207.10.in-addr.arpa. {
    primary 10.207.29.1;
    key rndc-key;
}
subnet 10.207.39.0 netmask 255.255.255.0 {
#  ---  default gateway
    option routers             10.207.39.1;
    option subnet-mask         255.255.255.0;
    option domain-name         "vmem.org";
    option domain-name-servers 10.207.39.1;
#    option ntp-servers        10.207.39.1;
    ddns-domainname            "vmem.org.";
    ddns-rev-domainname        "in-addr.arpa.";
    range                      10.207.39.70 10.207.39.254;
    default-lease-time         1814400;
    max-lease-time             1814400;
}
subnet 10.207.29.0 netmask 255.255.255.0 {
#  ---  default gateway
    option routers             10.207.29.1;
    option subnet-mask         255.255.255.0;
    option domain-name         "mccc.org";
    option domain-name-servers 10.207.29.1;
#    option ntp-servers        10.207.29.1;
    ddns-domainname            "mccc.org.";
    ddns-rev-domainname        "in-addr.arpa.";
    range                      10.207.29.70 10.207.29.254;
    default-lease-time         1814400;
    max-lease-time             1814400;
}
gstanden@u1504:~/Downloads$

Note that these are the updated files to support multiple domains and multiple networks on the desktop.  Also, the /etc/NetworkManager/dnsmasq.d/local file is shown below.  This file tells dnsmasq not to listen on these subnets and allow named to manage these subnets exclusively.

gstanden@u1504:~/dnsmasq.d$ more local

server=/vmem.org/10.207.39.1
server=/39.207.10.in-addr.arpa/10.207.39.1
server=/mccc.org/10.207.29.1
server=/29.207.10.in-addr.arpa/10.207.29.1

gstanden@u1504:~/dnsmasq.d$

Backup DHCP Default Configuration Files

Now backup the isc-dhcp-server (DHCP) original install configuration files as shown below.  With the addition of extra subnets and domains, dhclient.conf has some changes from default, because it has been used to add search domains to /etc/resolv.conf at boot, so both dhclient.conf and dhcpd.conf are changed from default as shown below.

gstanden@u1504:~/Downloads$ sudo su -

root@u1504:~# cd /etc/dhcp

root@u1504:/etc/dhcp# ls -lrt
total 20
-rw-r--r-- 1 root root  3641 Nov  5  2014 dhcpd.conf
-rw-r--r-- 1 root root  1830 Nov  5  2014 dhclient.conf
drwxr-xr-x 2 root root  4096 May 16 20:51 dhclient-enter-hooks.d
drwxr-xr-x 2 root root  4096 May 16 20:51 dhclient-exit-hooks.d
drwxr-x--- 2 root dhcpd 4096 May 16 21:04 ddns-keys

root@u1504:/etc/dhcp# cp -p dhcpd.conf dhcpd.conf.original.install.bak
root@u1504:/etc/dhcp# cp -p dhclient.conf dhclient.conf.original.install.bak

root@u1504:/etc/dhcp# exit
logout

gstanden@u1504:~/Downloads$ ls -lrt
total 8

-rw-rw-r-- 1 gstanden gstanden 1019 May 16 21:39 dhclient.conf
-rw-rw-r-- 1 gstanden gstanden 1596 May 16 21:39 dhcpd.conf

gstanden@u1504:~/Downloads$ sudo cp -p dh*.conf /etc/dhcp/.
gstanden@u1504:~/Downloads$ cd /etc/dhcp

gstanden@u1504:/etc/dhcp$ ls -lrt
total 28

-rw-r--r-- 1 root     root     3641 Nov  5  2014 dhcpd.conf.original.install.bak
-rw-r--r-- 1 root     root     1830 Nov  5  2014 dhclient.conf.original.install.bak
drwxr-xr-x 2 root     root     4096 May 16 20:51 dhclient-enter-hooks.d
drwxr-xr-x 2 root     root     4096 May 16 20:51 dhclient-exit-hooks.d
drwxr-x--- 2 root     dhcpd    4096 May 16 21:04 ddns-keys
-rw-rw-r-- 1 gstanden gstanden 1019 May 16 21:39 dhclient.conf
-rw-rw-r-- 1 gstanden gstanden 1596 May 16 21:39 dhcpd.conf

gstanden@u1504:/etc/dhcp$

Update Begin Ubuntu 15.04 2015-05-16

There is a change to the /etc/dhcp/dhclient.conf file for the added mccc.org domain as shown below.  This is set now in /etc/network/interfaces file, not here anymore.

gstanden@u1504:/etc/dhcp$ cat dhclient.conf

# Configuration file for /sbin/dhclient, which is included in Debian's
#    dhcp3-client package.
#
# This is a sample configuration file for dhclient. See dhclient.conf's
#    man page for more information about the syntax of this file
#    and a more comprehensive list of the parameters understood by
#    dhclient.
#
# Normally, if the DHCP server provides reasonable information and does
#    not leave anything out (like the domain name, for example), then
#    few changes must be made to this file, if any.
#

option rfc3442-classless-static-routes code 121 = array of unsigned integer 8;

send host-name = gethostname();
prepend domain-name-servers 127.0.0.1;

# Oracle GNS and additional domains
# append domain-name " mccc.org"; <-- this is set in /etc/network/interfaces now, not using this line anymore.

request subnet-mask, broadcast-address, time-offset, routers,
    domain-name, domain-name-servers, domain-search, host-name,
    dhcp6.name-servers, dhcp6.domain-search,
    netbios-name-servers, netbios-scope, interface-mtu,
    rfc3442-classless-static-routes, ntp-servers,
    dhcp6.fqdn, dhcp6.sntp-servers;

gstanden@u1504:/etc/dhcp$

Install Downloaded DHCP Files

Move the downloaded "dhcpd.conf" to "/etc/dhcp" directory as shown below and ensure it has the correct value from /etc/bind/rndc.key file as shown below..

gstanden@u1504:/etc/init$ sudo cat /etc/bind/rndc.key
[sudo] password for gstanden:

key "rndc-key" {
    algorithm hmac-md5;
    secret "FLf9C0hIeYhEeE23lfN1Mw==";    <-- Be sure to copy this to /etc/dhcp/dhcpd.conf file as shown below.
};

gstanden@u1504:/etc/init$ cat /etc/dhcp/dhcpd.conf

#
# Configuration file for ISC dhcpd for Ubuntu 14.04
# Configuration file for ISC dhcpd for Ubuntu 15.04 GLS Tested on 15.04 2015.05.02
# Added a second subnet configuration (10.207.29.1/24) for mediacomcorp.org domain
#

ddns-updates on;  
ddns-update-style interim;  
update-static-leases on;  
authoritative;  
key rndc-key { algorithm hmac-md5; secret "FLf9C0hIeYhEeE23lfN1Mw==";}  
allow unknown-clients;  
use-host-decl-names on;  
default-lease-time 1814400;
max-lease-time 1814400;
log-facility local7;  

zone vmem.org. {
    primary 10.207.39.1;
    key rndc-key;
}
zone mccc.org. {
    primary 10.207.29.1;
    key rndc-key;
}
zone 39.207.10.in-addr.arpa. {
    primary 10.207.39.1;
    key rndc-key;
}
zone 29.207.10.in-addr.arpa. {
    primary 10.207.29.1;
    key rndc-key;
}
subnet 10.207.39.0 netmask 255.255.255.0 {
#  ---  default gateway
    option routers             10.207.39.1;
    option subnet-mask         255.255.255.0;
    option domain-name         "vmem.org";
    option domain-name-servers 10.207.39.1;
#    option ntp-servers        10.207.39.1;
    ddns-domainname            "vmem.org.";
    ddns-rev-domainname        "in-addr.arpa.";
    range                      10.207.39.70 10.207.39.254;
    default-lease-time         1814400;
    max-lease-time             1814400;
}
subnet 10.207.29.0 netmask 255.255.255.0 {
#  ---  default gateway
    option routers             10.207.29.1;
    option subnet-mask         255.255.255.0;
    option domain-name         "mccc.org";
    option domain-name-servers 10.207.29.1;
#    option ntp-servers        10.207.29.1;
    ddns-domainname            "mccc.org.";
    ddns-rev-domainname        "in-addr.arpa.";
    range                      10.207.29.70 10.207.29.254;
    default-lease-time         1814400;
    max-lease-time             1814400;
}

gstanden@u1504:/etc/init$

Set Ownership and Permissions of DHCP Configuration Files

Change ownership of dhcp configuration files to root as shown below.

root@W520:/etc/dhcp# chown root:root dhclient.conf dhcpd.conf
root@W520:/etc/dhcp# ls -lrt *.conf
total 28
-rw-rw-r-- 1 root root   940 Dec 26 10:15 dhclient.conf
-rw-rw-r-- 1 root root   890 Dec 26 10:25 dhcpd.conf

root@W520:/etc/dhcp#

Install DNS Forward and Reverse Zone Configuration Files

Download the "fwd.vmem.org" and the "rev.vmem.org" zone files from the downloadable files at the end of this blog, and also the fwd.mccc.org and the rev.mccc.org files. Install the downloaded zone files into correct location as shown below.  Download the "named.conf.options" and the "named.conf.local" file from the downloadable files at the end of the blog and install.

gstanden@u1504:~$ cd Downloads
gstanden@u1504:~/Downloads$ ls -lrt

total 264
-rw-rw-r-- 1 gstanden gstanden 250832 Dec 25 22:54 rev.vmem.org
-rw-rw-r-- 1 gstanden gstanden   1096 Dec 26 10:15 fwd.vmem.org
-rw-rw-r-- 1 gstanden gstanden    435 Dec 26 10:15 fwd.mccc.org
-rw-rw-r-- 1 gstanden gstanden    299 Dec 26 10:15 rev.mccc.org

gstanden@u1504:~/Downloads$ sudo mv fwd.*.org /var/lib/bind/.
gstanden@u1504:~/Downloads$ sudo mv rev.*.org /var/lib/bind/.

Edit DNS Forward and Reverse Zone Files

Setup the forward and reverse zone files.  Change the domain name information if desired. Be careful when editing not to remove any of the "." characters in the file, they matter!  Also, verify the hostname as shown and make sure to update the forward and reverse zone files with the name of your desktop host (in this example the desktop hostname is "u1504").  When installing the downloaded forward and reverse lookup files "fwd.vmem.org" and "rev.vmem.org" it will be necessary to change the "u1504" to the name of the actual Ubuntu desktop system.  Also, the domain can be changed from "vmem.org" to some-other-domain.[org|com|net|biz|...]" if desired.  Note that if it is changed to something else, changes will need to be made in all the other corresponding files, such as "/etc/network/interfaces", "named.conf.local" etc.

gstanden@u1504:~/Downloads$ sudo su -
root@u1504:~# cd /var/lib/bind
root@u1504:/var/lib/bind# ls -lrt
total 20
-rw-r--r-- 1 root     root      53 May 16 21:03 bind9-default.md5sum
-rw-rw-r-- 1 gstanden gstanden 394 May 17 21:18 fwd.mccc.org
-rw-rw-r-- 1 gstanden gstanden 342 May 17 21:18 rev.mccc.org
-rw-rw-r-- 1 gstanden gstanden 506 May 17 21:20 fwd.vmem.org
-rw-rw-r-- 1 gstanden gstanden 456 May 17 21:21 rev.vmem.org

root@u1504:/var/lib/bind# named-checkzone mccc.org fwd.mccc.org
zone mccc.org/IN: loaded serial 1505021309
OK
root@u1504:/var/lib/bind# named-checkzone 29.207.10.in-addr.arpa rev.mccc.org
zone 29.207.10.in-addr.arpa/IN: loaded serial 1411021420
OK
root@u1504:/var/lib/bind# named-checkzone 39.207.10.in-addr.arpa rev.vmem.org
zone 39.207.10.in-addr.arpa/IN: loaded serial 1412261100
OK
root@u1504:/var/lib/bind# named-checkzone vmem.org fwd.vmem.org
zone vmem.org/IN: loaded serial 1412261100
OK
root@u1504:/var/lib/bind# cat fwd.vmem.org

$ORIGIN .
$TTL 86400            ; time-to-live in seconds
vmem.org        IN SOA    u1504.vmem.org. postmaster.vmem.org. (
                1412261100 ; serial
                60         ; refresh (1 minute)
                1800       ; retry (30 minutes)
                604800     ; expire (1 week)
                86400      ; minimum (1 day)
                )
            NS    u1504.vmem.org.
$ORIGIN vmem.org.
_sflow._udp        TXT    "txtvers=1" "polling=20" "sampling=512"
            SRV    0 0 6343 u1504
u1504            A    10.207.39.1
lxc1-gns-vip.vmem.org.    A    10.207.39.3
$ORIGIN gns1.vmem.org.
@    IN        NS    lxc1-gns-vip.vmem.org.
root@u1504:/var/lib/bind# cat rev.vmem.org

$ORIGIN .
$TTL 86400            ; time-to-live in seconds
39.207.10.in-addr.arpa    IN SOA    u1504.vmem.org. postmaster.vmem.org. (
                1412261100 ; serial
                60         ; refresh (1 hour)
                1800       ; retry (30 minutes)
                604800     ; expire (1 week)
                86400      ; minimum (1 day)
                )
            NS    u1504.vmem.org.
$ORIGIN 39.207.10.in-addr.arpa.
1            PTR    u1504.vmem.org.
3            PTR    lxc1-gns-vip.vmem.org.
39.207.10.in-addr.arpa    NS    u1504.vmem.org.
W520            A    10.207.39.1

root@u1504:/var/lib/bind# cat fwd.mccc.org
$ORIGIN .
$TTL 86400    ; 1 day
mccc.org        IN SOA    mccc1.mccc.org. postmaster.mccc.org. (
                1505021309 ; serial
                60         ; refresh (1 minute)
                1800       ; retry (30 minutes)
                604800     ; expire (1 week)
                86400      ; minimum (1 day)
                )
            NS    mccc1.mccc.org.
$ORIGIN mccc.org.
_sflow._udp        TXT    "txtvers=1" "polling=20" "sampling=512"
            SRV    0 0 6343 mccc1
mccc1            A    10.207.29.1

root@u1504:/var/lib/bind# cat rev.mccc.org
$ORIGIN .
$TTL 86400    ; 1 day
29.207.10.in-addr.arpa    IN SOA    mccc1.mccc.org. postmaster.mccc.org. (
                1411021420 ; serial
                3600       ; refresh (1 hour)
                1800       ; retry (30 minutes)
                604800     ; expire (1 week)
                86400      ; minimum (1 day)
                )
            NS    mccc1.mccc.org.
$ORIGIN 29.207.10.in-addr.arpa.
1            PTR    mccc1.mccc.org.
root@u1504:/var/lib/bind# 

Configure dnsmasq-base

Now configure dnsmasq-base ("NetworkManager" component) to IGNORE DNS requests for the local-only vmem.org domain as shown below. In other words, this step puts the local domain "vmem.org" under the control of bind9, and tells dnsmasq-base to ignore this domain, and also to ignore mccc.org domain as well.  The additonal lines in bold are for the added mccc.org domain, and additonal domains can be added with similar two-line additions of the same form.

root@u1504:/var/lib/bind# cat /etc/NetworkManager/dnsmasq.d/local

server=/vmem.org/10.207.39.1
server=/39.207.10.in-addr.arpa/10.207.39.1
server=/mccc.org/10.207.29.1
server=/29.207.10.in-addr.arpa/10.207.29.1


root@u1504:/var/lib/bind#

Configure /etc/network/interfaces

Ensure that "/etc/network/interfaces" file has the following three lines added after "iface" line.  These lines are responsible for configuring the "/etc/resolv.conf" file.  Note that "mccc.org" is added to /etc/resolv.conf here.  In earlier versions it had been added at /etc/dhcp/dhclient.conf file, but that method is not used here; all domains are added here at /etc/network/interfaces file.

gstanden@u1504:~$ cat /etc/network/interfaces

# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback
    dns-domain vmem.org
    dns-search mccc.org gns1.vmem.org
    dns-nameserver 127.0.0.1


gstanden@u1504:~$ cat /etc/resolv.conf

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1
search vmem.org mccc.org gns1.vmem.org    <-- /etc/network/interfaces above produces this /etc/resolv.conf file

gstanden@u1504:~$

At this point, the DNS-DHCP should be configured.  Verify by restarting bind9 and isc-dhcp-server as shown below and testing lookups on both domains and subnets, both forward and reverse lookups, and verify that /etc/resolv.conf file has the desired search and domain settings.  Note that "vmem.org" is the domain for the desktop, and that "mccc.org" is an added "search domain".  There would typically be some LXC Linux Containers, or KVM guests that might be on this "mccc.org" domain, but that the domain of the desktop Ubuntu system is "vmem.org".

root@u1504:/var/lib/bind# service bind9 restart
root@u1504:/var/lib/bind# service bind9 status

● bind9.service - BIND Domain Name Server
   Loaded: loaded (/lib/systemd/system/bind9.service; enabled; vendor preset: enabled)
  Drop-In: /run/systemd/generator/bind9.service.d
           └─50-insserv.conf-$named.conf
   Active: active (running) since Sun 2015-05-17 21:38:34 CDT; 3s ago
     Docs: man:named(8)
  Process: 5353 ExecStop=/usr/sbin/rndc stop (code=exited, status=0/SUCCESS)
 Main PID: 5358 (named)
   Memory: 24.2M
   CGroup: /system.slice/bind9.service
           └─5358 /usr/sbin/named -f -u bind

May 17 21:38:34 u1504 named[5358]: zone 21.172.in-addr.arpa/IN: loaded serial 1
May 17 21:38:34 u1504 named[5358]: zone 22.172.in-addr.arpa/IN: loaded serial 1
May 17 21:38:34 u1504 named[5358]: zone localhost/IN: loaded serial 2
May 17 21:38:34 u1504 named[5358]: zone 127.in-addr.arpa/IN: loaded serial 1
May 17 21:38:34 u1504 named[5358]: zone 27.172.in-addr.arpa/IN: loaded serial 1
May 17 21:38:34 u1504 named[5358]: zone 16.172.in-addr.arpa/IN: loaded serial 1
May 17 21:38:34 u1504 named[5358]: zone 24.172.in-addr.arpa/IN: loaded serial 1
May 17 21:38:34 u1504 named[5358]: zone mccc.org/IN: loaded serial 1505021309
May 17 21:38:34 u1504 named[5358]: all zones loaded
May 17 21:38:34 u1504 named[5358]: running

root@u1504:/var/lib/bind# service isc-dhcp-server restart
root@u1504:/var/lib/bind# service isc-dhcp-server status

● isc-dhcp-server.service - ISC DHCP IPv4 server
   Loaded: loaded (/lib/systemd/system/isc-dhcp-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2015-05-17 21:38:45 CDT; 20s ago
     Docs: man:dhcpd(8)
  Process: 5387 ExecStartPre=/bin/chown dhcpd:dhcpd /run/dhcp-server (code=exited, status=0/SUCCESS)
 Main PID: 5390 (dhcpd)
   Memory: 10.3M
   CGroup: /system.slice/isc-dhcp-server.service
           └─5390 dhcpd -user dhcpd -group dhcpd -f -4 -pf /run/dhcp-server/dhcpd.pid -cf /etc/dhcp/dhcpd.conf

May 17 21:38:45 u1504 sh[5390]: you want, please write a subnet declaration
May 17 21:38:45 u1504 sh[5390]: in your dhcpd.conf file for the network segment
May 17 21:38:45 u1504 sh[5390]: to which interface wlan0 is attached. **
May 17 21:38:45 u1504 sh[5390]: No subnet declaration for eth0 (no IPv4 addresses).
May 17 21:38:45 u1504 sh[5390]: ** Ignoring requests on eth0.  If this is not what
May 17 21:38:45 u1504 sh[5390]: you want, please write a subnet declaration
May 17 21:38:45 u1504 sh[5390]: in your dhcpd.conf file for the network segment
May 17 21:38:45 u1504 sh[5390]: to which interface eth0 is attached. **
May 17 21:38:45 u1504 sh[5390]: Sending on   Socket/fallback/fallback-net
May 17 21:38:45 u1504 sh[5390]: Can't create PID file /run/dhcp-server/dhcpd.pid: Permission denied.

root@u1504:/var/lib/bind# nslookup u1504
Server:        127.0.1.1
Address:    127.0.1.1#53

Name:    u1504.vmem.org
Address: 10.207.39.1

root@u1504:/var/lib/bind# nslookup mccc1
Server:        127.0.1.1
Address:    127.0.1.1#53

Name:    mccc1.mccc.org
Address: 10.207.29.1

root@u1504:/var/lib/bind# cat /etc/resolv.conf

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
search vmem.org mccc.org gns1.vmem.org <-- Note until reboot/network restart, may have to add domains manually to resolv.conf

root@u1504:/var/lib/bind# 

root@u1504:/var/lib/bind# nslookup 10.207.39.1
Server:        127.0.1.1
Address:    127.0.1.1#53

1.39.207.10.in-addr.arpa    name = u1504.vmem.org.

root@u1504:/var/lib/bind# nslookup 10.207.29.1
Server:        127.0.1.1
Address:    127.0.1.1#53

1.29.207.10.in-addr.arpa    name = mccc1.mccc.org.

root@u1504:/var/lib/bind#

Configure /etc/sysctl.conf

Make the settings values changes to existing parameters in the /etc/sysctl.conf as shown below thanks to the blogpost from Venu Murthy here. Additional reference material and full-resolution schematics of OpenvSwitch OpenStack networking can be found at OpenStack documentation.

gstanden@u1504:~/Downloads$ cat /etc/sysctl.conf | egrep -B1 'rp_filter|ip_forward'

# GLS 20141226 http://thenewstack.io/solving-a-common-beginners-problem-when-pinging-from-an-openstack-instance/
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0

--
# GLS 20141226 http://thenewstack.io/solving-a-common-beginners-problem-when-pinging-from-an-openstack-instance/
net.ipv4.ip_forward=1

gstanden@u1504:~/Downloads$

Install and Configure Required Apparmor Package

Next install apparmor-utils package as shown below.

gstanden@u1504:~$ sudo apt-get install apparmor-utils

gstanden@u1504:~/Downloads$ sudo apt-get install apparmor-utils
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  python3-apparmor python3-libapparmor
Suggested packages:
  apparmor-docs vim-addon-manager
The following NEW packages will be installed:
  apparmor-utils python3-apparmor python3-libapparmor
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 141 kB of archives.
After this operation, 911 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://us.archive.ubuntu.com/ubuntu/ vivid/main python3-libapparmor amd64 2.9.1-0ubuntu9 [24.9 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ vivid/main python3-apparmor amd64 2.9.1-0ubuntu9 [63.2 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ vivid/main apparmor-utils amd64 2.9.1-0ubuntu9 [53.4 kB]
Fetched 141 kB in 0s (364 kB/s)          
Selecting previously unselected package python3-libapparmor.
(Reading database ... 205593 files and directories currently installed.)
Preparing to unpack .../python3-libapparmor_2.9.1-0ubuntu9_amd64.deb ...
Unpacking python3-libapparmor (2.9.1-0ubuntu9) ...
Selecting previously unselected package python3-apparmor.
Preparing to unpack .../python3-apparmor_2.9.1-0ubuntu9_amd64.deb ...
Unpacking python3-apparmor (2.9.1-0ubuntu9) ...
Selecting previously unselected package apparmor-utils.
Preparing to unpack .../apparmor-utils_2.9.1-0ubuntu9_amd64.deb ...
Unpacking apparmor-utils (2.9.1-0ubuntu9) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up python3-libapparmor (2.9.1-0ubuntu9) ...
Setting up python3-apparmor (2.9.1-0ubuntu9) ...
Setting up apparmor-utils (2.9.1-0ubuntu9) ...
gstanden@u1504:~/Downloads$ 

Set the apparmor profile for lxc-start to "complain" mode as shown below.  Currently, this is considered by the author to be the Ubuntu Linux equivalent of setting selinux to "permissive" but not absolutely sure of that.  This then would mean that this is the equivalent step in Venu's blog post for setting selinux to permissive mode in the apparmor paradigm.

gstanden@u1504:~/Downloads$ sudo aa-complain /usr/bin/lxc-start
Setting /usr/bin/lxc-start to complain mode.
gstanden@u1504:~/Downloads$

Install and configure openssh-server

The openssh-server package is needed so that ssh from the container to the Ubuntu host OS will work.  Install the openssh-server package as shown below.

[root@lxcora01 ~]# ssh gstanden@u1504.vmem.org

ssh: connect to host u1504.vmem.org port 22: Connection refused

[root@lxcora01 ~]# exit
logout

Connection to lxcora01 closed.

gstanden@u1504:~$ sudo apt-get install openssh-server
[sudo] password for gstanden:
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following packages were automatically installed and are no longer required:
  linux-headers-3.19.0-15 linux-headers-3.19.0-15-generic linux-image-3.19.0-15-generic linux-image-extra-3.19.0-15-generic
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  ncurses-term openssh-sftp-server ssh-import-id
Suggested packages:
  rssh molly-guard monkeysphere
The following NEW packages will be installed:
  ncurses-term openssh-server openssh-sftp-server ssh-import-id
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 630 kB of archives.
After this operation, 3,441 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://us.archive.ubuntu.com/ubuntu/ vivid/main ncurses-term all 5.9+20140712-2ubuntu2 [248 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ vivid/main openssh-sftp-server amd64 1:6.7p1-5ubuntu1 [38.3 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ vivid/main openssh-server amd64 1:6.7p1-5ubuntu1 [334 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu/ vivid/main ssh-import-id all 4.1-0ubuntu1 [9,344 B]
Fetched 630 kB in 0s (631 kB/s)         
Preconfiguring packages ...
Selecting previously unselected package ncurses-term.
(Reading database ... 236321 files and directories currently installed.)
Preparing to unpack .../ncurses-term_5.9+20140712-2ubuntu2_all.deb ...
Unpacking ncurses-term (5.9+20140712-2ubuntu2) ...
Selecting previously unselected package openssh-sftp-server.
Preparing to unpack .../openssh-sftp-server_1%3a6.7p1-5ubuntu1_amd64.deb ...
Unpacking openssh-sftp-server (1:6.7p1-5ubuntu1) ...
Selecting previously unselected package openssh-server.
Preparing to unpack .../openssh-server_1%3a6.7p1-5ubuntu1_amd64.deb ...
Unpacking openssh-server (1:6.7p1-5ubuntu1) ...
Selecting previously unselected package ssh-import-id.
Preparing to unpack .../ssh-import-id_4.1-0ubuntu1_all.deb ...
Unpacking ssh-import-id (4.1-0ubuntu1) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for systemd (219-7ubuntu5) ...
Processing triggers for ureadahead (0.100.0-19) ...
ureadahead will be reprofiled on next reboot
Processing triggers for ufw (0.34~rc-0ubuntu5) ...
Setting up ncurses-term (5.9+20140712-2ubuntu2) ...
Setting up openssh-sftp-server (1:6.7p1-5ubuntu1) ...
Setting up openssh-server (1:6.7p1-5ubuntu1) ...
Creating SSH2 RSA key; this may take some time ...
2048 dc:07:ad:02:9a:59:e6:a4:ec:12:85:b1:86:14:2b:48 /etc/ssh/ssh_host_rsa_key.pub (RSA)
Creating SSH2 DSA key; this may take some time ...
1024 93:ef:1b:01:ef:c2:b4:d1:ca:30:db:4e:b2:9b:f6:3a /etc/ssh/ssh_host_dsa_key.pub (DSA)
Creating SSH2 ECDSA key; this may take some time ...
256 d7:9f:a8:e2:b6:7d:71:93:ae:f8:7d:d5:70:54:26:c8 /etc/ssh/ssh_host_ecdsa_key.pub (ECDSA)
Creating SSH2 ED25519 key; this may take some time ...
256 9b:86:f9:41:a6:7a:67:8f:08:ed:95:fd:13:69:75:18 /etc/ssh/ssh_host_ed25519_key.pub (ED25519)
Setting up ssh-import-id (4.1-0ubuntu1) ...
Processing triggers for systemd (219-7ubuntu5) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for ufw (0.34~rc-0ubuntu5) ...

gstanden@u1504:~$

Add Settings to /etc/sysctl.conf File

Next add the following settings at the end of /etc/sysctl.conf for Oracle as shown below.  Hugepages value is optional and can be adjusted downward depending on whether Hugepages are used or not and depending on what size of Oracle SGA.

# Oracle

kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576

net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

vm.nr_hugepages = 2060
kernel.panic_on_oops = 1

The final /etc/sysctl.conf file after all above edits should be similar to the one shown below. Sections edited and added are shown in bold.

gstanden@u1504:~/Downloads$ cat /etc/sysctl.conf

# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
#

#kernel.domainname = example.com

# Uncomment the following to stop low-level messages on console
#kernel.printk = 3 4 1 3

##############################################################3
# Functions previously found in netbase
#

# Uncomment the next two lines to enable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks

# GLS 20141226 http://thenewstack.io/solving-a-common-beginners-problem-when-pinging-from-an-openstack-instance/
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0


# Uncomment the next line to enable TCP/IP SYN cookies
# See http://lwn.net/Articles/277146/
# Note: This may impact IPv6 TCP sessions too
#net.ipv4.tcp_syncookies=1

# Uncomment the next line to enable packet forwarding for IPv4
# GLS 20141226 http://thenewstack.io/solving-a-common-beginners-problem-when-pinging-from-an-openstack-instance/
net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPv6
#  Enabling this option disables Stateless Address Autoconfiguration
#  based on Router Advertisements for this host
#net.ipv6.conf.all.forwarding=1


###################################################################
# Additional settings - these settings can improve the network
# security of the host and prevent against some network attacks
# including spoofing attacks and man in the middle attacks through
# redirection. Some network environments, however, require that these
# settings are disabled so review and enable them as needed.
#
# Do not accept ICMP redirects (prevent MITM attacks)
#net.ipv4.conf.all.accept_redirects = 0
#net.ipv6.conf.all.accept_redirects = 0
# _or_
# Accept ICMP redirects only for gateways listed in our default
# gateway list (enabled by default)
# net.ipv4.conf.all.secure_redirects = 1
#
# Do not send ICMP redirects (we are not a router)
#net.ipv4.conf.all.send_redirects = 0
#
# Do not accept IP source route packets (we are not a router)
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv6.conf.all.accept_source_route = 0
#
# Log Martian Packets
#net.ipv4.conf.all.log_martians = 1
#
# Oracle
#
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
vm.nr_hugepages = 2060
kernel.panic_on_oops = 1


gstanden@u1504:~/Downloads$ 

Apply the changes to the system using the command as shown below and verify that all new settings were accepted and applied.

gstanden@u1504:~/Downloads$ sudo sysctl -p

net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.ip_forward = 1
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
vm.nr_hugepages = 2060
kernel.panic_on_oops = 1

gstanden@u1504:~/Downloads$ 

Restart DNS and DHCP to Verify Configuration

Reboot the desktop host to reset the "/etc/resolv.conf" file to the new settings, and verify DNS and DHCP configuration is correct by doing an nslookup on "w520" and "w520.vmem.org".  Both should return the local IP address for that server. Run the tests as shown below to:

  • Verify DNS is working;
  • Verify that "named" is monitoring 10.207.39.1
  • Verify that "named" in monitoring 10.207.29.1 (new network for mccc.org)  GLS 2015-05-16
  • Verify that "dnsmasq" is monitoring 10.0.3.1
  • Verify that the OpenvSwitch switches are created
  • Verify that "/etc/resolv.conf" has the correct settings

These tests are shown below with correct expected output.

gstanden@u1504:~/Downloads$ cat /etc/resolv.conf

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1
search vmem.org mccc.org gns1.vmem.org

gstanden@u1504:~/Downloads$ nslookup u1504
Server:        127.0.0.1
Address:    127.0.0.1#53

Name:    u1504.vmem.org
Address: 10.207.39.1

gstanden@u1504:~/Downloads$ nslookup u1504.vmem.org
Server:        127.0.0.1
Address:    127.0.0.1#53

Name:    u1504.vmem.org
Address: 10.207.39.1

gstanden@u1504:~/Downloads$ nslookup mccc1
Server:        127.0.0.1
Address:    127.0.0.1#53

Name:    mccc1.mccc.org
Address: 10.207.29.1

gstanden@u1504:~/Downloads$ nslookup mccc1.mccc.org
Server:        127.0.0.1
Address:    127.0.0.1#53

Name:    mccc1.mccc.org
Address: 10.207.29.1

gstanden@u1504:~/Downloads$ nslookup 10.207.39.1
Server:        127.0.0.1
Address:    127.0.0.1#53

1.39.207.10.in-addr.arpa    name = u1504.vmem.org.

gstanden@u1504:~/Downloads$ nslookup 10.207.29.1
Server:        127.0.0.1
Address:    127.0.0.1#53

1.29.207.10.in-addr.arpa    name = mccc1.mccc.org.

gstanden@u1504:~/Downloads$ sudo netstat -ulnp | grep 53

udp        0      0 0.0.0.0:5353            0.0.0.0:*                           861/avahi-daemon: r
udp        0      0 10.207.29.1:53          0.0.0.0:*                           2861/named      
udp        0      0 10.207.39.1:53          0.0.0.0:*                           2861/named      
udp        0      0 127.0.0.1:53            0.0.0.0:*                           2861/named      
udp        0      0 127.0.1.1:53            0.0.0.0:*                           1444/dnsmasq    
udp        0      0 10.0.3.1:53             0.0.0.0:*                           1196/dnsmasq    
udp6       0      0 :::53974                :::*                                861/avahi-daemon: r
udp6       0      0 :::5353                 :::*                                861/avahi-daemon: r
udp6       0      0 :::53                   :::*                                2861/named      
udp6       0      0 fe80::a4d0:6dff:fe09:53 :::*                                1196/dnsmasq
  
gstanden@u1504:~/Downloads$ sudo ovs-vsctl show

684bc501-621d-4083-a692-697fccf1e0e6
    Bridge "sw6"
        Port "sw6"
            Interface "sw6"
                type: internal
    Bridge "sw4"
        Port "sw4"
            Interface "sw4"
                type: internal
    Bridge "sx1"
        Port "a4"
            Interface "a4"
        Port "sx1"
            tag: 10
            trunks: [10]
            Interface "sx1"
                type: internal
        Port "a3"
            Interface "a3"
        Port "a5"
            Interface "a5"
        Port "a1"
            Interface "a1"
        Port "a2"
            Interface "a2"
    Bridge "sw7"
        Port "sw7"
            Interface "sw7"
                type: internal
    Bridge "sw1"
        Port "s5"
            Interface "s5"
        Port "s3"
            Interface "s3"
        Port "s2"
            Interface "s2"
        Port "sw1"
            tag: 10
            trunks: [10]
            Interface "sw1"
                type: internal
        Port "s4"
            Interface "s4"
        Port "s1"
            Interface "s1"
    Bridge "sw3"
        Port "w5"
            Interface "w5"
        Port "sw3"
            tag: 90
            Interface "sw3"
                type: internal
        Port "w4"
            Interface "w4"
        Port "w2"
            Interface "w2"
        Port "w1"
            Interface "w1"
        Port "w3"
            Interface "w3"
    Bridge "sw8"
        Port "sw8"
            Interface "sw8"
                type: internal
    Bridge "sw2"
        Port "t3"
            Interface "t3"
        Port "sw2"
            tag: 80
            Interface "sw2"
                type: internal
        Port "t4"
            Interface "t4"
        Port "t1"
            Interface "t1"
        Port "t5"
            Interface "t5"
        Port "t2"
            Interface "t2"
    Bridge "sw5"
        Port "sw5"
            Interface "sw5"
                type: internal
    Bridge "sw9"
        Port "sw9"
            Interface "sw9"
                type: internal
    ovs_version: "2.3.1"

gstanden@u1504:~/Downloads$

Create Oracle Enteprise Linux 6.5 LXC Linux Container

Install Ubuntu rpm and yum Packages

Create an Oracle Enteprise Linux (OEL) 6.5 LXC Linux Container as shown below.  Notice that the "rpm" and "yum" deb packages must be installed first.  As shown below, LXC container creation is not possible without the Ubuntu versions of "rpm" and "yum" installed first.

gstanden@u1504:~/Downloads$ sudo apt-get install rpm yum

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  debugedit librpm3 librpmbuild3 librpmio3 librpmsign1 libsqlite0 python-libxml2 python-lzma python-rpm python-sqlite python-sqlitecachec python-urlgrabber rpm-common
  rpm2cpio
Suggested packages:
  rpm-i18n python-sqlite-dbg alien elfutils rpmlint rpm2html
The following NEW packages will be installed:
  debugedit librpm3 librpmbuild3 librpmio3 librpmsign1 libsqlite0 python-libxml2 python-lzma python-rpm python-sqlite python-sqlitecachec python-urlgrabber rpm
  rpm-common rpm2cpio yum
0 upgraded, 16 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,372 kB of archives.
After this operation, 7,560 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://us.archive.ubuntu.com/ubuntu/ vivid/main librpmio3 amd64 4.11.3-1.1 [69.3 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ vivid/main debugedit amd64 4.11.3-1.1 [15.8 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ vivid/main librpm3 amd64 4.11.3-1.1 [155 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu/ vivid/main librpmbuild3 amd64 4.11.3-1.1 [58.0 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu/ vivid/main librpmsign1 amd64 4.11.3-1.1 [8,062 B]
Get:6 http://us.archive.ubuntu.com/ubuntu/ vivid/universe libsqlite0 amd64 2.8.17-12fakesync1 [139 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu/ vivid/main python-libxml2 amd64 2.9.2+dfsg1-3 [143 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu/ vivid/universe python-lzma amd64 0.5.3-2build1 [32.2 kB]
Get:9 http://us.archive.ubuntu.com/ubuntu/ vivid/universe python-sqlite amd64 1.0.1-11 [20.7 kB]
Get:10 http://us.archive.ubuntu.com/ubuntu/ vivid/main python-urlgrabber all 3.9.1-4ubuntu5 [42.4 kB]
Get:11 http://us.archive.ubuntu.com/ubuntu/ vivid/main rpm-common amd64 4.11.3-1.1 [26.3 kB]
Get:12 http://us.archive.ubuntu.com/ubuntu/ vivid/main rpm2cpio amd64 4.11.3-1.1 [5,228 B]
Get:13 http://us.archive.ubuntu.com/ubuntu/ vivid/main rpm amd64 4.11.3-1.1 [115 kB]
Get:14 http://us.archive.ubuntu.com/ubuntu/ vivid/universe python-rpm amd64 4.11.3-1.1 [33.6 kB]
Get:15 http://us.archive.ubuntu.com/ubuntu/ vivid/universe python-sqlitecachec amd64 1.1.4-1 [21.4 kB]
Get:16 http://us.archive.ubuntu.com/ubuntu/ vivid/universe yum all 3.4.3-3 [486 kB]
Fetched 1,372 kB in 2s (684 kB/s)
Selecting previously unselected package librpmio3.
(Reading database ... 205658 files and directories currently installed.)
Preparing to unpack .../librpmio3_4.11.3-1.1_amd64.deb ...
Unpacking librpmio3 (4.11.3-1.1) ...
Selecting previously unselected package debugedit.
Preparing to unpack .../debugedit_4.11.3-1.1_amd64.deb ...
Unpacking debugedit (4.11.3-1.1) ...
Selecting previously unselected package librpm3.
Preparing to unpack .../librpm3_4.11.3-1.1_amd64.deb ...
Unpacking librpm3 (4.11.3-1.1) ...
Selecting previously unselected package librpmbuild3.
Preparing to unpack .../librpmbuild3_4.11.3-1.1_amd64.deb ...
Unpacking librpmbuild3 (4.11.3-1.1) ...
Selecting previously unselected package librpmsign1.
Preparing to unpack .../librpmsign1_4.11.3-1.1_amd64.deb ...
Unpacking librpmsign1 (4.11.3-1.1) ...
Selecting previously unselected package libsqlite0.
Preparing to unpack .../libsqlite0_2.8.17-12fakesync1_amd64.deb ...
Unpacking libsqlite0 (2.8.17-12fakesync1) ...
Selecting previously unselected package python-libxml2.
Preparing to unpack .../python-libxml2_2.9.2+dfsg1-3_amd64.deb ...
Unpacking python-libxml2 (2.9.2+dfsg1-3) ...
Selecting previously unselected package python-lzma.
Preparing to unpack .../python-lzma_0.5.3-2build1_amd64.deb ...
Unpacking python-lzma (0.5.3-2build1) ...
Selecting previously unselected package python-sqlite.
Preparing to unpack .../python-sqlite_1.0.1-11_amd64.deb ...
Unpacking python-sqlite (1.0.1-11) ...
Selecting previously unselected package python-urlgrabber.
Preparing to unpack .../python-urlgrabber_3.9.1-4ubuntu5_all.deb ...
Unpacking python-urlgrabber (3.9.1-4ubuntu5) ...
Selecting previously unselected package rpm-common.
Preparing to unpack .../rpm-common_4.11.3-1.1_amd64.deb ...
Unpacking rpm-common (4.11.3-1.1) ...
Selecting previously unselected package rpm2cpio.
Preparing to unpack .../rpm2cpio_4.11.3-1.1_amd64.deb ...
Unpacking rpm2cpio (4.11.3-1.1) ...
Selecting previously unselected package rpm.
Preparing to unpack .../rpm_4.11.3-1.1_amd64.deb ...
Unpacking rpm (4.11.3-1.1) ...
Selecting previously unselected package python-rpm.
Preparing to unpack .../python-rpm_4.11.3-1.1_amd64.deb ...
Unpacking python-rpm (4.11.3-1.1) ...
Selecting previously unselected package python-sqlitecachec.
Preparing to unpack .../python-sqlitecachec_1.1.4-1_amd64.deb ...
Unpacking python-sqlitecachec (1.1.4-1) ...
Selecting previously unselected package yum.
Preparing to unpack .../archives/yum_3.4.3-3_all.deb ...
Unpacking yum (3.4.3-3) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up librpmio3 (4.11.3-1.1) ...
Setting up debugedit (4.11.3-1.1) ...
Setting up librpm3 (4.11.3-1.1) ...
Setting up librpmbuild3 (4.11.3-1.1) ...
Setting up librpmsign1 (4.11.3-1.1) ...
Setting up libsqlite0 (2.8.17-12fakesync1) ...
Setting up python-libxml2 (2.9.2+dfsg1-3) ...
Setting up python-lzma (0.5.3-2build1) ...
Setting up python-sqlite (1.0.1-11) ...
Setting up python-urlgrabber (3.9.1-4ubuntu5) ...
Setting up rpm-common (4.11.3-1.1) ...
Setting up rpm2cpio (4.11.3-1.1) ...
Setting up rpm (4.11.3-1.1) ...
Setting up python-rpm (4.11.3-1.1) ...
Setting up python-sqlitecachec (1.1.4-1) ...
Setting up yum (3.4.3-3) ...
Processing triggers for libc-bin (2.21-0ubuntu4) ...
Processing triggers for python-support (1.0.15) ...

gstanden@u1504:~/Downloads$

Create LXC Container

Now install the OEL 6.5 LXC Linux Container as shown below.  The full log of the installation is attached to the blog as "lxcora01.log".  Abridged information from the creation of the LXC is shown below.

gstanden@u1504:~/Downloads$ sudo lxc-create -t oracle -n lxcora01

Host is Ubuntu 15.04
No release specified with -R, defaulting to 6.5
Create configuration file /var/lib/lxc/lxcora01/config
Yum installing release 6.5 for x86_64
ol6_u5_base                                                                                                                                        | 1.4 kB     00:00    
ol6_u5_base/primary                                                                                                                                | 3.2 MB     00:02    
ol6_u5_base                                                                                                                                                     8573/8573
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package chkconfig.x86_64 0:1.3.49.3-2.el6_4.1 will be installed
--> Processing Dependency: libc.so.6(GLIBC_2.2.5)(64bit) for package: chkconfig-1.3.49.3-2.el6_4.1.x86_64
--> Processing Dependency: libc.so.6(GLIBC_2.8)(64bit) for package: chkconfig-1.3.49.3-2.el6_4.1.x86_64
--> Processing Dependency: libpopt.so.0(LIBPOPT_0)(64bit) for package: chkconfig-1.3.49.3-2.el6_4.1.x86_64
--> Processing Dependency: rtld(GNU_HASH) for package: chkconfig-1.3.49.3-2.el6_4.1.x86_64
--> Processing Dependency: libc.so.6(GLIBC_2.3.4)(64bit) for package: chkconfig-1.3.49.3-2.el6_4.1.x86_64
--> Processing Dependency: libc.so.6(GLIBC_2.4)(64bit) for package: chkconfig-1.3.49.3-2.el6_4.1.x86_64
--> Processing Dependency: libc.so.6(GLIBC_2.3)(64bit) for package: chkconfig-1.3.49.3-2.el6_4.1.x86_64
--> Processing Dependency: libpopt.so.0()(64bit) for package: chkconfig-1.3.49.3-2.el6_4.1.x86_64
--> Processing Dependency: libc.so.6()(64bit) for package: chkconfig-1.3.49.3-2.el6_4.1.x86_64
---> Package dhclient.x86_64 12:4.1.1-38.P1.0.1.el6 will be installed
--> Processing Dependency: dhcp-common = 12:4.1.1-38.P1.0.1.el6 for package: 12:dhclient-4.1.1-38.P1.0.1.el6.x86_64
--> Processing Dependency: grep for package: 12:dhclient-4.1.1-38.P1.0.1.el6.x86_64
--> Processing Dependency: coreutils for package: 12:dhclient-4.1.1-38.P1.0.1.el6.x86_64
--> Processing Dependency: /bin/bash for package: 12:dhclient-4.1.1-38.P1.0.1.el6.x86_64
--> Processing Dependency: /bin/sh for package: 12:dhclient-4.1.1-38.P1.0.1.el6.x86_64
--> Processing Dependency: libcap-ng.so.0()(64bit) for package: 12:dhclient-4.1.1-38.P1.0.1.el6.x86_64
---> Package initscripts.x86_64 0:9.03.40-2.0.1.el6 will be installed
--> Processing Dependency: upstart >= 0.6.0 for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: udev >= 125-1 for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: util-linux-ng >= 2.16 for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: sysvinit-tools >= 2.87 for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: ethtool >= 1.8-2 for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: /sbin/fuser for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: /sbin/arping for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: cpio for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: /usr/sbin/groupadd for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: /bin/find for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: /sbin/ip for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: /sbin/blkid for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: /bin/sed for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: net-tools for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: /sbin/sysctl for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: /bin/awk for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: module-init-tools for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: oracle-logos for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: mingetty for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: /sbin/pidof for package: initscripts-9.03.40-2.0.1.el6.x86_64
--> Processing Dependency: libglib-2.0.so.0()(64bit) for package: initscripts-9.03.40-2.0.1.el6.x86_64
---> Package openssh-clients.x86_64 0:5.3p1-94.el6 will be installed
--> Processing Dependency: openssh = 5.3p1-94.el6 for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libgssapi_krb5.so.2(gssapi_krb5_2_MIT)(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libnss3.so(NSS_3.4)(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libnss3.so(NSS_3.9.3)(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libnss3.so(NSS_3.5)(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libnss3.so(NSS_3.9)(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libcrypto.so.10(libcrypto.so.10)(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libcrypto.so.10(OPENSSL_1.0.1)(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libnss3.so(NSS_3.2)(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libnss3.so(NSS_3.6)(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libnss3.so(NSS_3.3)(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libnss3.so(NSS_3.9.2)(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libfipscheck.so.1()(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libk5crypto.so.3()(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libkrb5.so.3()(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libselinux.so.1()(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libedit.so.0()(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libcom_err.so.2()(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libplc4.so()(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libgssapi_krb5.so.2()(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libz.so.1()(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libcrypto.so.10()(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libnss3.so()(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
--> Processing Dependency: libncurses.so.5()(64bit) for package: openssh-clients-5.3p1-94.el6.x86_64
---> Package openssh-server.x86_64 0:5.3p1-94.el6 will be installed
--> Processing Dependency: pam >= 1.0.1-3 for package: openssh-server-5.3p1-94.el6.x86_64
--> Processing Dependency: libpam.so.0(LIBPAM_1.0)(64bit) for package: openssh-server-5.3p1-94.el6.x86_64
--> Processing Dependency: libaudit.so.1()(64bit) for package: openssh-server-5.3p1-94.el6.x86_64
--> Processing Dependency: libwrap.so.0()(64bit) for package: openssh-server-5.3p1-94.el6.x86_64
--> Processing Dependency: libpam.so.0()(64bit) for package: openssh-server-5.3p1-94.el6.x86_64
---> Package oraclelinux-release.x86_64 6:6Server-5.0.2 will be installed
--> Processing Dependency: redhat-release-server for package: 6:oraclelinux-release-6Server-5.0.2.x86_64
---> Package passwd.x86_64 0:0.77-4.el6_2.2 will be installed
--> Processing Dependency: libuser.so.1()(64bit) for package: passwd-0.77-4.el6_2.2.x86_64
---> Package policycoreutils.x86_64 0:2.0.83-19.39.0.1.el6 will be installed
--> Processing Dependency: libselinux-utils >= 2.0.90-3 for package: policycoreutils-2.0.83-19.39.0.1.el6.x86_64
--> Processing Dependency: libsepol >= 2.0.41-3 for package: policycoreutils-2.0.83-19.39.0.1.el6.x86_64
--> Processing Dependency: checkpolicy for package: policycoreutils-2.0.83-19.39.0.1.el6.x86_64
--> Processing Dependency: libsemanage.so.1(LIBSEMANAGE_1.0)(64bit) for package: policycoreutils-2.0.83-19.39.0.1.el6.x86_64
--> Processing Dependency: /usr/bin/diff for package: policycoreutils-2.0.83-19.39.0.1.el6.x86_64
--> Processing Dependency: rpm for package: policycoreutils-2.0.83-19.39.0.1.el6.x86_64
--> Processing Dependency: libdbus-1.so.3()(64bit) for package: policycoreutils-2.0.83-19.39.0.1.el6.x86_64
--> Processing Dependency: libdbus-glib-1.so.2()(64bit) for package: policycoreutils-2.0.83-19.39.0.1.el6.x86_64
--> Processing Dependency: libsemanage.so.1()(64bit) for package: policycoreutils-2.0.83-19.39.0.1.el6.x86_64
--> Processing Dependency: libsepol.so.1()(64bit) for package: policycoreutils-2.0.83-19.39.0.1.el6.x86_64
---> Package rootfiles.noarch 0:8.1-6.1.el6 will be installed
---> Package rsyslog.x86_64 0:5.8.10-8.0.1.el6 will be installed
--> Processing Dependency: logrotate >= 3.5.2 for package: rsyslog-5.8.10-8.0.1.el6.x86_64
--> Processing Dependency: libgcc_s.so.1(GCC_3.0)(64bit) for package: rsyslog-5.8.10-8.0.1.el6.x86_64
--> Processing Dependency: libgcc_s.so.1(GCC_3.3.1)(64bit) for package: rsyslog-5.8.10-8.0.1.el6.x86_64
--> Processing Dependency: libgcc_s.so.1()(64bit) for package: rsyslog-5.8.10-8.0.1.el6.x86_64
---> Package vim-minimal.x86_64 2:7.2.411-1.8.el6 will be installed
--> Processing Dependency: libacl.so.1(ACL_1.0)(64bit) for package: 2:vim-minimal-7.2.411-1.8.el6.x86_64
--> Processing Dependency: libacl.so.1()(64bit) for package: 2:vim-minimal-7.2.411-1.8.el6.x86_64
---> Package yum.noarch 0:3.2.29-40.0.1.el6 will be installed
--> Processing Dependency: python(abi) = 2.6 for package: yum-3.2.29-40.0.1.el6.noarch
--> Processing Dependency: python-urlgrabber >= 3.9.0-8 for package: yum-3.2.29-40.0.1.el6.noarch
--> Processing Dependency: python >= 2.4 for package: yum-3.2.29-40.0.1.el6.noarch
--> Processing Dependency: yum-metadata-parser >= 1.1.0 for package: yum-3.2.29-40.0.1.el6.noarch
--> Processing Dependency: python-iniparse for package: yum-3.2.29-40.0.1.el6.noarch
--> Processing Dependency: python-sqlite for package: yum-3.2.29-40.0.1.el6.noarch
--> Processing Dependency: pygpgme for package: yum-3.2.29-40.0.1.el6.noarch
--> Processing Dependency: rpm-python for package: yum-3.2.29-40.0.1.el6.noarch
--> Processing Dependency: /usr/bin/python for package: yum-3.2.29-40.0.1.el6.noarch
--> Running transaction check
---> Package audit-libs.x86_64 0:2.2-2.el6 will be installed
---> Package bash.x86_64 0:4.1.2-15.el6_4 will be installed
---> Package checkpolicy.x86_64 0:2.0.22-1.el6 will be installed
---> Package coreutils.x86_64 0:8.4-31.0.1.el6 will be installed
--> Processing Dependency: coreutils-libs = 8.4-31.0.1.el6 for package: coreutils-8.4-31.0.1.el6.x86_64
--> Processing Dependency: libattr for package: coreutils-8.4-31.0.1.el6.x86_64
--> Processing Dependency: libattr.so.1(ATTR_1.1)(64bit) for package: coreutils-8.4-31.0.1.el6.x86_64
--> Processing Dependency: ncurses for package: coreutils-8.4-31.0.1.el6.x86_64
--> Processing Dependency: gmp for package: coreutils-8.4-31.0.1.el6.x86_64
--> Processing Dependency: libcap for package: coreutils-8.4-31.0.1.el6.x86_64
--> Processing Dependency: /sbin/install-info for package: coreutils-8.4-31.0.1.el6.x86_64
--> Processing Dependency: libcap.so.2()(64bit) for package: coreutils-8.4-31.0.1.el6.x86_64
--> Processing Dependency: libattr.so.1()(64bit) for package: coreutils-8.4-31.0.1.el6.x86_64
--> Processing Dependency: libgmp.so.3()(64bit) for package: coreutils-8.4-31.0.1.el6.x86_64
---> Package cpio.x86_64 0:2.10-11.el6_3 will be installed
---> Package dbus-glib.x86_64 0:0.86-6.el6_4 will be installed
--> Processing Dependency: libexpat.so.1()(64bit) for package: dbus-glib-0.86-6.el6_4.x86_64
---> Package dbus-libs.x86_64 1:1.2.24-7.0.1.el6_3 will be installed
---> Package dhcp-common.x86_64 12:4.1.1-38.P1.0.1.el6 will be installed
---> Package diffutils.x86_64 0:2.8.1-28.el6 will be installed
---> Package ethtool.x86_64 2:3.5-1.el6 will be installed
---> Package findutils.x86_64 1:4.4.2-6.el6 will be installed
---> Package fipscheck-lib.x86_64 0:1.2.0-7.el6 will be installed
--> Processing Dependency: /usr/bin/fipscheck for package: fipscheck-lib-1.2.0-7.el6.x86_64
---> Package gawk.x86_64 0:3.1.7-10.el6 will be installed
---> Package glib2.x86_64 0:2.26.1-3.el6 will be installed
--> Processing Dependency: shared-mime-info for package: glib2-2.26.1-3.el6.x86_64
--> Processing Dependency: libfam.so.0()(64bit) for package: glib2-2.26.1-3.el6.x86_64
---> Package glibc.x86_64 0:2.12-1.132.el6 will be installed
--> Processing Dependency: glibc-common = 2.12-1.132.el6 for package: glibc-2.12-1.132.el6.x86_64
--> Processing Dependency: libfreebl3.so(NSSRAWHASH_3.12.3)(64bit) for package: glibc-2.12-1.132.el6.x86_64
--> Processing Dependency: basesystem for package: glibc-2.12-1.132.el6.x86_64
--> Processing Dependency: libfreebl3.so()(64bit) for package: glibc-2.12-1.132.el6.x86_64
---> Package grep.x86_64 0:2.6.3-4.el6 will be installed
--> Processing Dependency: libpcre.so.0()(64bit) for package: grep-2.6.3-4.el6.x86_64
---> Package iproute.x86_64 0:2.6.32-31.el6 will be installed
--> Processing Dependency: iptables >= 1.4.5 for package: iproute-2.6.32-31.el6.x86_64
--> Processing Dependency: libdb-4.7.so()(64bit) for package: iproute-2.6.32-31.el6.x86_64
--> Processing Dependency: libxtables.so.4()(64bit) for package: iproute-2.6.32-31.el6.x86_64
---> Package iputils.x86_64 0:20071127-17.el6_4.2 will be installed
--> Processing Dependency: libidn.so.11(LIBIDN_1.0)(64bit) for package: iputils-20071127-17.el6_4.2.x86_64
--> Processing Dependency: libidn.so.11()(64bit) for package: iputils-20071127-17.el6_4.2.x86_64
---> Package krb5-libs.x86_64 0:1.10.3-10.el6_4.6 will be installed
--> Processing Dependency: libkeyutils.so.1(KEYUTILS_0.3)(64bit) for package: krb5-libs-1.10.3-10.el6_4.6.x86_64
--> Processing Dependency: libkeyutils.so.1()(64bit) for package: krb5-libs-1.10.3-10.el6_4.6.x86_64
---> Package libacl.x86_64 0:2.2.49-6.el6 will be installed
---> Package libcap-ng.x86_64 0:0.6.4-3.el6_0.1 will be installed
---> Package libcom_err.x86_64 0:1.42.8-1.0.1.el6 will be installed
---> Package libedit.x86_64 0:2.11-4.20080712cvs.1.el6 will be installed
---> Package libgcc.x86_64 0:4.4.7-4.el6 will be installed
---> Package libselinux.x86_64 0:2.0.94-5.3.el6_4.1 will be installed
---> Package libselinux-utils.x86_64 0:2.0.94-5.3.el6_4.1 will be installed
---> Package libsemanage.x86_64 0:2.0.43-4.2.el6 will be installed
--> Processing Dependency: bzip2-libs for package: libsemanage-2.0.43-4.2.el6.x86_64
--> Processing Dependency: libustr-1.0.so.1(USTR_1.0)(64bit) for package: libsemanage-2.0.43-4.2.el6.x86_64
--> Processing Dependency: libustr-1.0.so.1(USTR_1.0.1)(64bit) for package: libsemanage-2.0.43-4.2.el6.x86_64
--> Processing Dependency: libustr-1.0.so.1()(64bit) for package: libsemanage-2.0.43-4.2.el6.x86_64
--> Processing Dependency: libbz2.so.1()(64bit) for package: libsemanage-2.0.43-4.2.el6.x86_64
---> Package libsepol.x86_64 0:2.0.41-4.el6 will be installed
---> Package libuser.x86_64 0:0.56.13-5.el6 will be installed
--> Processing Dependency: libldap-2.4.so.2()(64bit) for package: libuser-0.56.13-5.el6.x86_64
---> Package logrotate.x86_64 0:3.7.8-17.el6 will be installed
---> Package mingetty.x86_64 0:1.08-5.el6 will be installed
---> Package module-init-tools.x86_64 0:3.9-21.0.1.el6_4 will be installed
--> Processing Dependency: binutils for package: module-init-tools-3.9-21.0.1.el6_4.x86_64
---> Package ncurses-libs.x86_64 0:5.7-3.20090208.el6 will be installed
--> Processing Dependency: ncurses-base = 5.7-3.20090208.el6 for package: ncurses-libs-5.7-3.20090208.el6.x86_64
---> Package net-tools.x86_64 0:1.60-110.el6_2 will be installed
---> Package nspr.x86_64 0:4.10.0-1.el6 will be installed
---> Package nss.x86_64 0:3.15.1-15.0.1.el6 will be installed
--> Processing Dependency: nss-softokn(x86-64) >= 3.14.3 for package: nss-3.15.1-15.0.1.el6.x86_64
--> Processing Dependency: nss-util >= 3.15.1 for package: nss-3.15.1-15.0.1.el6.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.15)(64bit) for package: nss-3.15.1-15.0.1.el6.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.12.3)(64bit) for package: nss-3.15.1-15.0.1.el6.x86_64
--> Processing Dependency: nss-system-init for package: nss-3.15.1-15.0.1.el6.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.14)(64bit) for package: nss-3.15.1-15.0.1.el6.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.12)(64bit) for package: nss-3.15.1-15.0.1.el6.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.13)(64bit) for package: nss-3.15.1-15.0.1.el6.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.12.5)(64bit) for package: nss-3.15.1-15.0.1.el6.x86_64
--> Processing Dependency: libnssutil3.so()(64bit) for package: nss-3.15.1-15.0.1.el6.x86_64
--> Processing Dependency: libsoftokn3.so()(64bit) for package: nss-3.15.1-15.0.1.el6.x86_64
---> Package openssh.x86_64 0:5.3p1-94.el6 will be installed
---> Package openssl.x86_64 0:1.0.1e-15.el6 will be installed
--> Processing Dependency: ca-certificates >= 2008-5 for package: openssl-1.0.1e-15.el6.x86_64
--> Processing Dependency: make for package: openssl-1.0.1e-15.el6.x86_64
---> Package oracle-logos.noarch 0:60.0.14-1.0.1.el6 will be installed
---> Package pam.x86_64 0:1.1.1-17.el6 will be installed
--> Processing Dependency: cracklib-dicts >= 2.8 for package: pam-1.1.1-17.el6.x86_64
--> Processing Dependency: cracklib for package: pam-1.1.1-17.el6.x86_64
--> Processing Dependency: libcrack.so.2()(64bit) for package: pam-1.1.1-17.el6.x86_64
---> Package popt.x86_64 0:1.13-7.el6 will be installed
---> Package procps.x86_64 0:3.2.8-25.el6 will be installed
---> Package psmisc.x86_64 0:22.6-15.el6_0.1 will be installed
---> Package pygpgme.x86_64 0:0.1-18.20090824bzr68.el6 will be installed
--> Processing Dependency: libgpgme.so.11(GPGME_1.1)(64bit) for package: pygpgme-0.1-18.20090824bzr68.el6.x86_64
--> Processing Dependency: libgpgme.so.11(GPGME_1.0)(64bit) for package: pygpgme-0.1-18.20090824bzr68.el6.x86_64
--> Processing Dependency: libgpgme.so.11()(64bit) for package: pygpgme-0.1-18.20090824bzr68.el6.x86_64
--> Processing Dependency: libpython2.6.so.1.0()(64bit) for package: pygpgme-0.1-18.20090824bzr68.el6.x86_64
---> Package python.x86_64 0:2.6.6-51.el6 will be installed
---> Package python-iniparse.noarch 0:0.3.1-2.1.el6 will be installed
---> Package python-urlgrabber.noarch 0:3.9.1-9.el6 will be installed
--> Processing Dependency: python-pycurl for package: python-urlgrabber-3.9.1-9.el6.noarch
---> Package redhat-release-server.x86_64 1:6Server-6.5.0.1.0.1.el6 will be installed
---> Package rpm.x86_64 0:4.8.0-37.el6 will be installed
--> Processing Dependency: curl for package: rpm-4.8.0-37.el6.x86_64
--> Processing Dependency: db4-utils for package: rpm-4.8.0-37.el6.x86_64
--> Processing Dependency: librpm.so.1()(64bit) for package: rpm-4.8.0-37.el6.x86_64
--> Processing Dependency: liblua-5.1.so()(64bit) for package: rpm-4.8.0-37.el6.x86_64
--> Processing Dependency: libelf.so.1()(64bit) for package: rpm-4.8.0-37.el6.x86_64
--> Processing Dependency: librpmbuild.so.1()(64bit) for package: rpm-4.8.0-37.el6.x86_64
--> Processing Dependency: librpmio.so.1()(64bit) for package: rpm-4.8.0-37.el6.x86_64
--> Processing Dependency: liblzma.so.0()(64bit) for package: rpm-4.8.0-37.el6.x86_64
--> Processing Dependency: libmagic.so.1()(64bit) for package: rpm-4.8.0-37.el6.x86_64
---> Package rpm-python.x86_64 0:4.8.0-37.el6 will be installed
---> Package sed.x86_64 0:4.2.1-10.el6 will be installed
---> Package shadow-utils.x86_64 2:4.1.4.2-13.el6 will be installed
--> Processing Dependency: setup for package: 2:shadow-utils-4.1.4.2-13.el6.x86_64
---> Package sysvinit-tools.x86_64 0:2.87-5.dsf.el6 will be installed
---> Package tcp_wrappers-libs.x86_64 0:7.6-57.el6 will be installed
---> Package udev.x86_64 0:147-2.51.0.1.el6 will be installed
--> Processing Dependency: MAKEDEV >= 3.11 for package: udev-147-2.51.0.1.el6.x86_64
--> Processing Dependency: hwdata for package: udev-147-2.51.0.1.el6.x86_64
--> Processing Dependency: libusb-0.1.so.4()(64bit) for package: udev-147-2.51.0.1.el6.x86_64
---> Package upstart.x86_64 0:0.6.5-12.el6_4.1 will be installed
--> Processing Dependency: libnih-dbus.so.1(LIBNIH_DBUS_1_0)(64bit) for package: upstart-0.6.5-12.el6_4.1.x86_64
--> Processing Dependency: libnih.so.1(LIBNIH_1_0)(64bit) for package: upstart-0.6.5-12.el6_4.1.x86_64
--> Processing Dependency: libnih.so.1()(64bit) for package: upstart-0.6.5-12.el6_4.1.x86_64
--> Processing Dependency: libnih-dbus.so.1()(64bit) for package: upstart-0.6.5-12.el6_4.1.x86_64
---> Package util-linux-ng.x86_64 0:2.17.2-12.14.el6 will be installed
--> Processing Dependency: libuuid = 2.17.2-12.14.el6 for package: util-linux-ng-2.17.2-12.14.el6.x86_64
--> Processing Dependency: libblkid = 2.17.2-12.14.el6 for package: util-linux-ng-2.17.2-12.14.el6.x86_64
--> Processing Dependency: libuuid.so.1(UUID_1.0)(64bit) for package: util-linux-ng-2.17.2-12.14.el6.x86_64
--> Processing Dependency: libblkid.so.1(BLKID_1.0)(64bit) for package: util-linux-ng-2.17.2-12.14.el6.x86_64
--> Processing Dependency: libutempter.so.0(UTEMPTER_1.1)(64bit) for package: util-linux-ng-2.17.2-12.14.el6.x86_64
--> Processing Dependency: libblkid.so.1(BLKID_2.15)(64bit) for package: util-linux-ng-2.17.2-12.14.el6.x86_64
--> Processing Dependency: libblkid.so.1(BLKID_2.17)(64bit) for package: util-linux-ng-2.17.2-12.14.el6.x86_64
--> Processing Dependency: libutempter.so.0()(64bit) for package: util-linux-ng-2.17.2-12.14.el6.x86_64
--> Processing Dependency: libblkid.so.1()(64bit) for package: util-linux-ng-2.17.2-12.14.el6.x86_64
--> Processing Dependency: libuuid.so.1()(64bit) for package: util-linux-ng-2.17.2-12.14.el6.x86_64
---> Package yum-metadata-parser.x86_64 0:1.1.2-16.el6 will be installed
--> Processing Dependency: libxml2.so.2(LIBXML2_2.4.30)(64bit) for package: yum-metadata-parser-1.1.2-16.el6.x86_64
--> Processing Dependency: libxml2.so.2()(64bit) for package: yum-metadata-parser-1.1.2-16.el6.x86_64
--> Processing Dependency: libsqlite3.so.0()(64bit) for package: yum-metadata-parser-1.1.2-16.el6.x86_64
---> Package zlib.x86_64 0:1.2.3-29.el6 will be installed
--> Running transaction check
---> Package MAKEDEV.x86_64 0:3.24-6.el6 will be installed
---> Package basesystem.noarch 0:10.0-4.0.1.el6 will be installed
--> Processing Dependency: filesystem for package: basesystem-10.0-4.0.1.el6.noarch
---> Package binutils.x86_64 0:2.20.51.0.2-5.36.el6 will be installed
---> Package bzip2-libs.x86_64 0:1.0.5-7.el6_0 will be installed
---> Package ca-certificates.noarch 0:2013.1.94-65.0.el6 will be installed
--> Processing Dependency: p11-kit-trust >= 0.18.4-2 for package: ca-certificates-2013.1.94-65.0.el6.noarch
--> Processing Dependency: p11-kit >= 0.18.4-2 for package: ca-certificates-2013.1.94-65.0.el6.noarch
---> Package coreutils-libs.x86_64 0:8.4-31.0.1.el6 will be installed
---> Package cracklib.x86_64 0:2.8.16-4.el6 will be installed
--> Processing Dependency: gzip for package: cracklib-2.8.16-4.el6.x86_64
---> Package cracklib-dicts.x86_64 0:2.8.16-4.el6 will be installed
---> Package curl.x86_64 0:7.19.7-37.el6_4 will be installed
--> Processing Dependency: libcurl = 7.19.7-37.el6_4 for package: curl-7.19.7-37.el6_4.x86_64
--> Processing Dependency: libssh2.so.1()(64bit) for package: curl-7.19.7-37.el6_4.x86_64
--> Processing Dependency: libcurl.so.4()(64bit) for package: curl-7.19.7-37.el6_4.x86_64
---> Package db4.x86_64 0:4.7.25-18.el6_4 will be installed
---> Package db4-utils.x86_64 0:4.7.25-18.el6_4 will be installed
---> Package elfutils-libelf.x86_64 0:0.152-1.el6 will be installed
---> Package expat.x86_64 0:2.0.1-11.el6_2 will be installed
---> Package file-libs.x86_64 0:5.04-15.el6 will be installed
---> Package fipscheck.x86_64 0:1.2.0-7.el6 will be installed
---> Package gamin.x86_64 0:0.1.10-9.el6 will be installed
---> Package glibc-common.x86_64 0:2.12-1.132.el6 will be installed
--> Processing Dependency: tzdata >= 2003a for package: glibc-common-2.12-1.132.el6.x86_64
---> Package gmp.x86_64 0:4.3.1-7.el6_2.2 will be installed
--> Processing Dependency: libstdc++.so.6(CXXABI_1.3)(64bit) for package: gmp-4.3.1-7.el6_2.2.x86_64
--> Processing Dependency: libstdc++.so.6(GLIBCXX_3.4.11)(64bit) for package: gmp-4.3.1-7.el6_2.2.x86_64
--> Processing Dependency: libstdc++.so.6(GLIBCXX_3.4)(64bit) for package: gmp-4.3.1-7.el6_2.2.x86_64
--> Processing Dependency: libstdc++.so.6()(64bit) for package: gmp-4.3.1-7.el6_2.2.x86_64
---> Package gpgme.x86_64 0:1.1.8-3.el6 will be installed
--> Processing Dependency: gnupg2 for package: gpgme-1.1.8-3.el6.x86_64
--> Processing Dependency: libpth.so.20()(64bit) for package: gpgme-1.1.8-3.el6.x86_64
--> Processing Dependency: libgpg-error.so.0()(64bit) for package: gpgme-1.1.8-3.el6.x86_64
---> Package hwdata.noarch 0:0.233-9.1.el6 will be installed
---> Package info.x86_64 0:4.13a-8.el6 will be installed
---> Package iptables.x86_64 0:1.4.7-11.0.1.el6 will be installed
---> Package keyutils-libs.x86_64 0:1.4-4.el6 will be installed
---> Package libattr.x86_64 0:2.4.44-7.el6 will be installed
---> Package libblkid.x86_64 0:2.17.2-12.14.el6 will be installed
---> Package libcap.x86_64 0:2.16-5.5.el6 will be installed
---> Package libidn.x86_64 0:1.18-2.el6 will be installed
---> Package libnih.x86_64 0:1.0.1-7.el6 will be installed
---> Package libusb.x86_64 0:0.1.12-23.el6 will be installed
---> Package libutempter.x86_64 0:1.1.5-4.1.el6 will be installed
---> Package libuuid.x86_64 0:2.17.2-12.14.el6 will be installed
---> Package libxml2.x86_64 0:2.7.6-14.0.1.el6 will be installed
---> Package lua.x86_64 0:5.1.4-4.1.el6 will be installed
--> Processing Dependency: libreadline.so.6()(64bit) for package: lua-5.1.4-4.1.el6.x86_64
---> Package make.x86_64 1:3.81-20.el6 will be installed
---> Package ncurses.x86_64 0:5.7-3.20090208.el6 will be installed
---> Package ncurses-base.x86_64 0:5.7-3.20090208.el6 will be installed
---> Package nss-softokn.x86_64 0:3.14.3-9.el6 will be installed
---> Package nss-softokn-freebl.x86_64 0:3.14.3-9.el6 will be installed
---> Package nss-sysinit.x86_64 0:3.15.1-15.0.1.el6 will be installed
---> Package nss-util.x86_64 0:3.15.1-3.el6 will be installed
---> Package openldap.x86_64 0:2.4.23-32.el6_4.1 will be installed
--> Processing Dependency: nss-tools for package: openldap-2.4.23-32.el6_4.1.x86_64
--> Processing Dependency: libsasl2.so.2()(64bit) for package: openldap-2.4.23-32.el6_4.1.x86_64
---> Package pcre.x86_64 0:7.8-6.el6 will be installed
---> Package python-libs.x86_64 0:2.6.6-51.el6 will be installed
--> Processing Dependency: libffi.so.5()(64bit) for package: python-libs-2.6.6-51.el6.x86_64
--> Processing Dependency: libgdbm.so.2()(64bit) for package: python-libs-2.6.6-51.el6.x86_64
---> Package python-pycurl.x86_64 0:7.19.0-8.el6 will be installed
---> Package rpm-libs.x86_64 0:4.8.0-37.el6 will be installed
---> Package setup.noarch 0:2.8.14-20.el6_4.1 will be installed
---> Package shared-mime-info.x86_64 0:0.70-4.el6 will be installed
--> Processing Dependency: /usr/bin/pkg-config for package: shared-mime-info-0.70-4.el6.x86_64
--> Processing Dependency: pkgconfig for package: shared-mime-info-0.70-4.el6.x86_64
---> Package sqlite.x86_64 0:3.6.20-1.el6 will be installed
---> Package ustr.x86_64 0:1.0.4-9.1.el6 will be installed
---> Package xz-libs.x86_64 0:4.999.9-0.3.beta.20091007git.el6 will be installed
--> Running transaction check
---> Package cyrus-sasl-lib.x86_64 0:2.1.23-13.el6_3.1 will be installed
---> Package filesystem.x86_64 0:2.4.30-3.el6 will be installed
---> Package gdbm.x86_64 0:1.8.0-36.el6 will be installed
---> Package gnupg2.x86_64 0:2.0.14-6.el6_4 will be installed
--> Processing Dependency: libgcrypt.so.11(GCRYPT_1.2)(64bit) for package: gnupg2-2.0.14-6.el6_4.x86_64
--> Processing Dependency: pinentry for package: gnupg2-2.0.14-6.el6_4.x86_64
--> Processing Dependency: libgcrypt.so.11()(64bit) for package: gnupg2-2.0.14-6.el6_4.x86_64
---> Package gzip.x86_64 0:1.3.12-19.el6_4 will be installed
--> Processing Dependency: less for package: gzip-1.3.12-19.el6_4.x86_64
---> Package libcurl.x86_64 0:7.19.7-37.el6_4 will be installed
---> Package libffi.x86_64 0:3.0.5-3.2.el6 will be installed
---> Package libgpg-error.x86_64 0:1.7-4.el6 will be installed
---> Package libssh2.x86_64 0:1.4.2-1.el6 will be installed
---> Package libstdc++.x86_64 0:4.4.7-4.el6 will be installed
---> Package nss-tools.x86_64 0:3.15.1-15.0.1.el6 will be installed
---> Package p11-kit.x86_64 0:0.18.5-2.el6 will be installed
--> Processing Dependency: libtasn1.so.3(LIBTASN1_0_3)(64bit) for package: p11-kit-0.18.5-2.el6.x86_64
--> Processing Dependency: libtasn1.so.3()(64bit) for package: p11-kit-0.18.5-2.el6.x86_64
---> Package p11-kit-trust.x86_64 0:0.18.5-2.el6 will be installed
---> Package pkgconfig.x86_64 1:0.23-9.1.el6 will be installed
---> Package pth.x86_64 0:2.0.7-9.3.el6 will be installed
---> Package readline.x86_64 0:6.0-4.el6 will be installed
---> Package tzdata.noarch 0:2013g-1.el6 will be installed
--> Running transaction check
---> Package less.x86_64 0:436-10.el6 will be installed
--> Processing Dependency: groff for package: less-436-10.el6.x86_64
---> Package libgcrypt.x86_64 0:1.4.5-11.el6_4 will be installed
---> Package libtasn1.x86_64 0:2.3-3.el6_2.1 will be installed
---> Package pinentry.x86_64 0:0.7.6-6.el6 will be installed
--> Running transaction check
---> Package groff.x86_64 0:1.18.1.4-21.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================
 Package                                     Arch                         Version                                                 Repository                         Size
==========================================================================================================================================================================
Installing:
 chkconfig                                   x86_64                       1.3.49.3-2.el6_4.1                                      ol6_u5_base                       158 k
 dhclient                                    x86_64                       12:4.1.1-38.P1.0.1.el6                                  ol6_u5_base                       317 k
 initscripts                                 x86_64                       9.03.40-2.0.1.el6                                       ol6_u5_base                       940 k
 openssh-clients                             x86_64                       5.3p1-94.el6                                            ol6_u5_base                       401 k
 openssh-server                              x86_64                       5.3p1-94.el6                                            ol6_u5_base                       311 k
 oraclelinux-release                         x86_64                       6:6Server-5.0.2                                         ol6_u5_base                        22 k
 passwd                                      x86_64                       0.77-4.el6_2.2                                          ol6_u5_base                        89 k
 policycoreutils                             x86_64                       2.0.83-19.39.0.1.el6                                    ol6_u5_base                       648 k
 rootfiles                                   noarch                       8.1-6.1.el6                                             ol6_u5_base                       6.3 k
 rsyslog                                     x86_64                       5.8.10-8.0.1.el6                                        ol6_u5_base                       649 k
 vim-minimal                                 x86_64                       2:7.2.411-1.8.el6                                       ol6_u5_base                       363 k
 yum                                         noarch                       3.2.29-40.0.1.el6                                       ol6_u5_base                       995 k
Installing for dependencies:
 MAKEDEV                                     x86_64                       3.24-6.el6                                              ol6_u5_base                        88 k
 audit-libs                                  x86_64                       2.2-2.el6                                               ol6_u5_base                        60 k
 basesystem                                  noarch                       10.0-4.0.1.el6                                          ol6_u5_base                       4.3 k
 bash                                        x86_64                       4.1.2-15.el6_4                                          ol6_u5_base                       904 k
 binutils                                    x86_64                       2.20.51.0.2-5.36.el6                                    ol6_u5_base                       2.8 M
 bzip2-libs                                  x86_64                       1.0.5-7.el6_0                                           ol6_u5_base                        36 k
 ca-certificates                             noarch                       2013.1.94-65.0.el6                                      ol6_u5_base                       1.1 M
 checkpolicy                                 x86_64                       2.0.22-1.el6                                            ol6_u5_base                       189 k
 coreutils                                   x86_64                       8.4-31.0.1.el6                                          ol6_u5_base                       3.0 M
 coreutils-libs                              x86_64                       8.4-31.0.1.el6                                          ol6_u5_base                        49 k
 cpio                                        x86_64                       2.10-11.el6_3                                           ol6_u5_base                       191 k
 cracklib                                    x86_64                       2.8.16-4.el6                                            ol6_u5_base                        70 k
 cracklib-dicts                              x86_64                       2.8.16-4.el6                                            ol6_u5_base                       3.6 M
 curl                                        x86_64                       7.19.7-37.el6_4                                         ol6_u5_base                       193 k
 cyrus-sasl-lib                              x86_64                       2.1.23-13.el6_3.1                                       ol6_u5_base                       135 k
 db4                                         x86_64                       4.7.25-18.el6_4                                         ol6_u5_base                       562 k
 db4-utils                                   x86_64                       4.7.25-18.el6_4                                         ol6_u5_base                       130 k
 dbus-glib                                   x86_64                       0.86-6.el6_4                                            ol6_u5_base                       169 k
 dbus-libs                                   x86_64                       1:1.2.24-7.0.1.el6_3                                    ol6_u5_base                       126 k
 dhcp-common                                 x86_64                       12:4.1.1-38.P1.0.1.el6                                  ol6_u5_base                       141 k
 diffutils                                   x86_64                       2.8.1-28.el6                                            ol6_u5_base                       198 k
 elfutils-libelf                             x86_64                       0.152-1.el6                                             ol6_u5_base                       169 k
 ethtool                                     x86_64                       2:3.5-1.el6                                             ol6_u5_base                       100 k
 expat                                       x86_64                       2.0.1-11.el6_2                                          ol6_u5_base                        76 k
 file-libs                                   x86_64                       5.04-15.el6                                             ol6_u5_base                       310 k
 filesystem                                  x86_64                       2.4.30-3.el6                                            ol6_u5_base                       1.0 M
 findutils                                   x86_64                       1:4.4.2-6.el6                                           ol6_u5_base                       447 k
 fipscheck                                   x86_64                       1.2.0-7.el6                                             ol6_u5_base                        14 k
 fipscheck-lib                               x86_64                       1.2.0-7.el6                                             ol6_u5_base                       7.8 k
 gamin                                       x86_64                       0.1.10-9.el6                                            ol6_u5_base                       122 k
 gawk                                        x86_64                       3.1.7-10.el6                                            ol6_u5_base                       774 k
 gdbm                                        x86_64                       1.8.0-36.el6                                            ol6_u5_base                        28 k
 glib2                                       x86_64                       2.26.1-3.el6                                            ol6_u5_base                       1.6 M
 glibc                                       x86_64                       2.12-1.132.el6                                          ol6_u5_base                       3.8 M
 glibc-common                                x86_64                       2.12-1.132.el6                                          ol6_u5_base                        14 M
 gmp                                         x86_64                       4.3.1-7.el6_2.2                                         ol6_u5_base                       206 k
 gnupg2                                      x86_64                       2.0.14-6.el6_4                                          ol6_u5_base                       1.6 M
 gpgme                                       x86_64                       1.1.8-3.el6                                             ol6_u5_base                       144 k
 grep                                        x86_64                       2.6.3-4.el6                                             ol6_u5_base                       228 k
 groff                                       x86_64                       1.18.1.4-21.el6                                         ol6_u5_base                       1.5 M
 gzip                                        x86_64                       1.3.12-19.el6_4                                         ol6_u5_base                       115 k
 hwdata                                      noarch                       0.233-9.1.el6                                           ol6_u5_base                       1.2 M
 info                                        x86_64                       4.13a-8.el6                                             ol6_u5_base                       177 k
 iproute                                     x86_64                       2.6.32-31.el6                                           ol6_u5_base                       364 k
 iptables                                    x86_64                       1.4.7-11.0.1.el6                                        ol6_u5_base                       252 k
 iputils                                     x86_64                       20071127-17.el6_4.2                                     ol6_u5_base                       120 k
 keyutils-libs                               x86_64                       1.4-4.el6                                               ol6_u5_base                        19 k
 krb5-libs                                   x86_64                       1.10.3-10.el6_4.6                                       ol6_u5_base                       760 k
 less                                        x86_64                       436-10.el6                                              ol6_u5_base                       106 k
 libacl                                      x86_64                       2.2.49-6.el6                                            ol6_u5_base                        23 k
 libattr                                     x86_64                       2.4.44-7.el6                                            ol6_u5_base                        15 k
 libblkid                                    x86_64                       2.17.2-12.14.el6                                        ol6_u5_base                       114 k
 libcap                                      x86_64                       2.16-5.5.el6                                            ol6_u5_base                        31 k
 libcap-ng                                   x86_64                       0.6.4-3.el6_0.1                                         ol6_u5_base                        21 k
 libcom_err                                  x86_64                       1.42.8-1.0.1.el6                                        ol6_u5_base                        36 k
 libcurl                                     x86_64                       7.19.7-37.el6_4                                         ol6_u5_base                       164 k
 libedit                                     x86_64                       2.11-4.20080712cvs.1.el6                                ol6_u5_base                        74 k
 libffi                                      x86_64                       3.0.5-3.2.el6                                           ol6_u5_base                        25 k
 libgcc                                      x86_64                       4.4.7-4.el6                                             ol6_u5_base                       100 k
 libgcrypt                                   x86_64                       1.4.5-11.el6_4                                          ol6_u5_base                       228 k
 libgpg-error                                x86_64                       1.7-4.el6                                               ol6_u5_base                        59 k
 libidn                                      x86_64                       1.18-2.el6                                              ol6_u5_base                       205 k
 libnih                                      x86_64                       1.0.1-7.el6                                             ol6_u5_base                       137 k
 libselinux                                  x86_64                       2.0.94-5.3.el6_4.1                                      ol6_u5_base                       107 k
 libselinux-utils                            x86_64                       2.0.94-5.3.el6_4.1                                      ol6_u5_base                        80 k
 libsemanage                                 x86_64                       2.0.43-4.2.el6                                          ol6_u5_base                       103 k
 libsepol                                    x86_64                       2.0.41-4.el6                                            ol6_u5_base                       128 k
 libssh2                                     x86_64                       1.4.2-1.el6                                             ol6_u5_base                       122 k
 libstdc++                                   x86_64                       4.4.7-4.el6                                             ol6_u5_base                       292 k
 libtasn1                                    x86_64                       2.3-3.el6_2.1                                           ol6_u5_base                       237 k
 libusb                                      x86_64                       0.1.12-23.el6                                           ol6_u5_base                        27 k
 libuser                                     x86_64                       0.56.13-5.el6                                           ol6_u5_base                       368 k
 libutempter                                 x86_64                       1.1.5-4.1.el6                                           ol6_u5_base                        22 k
 libuuid                                     x86_64                       2.17.2-12.14.el6                                        ol6_u5_base                        67 k
 libxml2                                     x86_64                       2.7.6-14.0.1.el6                                        ol6_u5_base                       799 k
 logrotate                                   x86_64                       3.7.8-17.el6                                            ol6_u5_base                        55 k
 lua                                         x86_64                       5.1.4-4.1.el6                                           ol6_u5_base                       184 k
 make                                        x86_64                       1:3.81-20.el6                                           ol6_u5_base                       388 k
 mingetty                                    x86_64                       1.08-5.el6                                              ol6_u5_base                        20 k
 module-init-tools                           x86_64                       3.9-21.0.1.el6_4                                        ol6_u5_base                       463 k
 ncurses                                     x86_64                       5.7-3.20090208.el6                                      ol6_u5_base                       267 k
 ncurses-base                                x86_64                       5.7-3.20090208.el6                                      ol6_u5_base                        60 k
 ncurses-libs                                x86_64                       5.7-3.20090208.el6                                      ol6_u5_base                       245 k
 net-tools                                   x86_64                       1.60-110.el6_2                                          ol6_u5_base                       268 k
 nspr                                        x86_64                       4.10.0-1.el6                                            ol6_u5_base                       112 k
 nss                                         x86_64                       3.15.1-15.0.1.el6                                       ol6_u5_base                       820 k
 nss-softokn                                 x86_64                       3.14.3-9.el6                                            ol6_u5_base                       264 k
 nss-softokn-freebl                          x86_64                       3.14.3-9.el6                                            ol6_u5_base                       157 k
 nss-sysinit                                 x86_64                       3.15.1-15.0.1.el6                                       ol6_u5_base                        39 k
 nss-tools                                   x86_64                       3.15.1-15.0.1.el6                                       ol6_u5_base                       356 k
 nss-util                                    x86_64                       3.15.1-3.el6                                            ol6_u5_base                        63 k
 openldap                                    x86_64                       2.4.23-32.el6_4.1                                       ol6_u5_base                       265 k
 openssh                                     x86_64                       5.3p1-94.el6                                            ol6_u5_base                       257 k
 openssl                                     x86_64                       1.0.1e-15.el6                                           ol6_u5_base                       1.5 M
 oracle-logos                                noarch                       60.0.14-1.0.1.el6                                       ol6_u5_base                        12 M
 p11-kit                                     x86_64                       0.18.5-2.el6                                            ol6_u5_base                        93 k
 p11-kit-trust                               x86_64                       0.18.5-2.el6                                            ol6_u5_base                        71 k
 pam                                         x86_64                       1.1.1-17.el6                                            ol6_u5_base                       658 k
 pcre                                        x86_64                       7.8-6.el6                                               ol6_u5_base                       194 k
 pinentry                                    x86_64                       0.7.6-6.el6                                             ol6_u5_base                        65 k
 pkgconfig                                   x86_64                       1:0.23-9.1.el6                                          ol6_u5_base                        70 k
 popt                                        x86_64                       1.13-7.el6                                              ol6_u5_base                        38 k
 procps                                      x86_64                       3.2.8-25.el6                                            ol6_u5_base                       211 k
 psmisc                                      x86_64                       22.6-15.el6_0.1                                         ol6_u5_base                        80 k
 pth                                         x86_64                       2.0.7-9.3.el6                                           ol6_u5_base                        85 k
 pygpgme                                     x86_64                       0.1-18.20090824bzr68.el6                                ol6_u5_base                        70 k
 python                                      x86_64                       2.6.6-51.el6                                            ol6_u5_base                        73 k
 python-iniparse                             noarch                       0.3.1-2.1.el6                                           ol6_u5_base                        36 k
 python-libs                                 x86_64                       2.6.6-51.el6                                            ol6_u5_base                       5.3 M
 python-pycurl                               x86_64                       7.19.0-8.el6                                            ol6_u5_base                        76 k
 python-urlgrabber                           noarch                       3.9.1-9.el6                                             ol6_u5_base                        85 k
 readline                                    x86_64                       6.0-4.el6                                               ol6_u5_base                       178 k
 redhat-release-server                       x86_64                       1:6Server-6.5.0.1.0.1.el6                               ol6_u5_base                       2.6 k
 rpm                                         x86_64                       4.8.0-37.el6                                            ol6_u5_base                       901 k
 rpm-libs                                    x86_64                       4.8.0-37.el6                                            ol6_u5_base                       312 k
 rpm-python                                  x86_64                       4.8.0-37.el6                                            ol6_u5_base                        56 k
 sed                                         x86_64                       4.2.1-10.el6                                            ol6_u5_base                       215 k
 setup                                       noarch                       2.8.14-20.el6_4.1                                       ol6_u5_base                       150 k
 shadow-utils                                x86_64                       2:4.1.4.2-13.el6                                        ol6_u5_base                       896 k
 shared-mime-info                            x86_64                       0.70-4.el6                                              ol6_u5_base                       208 k
 sqlite                                      x86_64                       3.6.20-1.el6                                            ol6_u5_base                       301 k
 sysvinit-tools                              x86_64                       2.87-5.dsf.el6                                          ol6_u5_base                        59 k
 tcp_wrappers-libs                           x86_64                       7.6-57.el6                                              ol6_u5_base                        62 k
 tzdata                                      noarch                       2013g-1.el6                                             ol6_u5_base                       457 k
 udev                                        x86_64                       147-2.51.0.1.el6                                        ol6_u5_base                       348 k
 upstart                                     x86_64                       0.6.5-12.el6_4.1                                        ol6_u5_base                       176 k
 ustr                                        x86_64                       1.0.4-9.1.el6                                           ol6_u5_base                        85 k
 util-linux-ng                               x86_64                       2.17.2-12.14.el6                                        ol6_u5_base                       1.5 M
 xz-libs                                     x86_64                       4.999.9-0.3.beta.20091007git.el6                        ol6_u5_base                        89 k
 yum-metadata-parser                         x86_64                       1.1.2-16.el6                                            ol6_u5_base                        26 k
 zlib                                        x86_64                       1.2.3-29.el6                                            ol6_u5_base                        72 k

Transaction Summary
==========================================================================================================================================================================
Install     143 Packages

Total download size: 82 M
Installed size: 303 M
Downloading Packages:
(1/143): MAKEDEV-3.24-6.el6.x86_64.rpm                                                                                                             |  88 kB     00:00    
(2/143): audit-libs-2.2-2.el6.x86_64.rpm                                                                                                           |  60 kB     00:00    
(3/143): basesystem-10.0-4.0.1.el6.noarch.rpm                                                                                                      | 4.3 kB     00:00    
(4/143): bash-4.1.2-15.el6_4.x86_64.rpm                                                                                                            | 904 kB     00:00    
(5/143): binutils-2.20.51.0.2-5.36.el6.x86_64.rpm                                                                                                  | 2.8 MB     00:01    
(6/143): bzip2-libs-1.0.5-7.el6_0.x86_64.rpm                                                                                                       |  36 kB     00:00    
(7/143): ca-certificates-2013.1.94-65.0.el6.noarch.rpm                                                                                             | 1.1 MB     00:00    
(8/143): checkpolicy-2.0.22-1.el6.x86_64.rpm                                                                                                       | 189 kB     00:00    
(9/143): chkconfig-1.3.49.3-2.el6_4.1.x86_64.rpm                                                                                                   | 158 kB     00:00    
(10/143): coreutils-8.4-31.0.1.el6.x86_64.rpm                                                                                                      | 3.0 MB     00:02    
(11/143): coreutils-libs-8.4-31.0.1.el6.x86_64.rpm                                                                                                 |  49 kB     00:00    
(12/143): cpio-2.10-11.el6_3.x86_64.rpm                                                                                                            | 191 kB     00:00    
(13/143): cracklib-2.8.16-4.el6.x86_64.rpm                                                                                                         |  70 kB     00:00    
(14/143): cracklib-dicts-2.8.16-4.el6.x86_64.rpm                                                                                                   | 3.6 MB     00:03    
(15/143): curl-7.19.7-37.el6_4.x86_64.rpm                                                                                                          | 193 kB     00:00    
(16/143): cyrus-sasl-lib-2.1.23-13.el6_3.1.x86_64.rpm                                                                                              | 135 kB     00:00    
(17/143): db4-4.7.25-18.el6_4.x86_64.rpm                                                                                                           | 562 kB     00:00    
(18/143): db4-utils-4.7.25-18.el6_4.x86_64.rpm                                                                                                     | 130 kB     00:00    
(19/143): dbus-glib-0.86-6.el6_4.x86_64.rpm                                                                                                        | 169 kB     00:00    
(20/143): dbus-libs-1.2.24-7.0.1.el6_3.x86_64.rpm                                                                                                  | 126 kB     00:00    
(21/143): dhclient-4.1.1-38.P1.0.1.el6.x86_64.rpm                                                                                                  | 317 kB     00:00    
(22/143): dhcp-common-4.1.1-38.P1.0.1.el6.x86_64.rpm                                                                                               | 141 kB     00:00    
(23/143): diffutils-2.8.1-28.el6.x86_64.rpm                                                                                                        | 198 kB     00:00    
(24/143): elfutils-libelf-0.152-1.el6.x86_64.rpm                                                                                                   | 169 kB     00:00    
(25/143): ethtool-3.5-1.el6.x86_64.rpm                                                                                                             | 100 kB     00:00    
(26/143): expat-2.0.1-11.el6_2.x86_64.rpm                                                                                                          |  76 kB     00:00    
(27/143): file-libs-5.04-15.el6.x86_64.rpm                                                                                                         | 310 kB     00:00    
(28/143): filesystem-2.4.30-3.el6.x86_64.rpm                                                                                                       | 1.0 MB     00:00    
(29/143): findutils-4.4.2-6.el6.x86_64.rpm                                                                                                         | 447 kB     00:00    
(30/143): fipscheck-1.2.0-7.el6.x86_64.rpm                                                                                                         |  14 kB     00:00    
(31/143): fipscheck-lib-1.2.0-7.el6.x86_64.rpm                                                                                                     | 7.8 kB     00:00    
(32/143): gamin-0.1.10-9.el6.x86_64.rpm                                                                                                            | 122 kB     00:00    
(33/143): gawk-3.1.7-10.el6.x86_64.rpm                                                                                                             | 774 kB     00:00    
(34/143): gdbm-1.8.0-36.el6.x86_64.rpm                                                                                                             |  28 kB     00:00    
(35/143): glib2-2.26.1-3.el6.x86_64.rpm                                                                                                            | 1.6 MB     00:01    
(36/143): glibc-2.12-1.132.el6.x86_64.rpm                                                                                                          | 3.8 MB     00:02    
(37/143): glibc-common-2.12-1.132.el6.x86_64.rpm                                                                                                   |  14 MB     00:09    
(38/143): gmp-4.3.1-7.el6_2.2.x86_64.rpm                                                                                                           | 206 kB     00:00    
(39/143): gnupg2-2.0.14-6.el6_4.x86_64.rpm                                                                                                         | 1.6 MB     00:01    
(40/143): gpgme-1.1.8-3.el6.x86_64.rpm                                                                                                             | 144 kB     00:00    
(41/143): grep-2.6.3-4.el6.x86_64.rpm                                                                                                              | 228 kB     00:00    
(42/143): groff-1.18.1.4-21.el6.x86_64.rpm                                                                                                         | 1.5 MB     00:01    
(43/143): gzip-1.3.12-19.el6_4.x86_64.rpm                                                                                                          | 115 kB     00:00    
(44/143): hwdata-0.233-9.1.el6.noarch.rpm                                                                                                          | 1.2 MB     00:01    
(45/143): info-4.13a-8.el6.x86_64.rpm                                                                                                              | 177 kB     00:00    
(46/143): initscripts-9.03.40-2.0.1.el6.x86_64.rpm                                                                                                 | 940 kB     00:01    
(47/143): iproute-2.6.32-31.el6.x86_64.rpm                                                                                                         | 364 kB     00:00    
(48/143): iptables-1.4.7-11.0.1.el6.x86_64.rpm                                                                                                     | 252 kB     00:00    
(49/143): iputils-20071127-17.el6_4.2.x86_64.rpm                                                                                                   | 120 kB     00:00    
(50/143): keyutils-libs-1.4-4.el6.x86_64.rpm                                                                                                       |  19 kB     00:00    
(51/143): krb5-libs-1.10.3-10.el6_4.6.x86_64.rpm                                                                                                   | 760 kB     00:01    
(52/143): less-436-10.el6.x86_64.rpm                                                                                                               | 106 kB     00:00    
(53/143): libacl-2.2.49-6.el6.x86_64.rpm                                                                                                           |  23 kB     00:00    
(54/143): libattr-2.4.44-7.el6.x86_64.rpm                                                                                                          |  15 kB     00:00    
(55/143): libblkid-2.17.2-12.14.el6.x86_64.rpm                                                                                                     | 114 kB     00:00    
(56/143): libcap-2.16-5.5.el6.x86_64.rpm                                                                                                           |  31 kB     00:00    
(57/143): libcap-ng-0.6.4-3.el6_0.1.x86_64.rpm                                                                                                     |  21 kB     00:00    
(58/143): libcom_err-1.42.8-1.0.1.el6.x86_64.rpm                                                                                                   |  36 kB     00:00    
(59/143): libcurl-7.19.7-37.el6_4.x86_64.rpm                                                                                                       | 164 kB     00:00    
(60/143): libedit-2.11-4.20080712cvs.1.el6.x86_64.rpm                                                                                              |  74 kB     00:00    
(61/143): libffi-3.0.5-3.2.el6.x86_64.rpm                                                                                                          |  25 kB     00:00    
(62/143): libgcc-4.4.7-4.el6.x86_64.rpm                                                                                                            | 100 kB     00:00    
(63/143): libgcrypt-1.4.5-11.el6_4.x86_64.rpm                                                                                                      | 228 kB     00:00    
(64/143): libgpg-error-1.7-4.el6.x86_64.rpm                                                                                                        |  59 kB     00:00    
(65/143): libidn-1.18-2.el6.x86_64.rpm                                                                                                             | 205 kB     00:00    
(66/143): libnih-1.0.1-7.el6.x86_64.rpm                                                                                                            | 137 kB     00:00    
(67/143): libselinux-2.0.94-5.3.el6_4.1.x86_64.rpm                                                                                                 | 107 kB     00:00    
(68/143): libselinux-utils-2.0.94-5.3.el6_4.1.x86_64.rpm                                                                                           |  80 kB     00:00    
(69/143): libsemanage-2.0.43-4.2.el6.x86_64.rpm                                                                                                    | 103 kB     00:00    
(70/143): libsepol-2.0.41-4.el6.x86_64.rpm                                                                                                         | 128 kB     00:00    
(71/143): libssh2-1.4.2-1.el6.x86_64.rpm                                                                                                           | 122 kB     00:00    
(72/143): libstdc++-4.4.7-4.el6.x86_64.rpm                                                                                                         | 292 kB     00:00    
(73/143): libtasn1-2.3-3.el6_2.1.x86_64.rpm                                                                                                        | 237 kB     00:00    
(74/143): libusb-0.1.12-23.el6.x86_64.rpm                                                                                                          |  27 kB     00:00    
(75/143): libuser-0.56.13-5.el6.x86_64.rpm                                                                                                         | 368 kB     00:00    
(76/143): libutempter-1.1.5-4.1.el6.x86_64.rpm                                                                                                     |  22 kB     00:00    
(77/143): libuuid-2.17.2-12.14.el6.x86_64.rpm                                                                                                      |  67 kB     00:00    
(78/143): libxml2-2.7.6-14.0.1.el6.x86_64.rpm                                                                                                      | 799 kB     00:00    
(79/143): logrotate-3.7.8-17.el6.x86_64.rpm                                                                                                        |  55 kB     00:00    
(80/143): lua-5.1.4-4.1.el6.x86_64.rpm                                                                                                             | 184 kB     00:00    
(81/143): make-3.81-20.el6.x86_64.rpm                                                                                                              | 388 kB     00:00    
(82/143): mingetty-1.08-5.el6.x86_64.rpm                                                                                                           |  20 kB     00:00    
(83/143): module-init-tools-3.9-21.0.1.el6_4.x86_64.rpm                                                                                            | 463 kB     00:00    
(84/143): ncurses-5.7-3.20090208.el6.x86_64.rpm                                                                                                    | 267 kB     00:00    
(85/143): ncurses-base-5.7-3.20090208.el6.x86_64.rpm                                                                                               |  60 kB     00:00    
(86/143): ncurses-libs-5.7-3.20090208.el6.x86_64.rpm                                                                                               | 245 kB     00:00    
(87/143): net-tools-1.60-110.el6_2.x86_64.rpm                                                                                                      | 268 kB     00:00    
(88/143): nspr-4.10.0-1.el6.x86_64.rpm                                                                                                             | 112 kB     00:00    
(89/143): nss-3.15.1-15.0.1.el6.x86_64.rpm                                                                                                         | 820 kB     00:00    
(90/143): nss-softokn-3.14.3-9.el6.x86_64.rpm                                                                                                      | 264 kB     00:00    
(91/143): nss-softokn-freebl-3.14.3-9.el6.x86_64.rpm                                                                                               | 157 kB     00:00    
(92/143): nss-sysinit-3.15.1-15.0.1.el6.x86_64.rpm                                                                                                 |  39 kB     00:00    
(93/143): nss-tools-3.15.1-15.0.1.el6.x86_64.rpm                                                                                                   | 356 kB     00:00    
(94/143): nss-util-3.15.1-3.el6.x86_64.rpm                                                                                                         |  63 kB     00:00    
(95/143): openldap-2.4.23-32.el6_4.1.x86_64.rpm                                                                                                    | 265 kB     00:00    
(96/143): openssh-5.3p1-94.el6.x86_64.rpm                                                                                                          | 257 kB     00:00    
(97/143): openssh-clients-5.3p1-94.el6.x86_64.rpm                                                                                                  | 401 kB     00:00    
(98/143): openssh-server-5.3p1-94.el6.x86_64.rpm                                                                                                   | 311 kB     00:00    
(99/143): openssl-1.0.1e-15.el6.x86_64.rpm                                                                                                         | 1.5 MB     00:00    
(100/143): oracle-logos-60.0.14-1.0.1.el6.noarch.rpm                                                                                               |  12 MB     00:08    
(101/143): oraclelinux-release-6Server-5.0.2.x86_64.rpm                                                                                            |  22 kB     00:00    
(102/143): p11-kit-0.18.5-2.el6.x86_64.rpm                                                                                                         |  93 kB     00:00    
(103/143): p11-kit-trust-0.18.5-2.el6.x86_64.rpm                                                                                                   |  71 kB     00:00    
(104/143): pam-1.1.1-17.el6.x86_64.rpm                                                                                                             | 658 kB     00:00    
(105/143): passwd-0.77-4.el6_2.2.x86_64.rpm                                                                                                        |  89 kB     00:00    
(106/143): pcre-7.8-6.el6.x86_64.rpm                                                                                                               | 194 kB     00:00    
(107/143): pinentry-0.7.6-6.el6.x86_64.rpm                                                                                                         |  65 kB     00:00    
(108/143): pkgconfig-0.23-9.1.el6.x86_64.rpm                                                                                                       |  70 kB     00:00    
(109/143): policycoreutils-2.0.83-19.39.0.1.el6.x86_64.rpm                                                                                         | 648 kB     00:00    
(110/143): popt-1.13-7.el6.x86_64.rpm                                                                                                              |  38 kB     00:00    
(111/143): procps-3.2.8-25.el6.x86_64.rpm                                                                                                          | 211 kB     00:00    
(112/143): psmisc-22.6-15.el6_0.1.x86_64.rpm                                                                                                       |  80 kB     00:00    
(113/143): pth-2.0.7-9.3.el6.x86_64.rpm                                                                                                            |  85 kB     00:00    
(114/143): pygpgme-0.1-18.20090824bzr68.el6.x86_64.rpm                                                                                             |  70 kB     00:00    
(115/143): python-2.6.6-51.el6.x86_64.rpm                                                                                                          |  73 kB     00:00    
(116/143): python-iniparse-0.3.1-2.1.el6.noarch.rpm                                                                                                |  36 kB     00:00    
(117/143): python-libs-2.6.6-51.el6.x86_64.rpm                                                                                                     | 5.3 MB     00:04    
(118/143): python-pycurl-7.19.0-8.el6.x86_64.rpm                                                                                                   |  76 kB     00:00    
(119/143): python-urlgrabber-3.9.1-9.el6.noarch.rpm                                                                                                |  85 kB     00:00    
(120/143): readline-6.0-4.el6.x86_64.rpm                                                                                                           | 178 kB     00:00    
(121/143): redhat-release-server-6Server-6.5.0.1.0.1.el6.x86_64.rpm                                                                                | 2.6 kB     00:00    
(122/143): rootfiles-8.1-6.1.el6.noarch.rpm                                                                                                        | 6.3 kB     00:00    
(123/143): rpm-4.8.0-37.el6.x86_64.rpm                                                                                                             | 901 kB     00:00    
(124/143): rpm-libs-4.8.0-37.el6.x86_64.rpm                                                                                                        | 312 kB     00:00    
(125/143): rpm-python-4.8.0-37.el6.x86_64.rpm                                                                                                      |  56 kB     00:00    
(126/143): rsyslog-5.8.10-8.0.1.el6.x86_64.rpm                                                                                                     | 649 kB     00:00    
(127/143): sed-4.2.1-10.el6.x86_64.rpm                                                                                                             | 215 kB     00:00    
(128/143): setup-2.8.14-20.el6_4.1.noarch.rpm                                                                                                      | 150 kB     00:00    
(129/143): shadow-utils-4.1.4.2-13.el6.x86_64.rpm                                                                                                  | 896 kB     00:00    
(130/143): shared-mime-info-0.70-4.el6.x86_64.rpm                                                                                                  | 208 kB     00:00    
(131/143): sqlite-3.6.20-1.el6.x86_64.rpm                                                                                                          | 301 kB     00:00    
(132/143): sysvinit-tools-2.87-5.dsf.el6.x86_64.rpm                                                                                                |  59 kB     00:00    
(133/143): tcp_wrappers-libs-7.6-57.el6.x86_64.rpm                                                                                                 |  62 kB     00:00    
(134/143): tzdata-2013g-1.el6.noarch.rpm                                                                                                           | 457 kB     00:00    
(135/143): udev-147-2.51.0.1.el6.x86_64.rpm                                                                                                        | 348 kB     00:00    
(136/143): upstart-0.6.5-12.el6_4.1.x86_64.rpm                                                                                                     | 176 kB     00:00    
(137/143): ustr-1.0.4-9.1.el6.x86_64.rpm                                                                                                           |  85 kB     00:00    
(138/143): util-linux-ng-2.17.2-12.14.el6.x86_64.rpm                                                                                               | 1.5 MB     00:00    
(139/143): vim-minimal-7.2.411-1.8.el6.x86_64.rpm                                                                                                  | 363 kB     00:00    
(140/143): xz-libs-4.999.9-0.3.beta.20091007git.el6.x86_64.rpm                                                                                     |  89 kB     00:00    
(141/143): yum-3.2.29-40.0.1.el6.noarch.rpm                                                                                                        | 995 kB     00:00    
(142/143): yum-metadata-parser-1.1.2-16.el6.x86_64.rpm                                                                                             |  26 kB     00:00    
(143/143): zlib-1.2.3-29.el6.x86_64.rpm                                                                                                            |  72 kB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                     798 kB/s |  82 MB     01:45    
Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : libgcc-4.4.7-4.el6.x86_64                                                                                                                            1/143
  Installing : setup-2.8.14-20.el6_4.1.noarch                                                                                                                       2/143
  Installing : filesystem-2.4.30-3.el6.x86_64                                                                                                                       3/143
  Installing : basesystem-10.0-4.0.1.el6.noarch                                                                                                                     4/143
  Installing : tzdata-2013g-1.el6.noarch                                                                                                                            5/143
  Installing : 1:redhat-release-server-6Server-6.5.0.1.0.1.el6.x86_64                                                                                               6/143
  Installing : 12:dhcp-common-4.1.1-38.P1.0.1.el6.x86_64                                                                                                            7/143
  Installing : ncurses-base-5.7-3.20090208.el6.x86_64                                                                                                               8/143
  Installing : nss-softokn-freebl-3.14.3-9.el6.x86_64                                                                                                               9/143
  Installing : glibc-common-2.12-1.132.el6.x86_64                                                                                                                  10/143
  Installing : glibc-2.12-1.132.el6.x86_64                                                                                                                         11/143
  Installing : ncurses-libs-5.7-3.20090208.el6.x86_64                                                                                                              12/143
  Installing : bash-4.1.2-15.el6_4.x86_64                                                                                                                          13/143
  Installing : libattr-2.4.44-7.el6.x86_64                                                                                                                         14/143
  Installing : libcap-2.16-5.5.el6.x86_64                                                                                                                          15/143
  Installing : zlib-1.2.3-29.el6.x86_64                                                                                                                            16/143
  Installing : info-4.13a-8.el6.x86_64                                                                                                                             17/143
  Installing : nspr-4.10.0-1.el6.x86_64                                                                                                                            18/143
  Installing : popt-1.13-7.el6.x86_64                                                                                                                              19/143
  Installing : chkconfig-1.3.49.3-2.el6_4.1.x86_64                                                                                                                 20/143
  Installing : libacl-2.2.49-6.el6.x86_64                                                                                                                          21/143
  Installing : db4-4.7.25-18.el6_4.x86_64                                                                                                                          22/143
  Installing : audit-libs-2.2-2.el6.x86_64                                                                                                                         23/143
  Installing : nss-util-3.15.1-3.el6.x86_64                                                                                                                        24/143
  Installing : libcom_err-1.42.8-1.0.1.el6.x86_64                                                                                                                  25/143
  Installing : libsepol-2.0.41-4.el6.x86_64                                                                                                                        26/143
  Installing : libselinux-2.0.94-5.3.el6_4.1.x86_64                                                                                                                27/143
  Installing : bzip2-libs-1.0.5-7.el6_0.x86_64                                                                                                                     28/143
  Installing : sed-4.2.1-10.el6.x86_64                                                                                                                             29/143
  Installing : 2:shadow-utils-4.1.4.2-13.el6.x86_64                                                                                                                30/143
  Installing : readline-6.0-4.el6.x86_64                                                                                                                           31/143
  Installing : 1:dbus-libs-1.2.24-7.0.1.el6_3.x86_64                                                                                                               32/143
  Installing : libstdc++-4.4.7-4.el6.x86_64                                                                                                                        33/143
  Installing : lua-5.1.4-4.1.el6.x86_64                                                                                                                            34/143
  Installing : sqlite-3.6.20-1.el6.x86_64                                                                                                                          35/143
  Installing : libidn-1.18-2.el6.x86_64                                                                                                                            36/143
  Installing : file-libs-5.04-15.el6.x86_64                                                                                                                        37/143
  Installing : xz-libs-4.999.9-0.3.beta.20091007git.el6.x86_64                                                                                                     38/143
  Installing : libgpg-error-1.7-4.el6.x86_64                                                                                                                       39/143
  Installing : elfutils-libelf-0.152-1.el6.x86_64                                                                                                                  40/143
  Installing : pcre-7.8-6.el6.x86_64                                                                                                                               41/143
  Installing : grep-2.6.3-4.el6.x86_64                                                                                                                             42/143
  Installing : gawk-3.1.7-10.el6.x86_64                                                                                                                            43/143
  Installing : libxml2-2.7.6-14.0.1.el6.x86_64                                                                                                                     44/143
  Installing : pth-2.0.7-9.3.el6.x86_64                                                                                                                            45/143
  Installing : libtasn1-2.3-3.el6_2.1.x86_64                                                                                                                       46/143
  Installing : p11-kit-0.18.5-2.el6.x86_64                                                                                                                         47/143
  Installing : libuuid-2.17.2-12.14.el6.x86_64                                                                                                                     48/143
  Installing : expat-2.0.1-11.el6_2.x86_64                                                                                                                         49/143
  Installing : sysvinit-tools-2.87-5.dsf.el6.x86_64                                                                                                                50/143
  Installing : libblkid-2.17.2-12.14.el6.x86_64                                                                                                                    51/143
  Installing : p11-kit-trust-0.18.5-2.el6.x86_64                                                                                                                   52/143
  Installing : ca-certificates-2013.1.94-65.0.el6.noarch                                                                                                           53/143
  Installing : libgcrypt-1.4.5-11.el6_4.x86_64                                                                                                                     54/143
  Installing : nss-softokn-3.14.3-9.el6.x86_64                                                                                                                     55/143
  Installing : gmp-4.3.1-7.el6_2.2.x86_64                                                                                                                          56/143
  Installing : libusb-0.1.12-23.el6.x86_64                                                                                                                         57/143
  Installing : libnih-1.0.1-7.el6.x86_64                                                                                                                           58/143
  Installing : upstart-0.6.5-12.el6_4.1.x86_64                                                                                                                     59/143
  Installing : MAKEDEV-3.24-6.el6.x86_64                                                                                                                           60/143
  Installing : libutempter-1.1.5-4.1.el6.x86_64                                                                                                                    61/143
  Installing : procps-3.2.8-25.el6.x86_64                                                                                                                          62/143
  Installing : libselinux-utils-2.0.94-5.3.el6_4.1.x86_64                                                                                                          63/143
  Installing : net-tools-1.60-110.el6_2.x86_64                                                                                                                     64/143
  Installing : psmisc-22.6-15.el6_0.1.x86_64                                                                                                                       65/143
  Installing : checkpolicy-2.0.22-1.el6.x86_64                                                                                                                     66/143
  Installing : 1:findutils-4.4.2-6.el6.x86_64                                                                                                                      67/143
  Installing : cyrus-sasl-lib-2.1.23-13.el6_3.1.x86_64                                                                                                             68/143
  Installing : db4-utils-4.7.25-18.el6_4.x86_64                                                                                                                    69/143
  Installing : pinentry-0.7.6-6.el6.x86_64                                                                                                                         70/143
  Installing : diffutils-2.8.1-28.el6.x86_64                                                                                                                       71/143
  Installing : binutils-2.20.51.0.2-5.36.el6.x86_64                                                                                                                72/143
  Installing : 1:make-3.81-20.el6.x86_64                                                                                                                           73/143
  Installing : cpio-2.10-11.el6_3.x86_64                                                                                                                           74/143
  Installing : 6:oraclelinux-release-6Server-5.0.2.x86_64                                                                                                          75/143
  Installing : ncurses-5.7-3.20090208.el6.x86_64                                                                                                                   76/143
  Installing : groff-1.18.1.4-21.el6.x86_64                                                                                                                        77/143
  Installing : less-436-10.el6.x86_64                                                                                                                              78/143
  Installing : gzip-1.3.12-19.el6_4.x86_64                                                                                                                         79/143
  Installing : cracklib-2.8.16-4.el6.x86_64                                                                                                                        80/143
  Installing : cracklib-dicts-2.8.16-4.el6.x86_64                                                                                                                  81/143
  Installing : coreutils-libs-8.4-31.0.1.el6.x86_64                                                                                                                82/143
  Installing : coreutils-8.4-31.0.1.el6.x86_64                                                                                                                     83/143
  Installing : pam-1.1.1-17.el6.x86_64                                                                                                                             84/143
  Installing : module-init-tools-3.9-21.0.1.el6_4.x86_64                                                                                                           85/143
  Installing : hwdata-0.233-9.1.el6.noarch                                                                                                                         86/143
  Installing : oracle-logos-60.0.14-1.0.1.el6.noarch                                                                                                               87/143
  Installing : logrotate-3.7.8-17.el6.x86_64                                                                                                                       88/143
  Installing : nss-3.15.1-15.0.1.el6.x86_64                                                                                                                        89/143
  Installing : nss-sysinit-3.15.1-15.0.1.el6.x86_64                                                                                                                90/143
  Installing : nss-tools-3.15.1-15.0.1.el6.x86_64                                                                                                                  91/143
  Installing : openldap-2.4.23-32.el6_4.1.x86_64                                                                                                                   92/143
  Installing : libedit-2.11-4.20080712cvs.1.el6.x86_64                                                                                                             93/143
  Installing : ustr-1.0.4-9.1.el6.x86_64                                                                                                                           94/143
  Installing : libsemanage-2.0.43-4.2.el6.x86_64                                                                                                                   95/143
  Installing : libffi-3.0.5-3.2.el6.x86_64                                                                                                                         96/143
  Installing : keyutils-libs-1.4-4.el6.x86_64                                                                                                                      97/143
  Installing : krb5-libs-1.10.3-10.el6_4.6.x86_64                                                                                                                  98/143
  Installing : openssl-1.0.1e-15.el6.x86_64                                                                                                                        99/143
  Installing : libssh2-1.4.2-1.el6.x86_64                                                                                                                         100/143
  Installing : libcurl-7.19.7-37.el6_4.x86_64                                                                                                                     101/143
  Installing : curl-7.19.7-37.el6_4.x86_64                                                                                                                        102/143
  Installing : rpm-libs-4.8.0-37.el6.x86_64                                                                                                                       103/143
  Installing : rpm-4.8.0-37.el6.x86_64                                                                                                                            104/143
  Installing : gnupg2-2.0.14-6.el6_4.x86_64                                                                                                                       105/143
  Installing : gpgme-1.1.8-3.el6.x86_64                                                                                                                           106/143
  Installing : fipscheck-lib-1.2.0-7.el6.x86_64                                                                                                                   107/143
  Installing : fipscheck-1.2.0-7.el6.x86_64                                                                                                                       108/143
  Installing : 1:pkgconfig-0.23-9.1.el6.x86_64                                                                                                                    109/143
  Installing : shared-mime-info-0.70-4.el6.x86_64                                                                                                                 110/143
Non-fatal POSTIN scriptlet failure in rpm package shared-mime-info-0.70-4.el6.x86_64
warning: %post(shared-mime-info-0.70-4.el6.x86_64) scriptlet failed, exit status 127
  Installing : glib2-2.26.1-3.el6.x86_64                                                                                                                          111/143
  Installing : gamin-0.1.10-9.el6.x86_64                                                                                                                          112/143
  Installing : dbus-glib-0.86-6.el6_4.x86_64                                                                                                                      113/143
  Installing : libuser-0.56.13-5.el6.x86_64                                                                                                                       114/143
  Installing : libcap-ng-0.6.4-3.el6_0.1.x86_64                                                                                                                   115/143
  Installing : 2:ethtool-3.5-1.el6.x86_64                                                                                                                         116/143
  Installing : gdbm-1.8.0-36.el6.x86_64                                                                                                                           117/143
  Installing : python-2.6.6-51.el6.x86_64                                                                                                                         118/143
  Installing : python-libs-2.6.6-51.el6.x86_64                                                                                                                    119/143
  Installing : rpm-python-4.8.0-37.el6.x86_64                                                                                                                     120/143
  Installing : yum-metadata-parser-1.1.2-16.el6.x86_64                                                                                                            121/143
  Installing : python-pycurl-7.19.0-8.el6.x86_64                                                                                                                  122/143
  Installing : python-urlgrabber-3.9.1-9.el6.noarch                                                                                                               123/143
  Installing : pygpgme-0.1-18.20090824bzr68.el6.x86_64                                                                                                            124/143
  Installing : python-iniparse-0.3.1-2.1.el6.noarch                                                                                                               125/143
  Installing : tcp_wrappers-libs-7.6-57.el6.x86_64                                                                                                                126/143
  Installing : mingetty-1.08-5.el6.x86_64                                                                                                                         127/143
  Installing : policycoreutils-2.0.83-19.39.0.1.el6.x86_64                                                                                                        128/143
  Installing : iptables-1.4.7-11.0.1.el6.x86_64                                                                                                                   129/143
  Installing : iproute-2.6.32-31.el6.x86_64                                                                                                                       130/143
  Installing : iputils-20071127-17.el6_4.2.x86_64                                                                                                                 131/143
  Installing : initscripts-9.03.40-2.0.1.el6.x86_64                                                                                                               132/143
  Installing : util-linux-ng-2.17.2-12.14.el6.x86_64                                                                                                              133/143
  Installing : udev-147-2.51.0.1.el6.x86_64                                                                                                                       134/143
  Installing : openssh-5.3p1-94.el6.x86_64                                                                                                                        135/143
  Installing : openssh-clients-5.3p1-94.el6.x86_64                                                                                                                136/143
  Installing : openssh-server-5.3p1-94.el6.x86_64                                                                                                                 137/143
  Installing : 12:dhclient-4.1.1-38.P1.0.1.el6.x86_64                                                                                                             138/143
  Installing : rsyslog-5.8.10-8.0.1.el6.x86_64                                                                                                                    139/143
  Installing : yum-3.2.29-40.0.1.el6.noarch                                                                                                                       140/143
  Installing : passwd-0.77-4.el6_2.2.x86_64                                                                                                                       141/143
  Installing : 2:vim-minimal-7.2.411-1.8.el6.x86_64                                                                                                               142/143
  Installing : rootfiles-8.1-6.1.el6.noarch                                                                                                                       143/143

Installed:
  chkconfig.x86_64 0:1.3.49.3-2.el6_4.1 dhclient.x86_64 12:4.1.1-38.P1.0.1.el6     initscripts.x86_64 0:9.03.40-2.0.1.el6 openssh-clients.x86_64 0:5.3p1-94.el6       
  openssh-server.x86_64 0:5.3p1-94.el6  oraclelinux-release.x86_64 6:6Server-5.0.2 passwd.x86_64 0:0.77-4.el6_2.2         policycoreutils.x86_64 0:2.0.83-19.39.0.1.el6
  rootfiles.noarch 0:8.1-6.1.el6        rsyslog.x86_64 0:5.8.10-8.0.1.el6          vim-minimal.x86_64 2:7.2.411-1.8.el6   yum.noarch 0:3.2.29-40.0.1.el6              

Dependency Installed:
  MAKEDEV.x86_64 0:3.24-6.el6                       audit-libs.x86_64 0:2.2-2.el6                                basesystem.noarch 0:10.0-4.0.1.el6                    
  bash.x86_64 0:4.1.2-15.el6_4                      binutils.x86_64 0:2.20.51.0.2-5.36.el6                       bzip2-libs.x86_64 0:1.0.5-7.el6_0                     
  ca-certificates.noarch 0:2013.1.94-65.0.el6       checkpolicy.x86_64 0:2.0.22-1.el6                            coreutils.x86_64 0:8.4-31.0.1.el6                     
  coreutils-libs.x86_64 0:8.4-31.0.1.el6            cpio.x86_64 0:2.10-11.el6_3                                  cracklib.x86_64 0:2.8.16-4.el6                        
  cracklib-dicts.x86_64 0:2.8.16-4.el6              curl.x86_64 0:7.19.7-37.el6_4                                cyrus-sasl-lib.x86_64 0:2.1.23-13.el6_3.1             
  db4.x86_64 0:4.7.25-18.el6_4                      db4-utils.x86_64 0:4.7.25-18.el6_4                           dbus-glib.x86_64 0:0.86-6.el6_4                       
  dbus-libs.x86_64 1:1.2.24-7.0.1.el6_3             dhcp-common.x86_64 12:4.1.1-38.P1.0.1.el6                    diffutils.x86_64 0:2.8.1-28.el6                       
  elfutils-libelf.x86_64 0:0.152-1.el6              ethtool.x86_64 2:3.5-1.el6                                   expat.x86_64 0:2.0.1-11.el6_2                         
  file-libs.x86_64 0:5.04-15.el6                    filesystem.x86_64 0:2.4.30-3.el6                             findutils.x86_64 1:4.4.2-6.el6                        
  fipscheck.x86_64 0:1.2.0-7.el6                    fipscheck-lib.x86_64 0:1.2.0-7.el6                           gamin.x86_64 0:0.1.10-9.el6                           
  gawk.x86_64 0:3.1.7-10.el6                        gdbm.x86_64 0:1.8.0-36.el6                                   glib2.x86_64 0:2.26.1-3.el6                           
  glibc.x86_64 0:2.12-1.132.el6                     glibc-common.x86_64 0:2.12-1.132.el6                         gmp.x86_64 0:4.3.1-7.el6_2.2                          
  gnupg2.x86_64 0:2.0.14-6.el6_4                    gpgme.x86_64 0:1.1.8-3.el6                                   grep.x86_64 0:2.6.3-4.el6                             
  groff.x86_64 0:1.18.1.4-21.el6                    gzip.x86_64 0:1.3.12-19.el6_4                                hwdata.noarch 0:0.233-9.1.el6                         
  info.x86_64 0:4.13a-8.el6                         iproute.x86_64 0:2.6.32-31.el6                               iptables.x86_64 0:1.4.7-11.0.1.el6                    
  iputils.x86_64 0:20071127-17.el6_4.2              keyutils-libs.x86_64 0:1.4-4.el6                             krb5-libs.x86_64 0:1.10.3-10.el6_4.6                  
  less.x86_64 0:436-10.el6                          libacl.x86_64 0:2.2.49-6.el6                                 libattr.x86_64 0:2.4.44-7.el6                         
  libblkid.x86_64 0:2.17.2-12.14.el6                libcap.x86_64 0:2.16-5.5.el6                                 libcap-ng.x86_64 0:0.6.4-3.el6_0.1                    
  libcom_err.x86_64 0:1.42.8-1.0.1.el6              libcurl.x86_64 0:7.19.7-37.el6_4                             libedit.x86_64 0:2.11-4.20080712cvs.1.el6             
  libffi.x86_64 0:3.0.5-3.2.el6                     libgcc.x86_64 0:4.4.7-4.el6                                  libgcrypt.x86_64 0:1.4.5-11.el6_4                     
  libgpg-error.x86_64 0:1.7-4.el6                   libidn.x86_64 0:1.18-2.el6                                   libnih.x86_64 0:1.0.1-7.el6                           
  libselinux.x86_64 0:2.0.94-5.3.el6_4.1            libselinux-utils.x86_64 0:2.0.94-5.3.el6_4.1                 libsemanage.x86_64 0:2.0.43-4.2.el6                   
  libsepol.x86_64 0:2.0.41-4.el6                    libssh2.x86_64 0:1.4.2-1.el6                                 libstdc++.x86_64 0:4.4.7-4.el6                        
  libtasn1.x86_64 0:2.3-3.el6_2.1                   libusb.x86_64 0:0.1.12-23.el6                                libuser.x86_64 0:0.56.13-5.el6                        
  libutempter.x86_64 0:1.1.5-4.1.el6                libuuid.x86_64 0:2.17.2-12.14.el6                            libxml2.x86_64 0:2.7.6-14.0.1.el6                     
  logrotate.x86_64 0:3.7.8-17.el6                   lua.x86_64 0:5.1.4-4.1.el6                                   make.x86_64 1:3.81-20.el6                             
  mingetty.x86_64 0:1.08-5.el6                      module-init-tools.x86_64 0:3.9-21.0.1.el6_4                  ncurses.x86_64 0:5.7-3.20090208.el6                   
  ncurses-base.x86_64 0:5.7-3.20090208.el6          ncurses-libs.x86_64 0:5.7-3.20090208.el6                     net-tools.x86_64 0:1.60-110.el6_2                     
  nspr.x86_64 0:4.10.0-1.el6                        nss.x86_64 0:3.15.1-15.0.1.el6                               nss-softokn.x86_64 0:3.14.3-9.el6                     
  nss-softokn-freebl.x86_64 0:3.14.3-9.el6          nss-sysinit.x86_64 0:3.15.1-15.0.1.el6                       nss-tools.x86_64 0:3.15.1-15.0.1.el6                  
  nss-util.x86_64 0:3.15.1-3.el6                    openldap.x86_64 0:2.4.23-32.el6_4.1                          openssh.x86_64 0:5.3p1-94.el6                         
  openssl.x86_64 0:1.0.1e-15.el6                    oracle-logos.noarch 0:60.0.14-1.0.1.el6                      p11-kit.x86_64 0:0.18.5-2.el6                         
  p11-kit-trust.x86_64 0:0.18.5-2.el6               pam.x86_64 0:1.1.1-17.el6                                    pcre.x86_64 0:7.8-6.el6                               
  pinentry.x86_64 0:0.7.6-6.el6                     pkgconfig.x86_64 1:0.23-9.1.el6                              popt.x86_64 0:1.13-7.el6                              
  procps.x86_64 0:3.2.8-25.el6                      psmisc.x86_64 0:22.6-15.el6_0.1                              pth.x86_64 0:2.0.7-9.3.el6                            
  pygpgme.x86_64 0:0.1-18.20090824bzr68.el6         python.x86_64 0:2.6.6-51.el6                                 python-iniparse.noarch 0:0.3.1-2.1.el6                
  python-libs.x86_64 0:2.6.6-51.el6                 python-pycurl.x86_64 0:7.19.0-8.el6                          python-urlgrabber.noarch 0:3.9.1-9.el6                
  readline.x86_64 0:6.0-4.el6                       redhat-release-server.x86_64 1:6Server-6.5.0.1.0.1.el6       rpm.x86_64 0:4.8.0-37.el6                             
  rpm-libs.x86_64 0:4.8.0-37.el6                    rpm-python.x86_64 0:4.8.0-37.el6                             sed.x86_64 0:4.2.1-10.el6                             
  setup.noarch 0:2.8.14-20.el6_4.1                  shadow-utils.x86_64 2:4.1.4.2-13.el6                         shared-mime-info.x86_64 0:0.70-4.el6                  
  sqlite.x86_64 0:3.6.20-1.el6                      sysvinit-tools.x86_64 0:2.87-5.dsf.el6                       tcp_wrappers-libs.x86_64 0:7.6-57.el6                 
  tzdata.noarch 0:2013g-1.el6                       udev.x86_64 0:147-2.51.0.1.el6                               upstart.x86_64 0:0.6.5-12.el6_4.1                     
  ustr.x86_64 0:1.0.4-9.1.el6                       util-linux-ng.x86_64 0:2.17.2-12.14.el6                      xz-libs.x86_64 0:4.999.9-0.3.beta.20091007git.el6     
  yum-metadata-parser.x86_64 0:1.1.2-16.el6         zlib.x86_64 0:1.2.3-29.el6     
                            

Complete!
Rebuilding rpm database
Patching container rootfs /var/lib/lxc/lxcora01/rootfs for Oracle Linux 6.5
Configuring container for Oracle Linux 6.5

Added container user:oracle password:oracle
Added container user:root password:root
Container : /var/lib/lxc/lxcora01/rootfs
Config    : /var/lib/lxc/lxcora01/config
Network   : eth0 (veth) on virbr0


gstanden@u1504:~/Downloads$ sudo lxc-ls -f

NAME      STATE    IPV4  IPV6  GROUPS  AUTOSTART  
------------------------------------------------
lxcora01  STOPPED  -     -     -       NO
        
gstanden@u1504:~/Downloads$


Start LXC Linux Container lxcora01 in foreground mode ("-F") so that console will connect on current startup session as shown below.  Notice that DHCP provided by default by dnsmasq-base listening on lxcbr0 (10.0.3.1) has provided a dhcp-issued IP address for the container.  This is the default networking provided by the standard template.  After verifying, shutdown the container using "shutdown -h now" as shown below.

Also, note the MAC address ("HWAddr") shown in bold below which will be used when editing the config file for the container for OpenvSwitch networking.

gstanden@u1504:~/Downloads$ sudo lxc-start -n lxcora01 -F

/etc/rc.d/rc.sysinit: line 30: 1mount: ambiguous redirect
        Welcome to Oracle Linux Server
Setting hostname lxcora01:                                 [  OK  ]
Checking filesystems
                                                           [  OK  ]
Mounting local filesystems:                                [  OK  ]
Enabling /etc/fstab swaps:                                 [  OK  ]
Entering non-interactive startup
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0: 
Determining IP information for eth0... done.
                                                           [  OK  ]
Starting system logger:                                    [  OK  ]
Mounting filesystems:                                      [  OK  ]
Generating SSH1 RSA host key:                              [  OK  ]
Generating SSH2 RSA host key:                              [  OK  ]
Generating SSH2 DSA host key:                              [  OK  ]
Starting sshd:                                             [  OK  ]

Oracle Linux Server release 6.5
Kernel 3.19.0-16-generic on an x86_64

lxcora01 login: root
Password:

[root@lxcora01 ~]# ifconfig

eth0      Link encap:Ethernet  HWaddr FE:86:AB:24:50:25                    <-- Same value in /var/lib/lxc/lxcora01/config file 
          inet addr:10.0.3.169  Bcast:10.0.3.255  Mask:255.255.255.0       <-- 10.0.3.169 is provided by dnsmasq
          inet6 addr: fe80::fc86:abff:fe24:5025/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:29 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4579 (4.4 KiB)  TX bytes:1458 (1.4 KiB)

lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

[root@lxcora01 ~]# 
shutdown -h now

Configure LXC Container for OpenvSwitch Networking

Make a backup of the original install LXC config file as shown below.

gstanden@u1504:~$ sudo su -

root@u1504:~# cd /var/lib/lxc/lxcora01

root@u1504:/var/lib/lxc/lxcora01# ls -lrt
total 12
-rw-r--r--  1 root root  984 May 18 22:19 config
dr-xr-xr-x 21 root root 4096 May 18 22:23 rootfs

root@u1504:/var/lib/lxc/lxcora01# cp -p config config.original.install.bak

Edit LXC Container config File for OpenvSwitch

The original "/var/lib/lxc/lxcora01/config" file is shown below.  In the next step, the config is edited to put the container on the OpenvSwitch network., The lines shown in bold will be commented out in the next step to remove the LXC container from Linux Bridge networking, and the file will be rearranged slightly for clarity and readability.

root@u1504:/var/lib/lxc/lxcora01# cat config.original.install.bak

# Template used to create this container: /usr/share/lxc/templates/lxc-oracle
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:2e:aa:66

lxc.rootfs = /var/lib/lxc/lxcora01/rootfs
# Common configuration
lxc.include = /usr/share/lxc/config/oracle.common.conf
# Container configuration for Oracle Linux 6.5
lxc.arch = x86_64
lxc.utsname = lxcora01
lxc.cap.drop = sys_resource
lxc.cap.drop = setfcap setpcap
# Networking
lxc.network.name = eth0
lxc.network.mtu = 1500

lxc.network.hwaddr = fe:86:ab:24:50:25

root@u1504:/var/lib/lxc/lxcora01#


The edited version of the "/var/lib/lxc/lxcora01/config" for use with OpenvSwitch networking is shown below.  TheLinux Bridge networking lines have been removed, and the file has also been rearranged into sections for better readability and clarity.

root@u1504:/var/lib/lxc/lxcora01# cat config

# Template used to create this container: /usr/share/lxc/templates/lxc-oracle
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)

# Filesystem
lxc.rootfs = /var/lib/lxc/lxcora01/rootfs

# Common configuration
lxc.include = /usr/share/lxc/config/oracle.common.conf

# Container configuration for Oracle Linux 6.5
lxc.arch = x86_64
lxc.utsname = lxcora01
lxc.cap.drop = sys_resource
lxc.cap.drop = setfcap setpcap

# OpenvSwitch Networking
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora01-pub-ifup-sw1
lxc.network.script.down = /etc/network/if-down.d/lxcora01-pub-ifdown-sw1
lxc.network.veth.pair = lxcora01-pub
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.hwaddr = fe:86:ab:24:50:25

# Linux Bridge Networking
# lxc.network.type = veth
# lxc.network.link = lxcbr0
# lxc.network.flags = up
# lxc.network.hwaddr = 00:16:3e:2e:aa:66
# lxc.network.name = eth0
# lxc.network.mtu = 1500

root@u1504:/var/lib/lxc/lxcora01#

Create Additional Required Networking Files

Next the two files "lxcora01-pub-ifup-sw1" and "lxcora01-pub-ifdown-sw1" must be installed and configured as shown below.

root@W520:/etc/network/if-up.d# chmod 744 lxcora01-pub-ifup-sw1
root@W520:/etc/network/if-up.d# cd ..
root@W520:/etc/network# cd if-down.d
root@W520:/etc/network/if-down.d# chmod 744 lxcora01-pub-ifdown-sw1
root@W520:/etc/network/if-down.d#

root@W520:/etc/network/if-up.d# cat lxcora01-pub-ifup-sw1

#!/bin/bash
ovsBr='sw1'
ovs-vsctl add-port ${ovsBr} $5
ovs-vsctl set port $5 tag=10                                           <-- Note the "tag=10" which is the VLAN needed


root@W520:/etc/network/if-up.d# cd ..
root@W520:/etc/network# cd if-down.d/

root@u1504:/var/lib/lxc/lxcora01# cat /etc/network/if-down.d/lxcora01-pub-ifdown-sw1

#!/bin/bash
ovsBr='sw1'
sudo ovs-vsctl del-port ${ovsBr} $5
function CheckPortExists
{
sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'
}
PortExists=$(CheckPortExists)
if [ "$PortExists" = "$5" ]
then
sudo ovs-vsctl del-port ${ovsBr} $5
fi

root@u1504:/var/lib/lxc/lxcora01#

Configure dhclient.conf File

Next edit the "/var/lib/lxc/lxcora01/rootfs/etc/dhcp/dhclient.conf" file for the LXC container as shown below.  This file configures the "/etc/resolv.conf" file of the LXC Linux Container at boot time ensuring that it always has the correct set of nameserver values and domain name values added dynamically at boot.  Note that the "vmem.org" domain name is not included because it is added to the container /etc/resolv.conf by DHCP of the host. Optionally, it could be included here, but typically this would result in "vmem.org" appearing twice in the resolv.conf file of the LXC container.

root@u1504:/var/lib/lxc/lxcora01# cat rootfs/etc/dhcp/dhclient.conf

append domain-name-servers 8.8.8.8, 10.207.39.3;
append domain-name " gns1.vmem.org";

root@u1504:/var/lib/lxc/lxcora01#

The "/var/lib/lxc/lxcora01/rootfs/etc/sysconfig/network-scripts/ifcfg-eth0" file can be listed and checked as well as shown below just to verify it is correct.  It should be similar to the one shown below.

root@u1504:/var/lib/lxc/lxcora01# cat rootfs/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
HOSTNAME=lxcora01
DHCP_HOSTNAME=lxcora01
NM_CONTROLLED=no
TYPE=Ethernet

root@u1504:/var/lib/lxc/lxcora01#

Verify Container Operation on DHCP and DNS with OpenvSwitch

Start the container (or reboot if it is already running) as shown below.  IP assignment ot eth0 should be successful.

gstanden@W520:~$ sudo lxc-start -n lxcora01 -F

        Welcome to Oracle Linux Server
Setting hostname lxcora01:                                 [  OK  ]
Checking filesystems
                                                           [  OK  ]
Mounting local filesystems:                                [  OK  ]
Enabling /etc/fstab swaps:                                 [  OK  ]
Entering non-interactive startup
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0: 
Determining IP information for eth0... done.
                                                           [  OK  ]
Starting system logger:                                    [  OK  ]
Mounting filesystems:                                      [  OK  ]
Starting sshd:                                             [  OK  ]

Oracle Linux Server release 6.5
Kernel 3.16.0-28-generic on an x86_64

lxcora01 login:

Verify Container on OpenvSwitch Network

Login to the container and verify that the container is now on the OpenvSwitch network as shown below.  In this case the first container gets ".70" as it's IP because this was set as the lowest IP in the DHCP reserved range.  IP addresses below 70 are reserved for static assignment on the 10.207.39.x subnet.

[root@lxcora01 ~]# ifconfig

eth0      Link encap:Ethernet  HWaddr FE:71:FA:7E:CB:AF 
          inet addr:10.207.39.70  Bcast:10.207.39.255  Mask:255.255.255.0
          inet6 addr: fe80::fc71:faff:fe7e:cbaf/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:124 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:24086 (23.5 KiB)  TX bytes:1074 (1.0 KiB)

lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Verify Container /etc/resolv.conf Correct Dynamic Formatting

Check that the /etc/resolv.conf has the correct entries assigned at startup by dhclient.conf file as shown below.

[root@lxcora01 ~]# cat /etc/resolv.conf

; generated by /sbin/dhclient-script
search vmem.org gns1.vmem.org
nameserver 10.207.39.1
nameserver 8.8.8.8
nameserver 10.207.39.3


[root@lxcora01 ~]#

Install Package "bind-utils" into LXC Container

Install the "bind-utils" package inside the LXC container using "yum" as shown below, and check then check that "nslookup" can resolv "lxcora01" the hostname of the LXC container as shown below.  The bind-utils package is needed to obtain "nslookup".

[root@lxcora01 ~]# yum install bind-utils
Loaded plugins: lxc-patch
ol6_u5_base                                                                                                                                        | 1.4 kB     00:00    
ol6_u5_base/primary                                                                                                                                | 3.2 MB     00:01    
ol6_u5_base                                                                                                                                                     8573/8573
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package bind-utils.x86_64 32:9.8.2-0.17.rc1.0.2.el6_4.6 will be installed
--> Processing Dependency: bind-libs = 32:9.8.2-0.17.rc1.0.2.el6_4.6 for package: 32:bind-utils-9.8.2-0.17.rc1.0.2.el6_4.6.x86_64
--> Processing Dependency: libdns.so.81()(64bit) for package: 32:bind-utils-9.8.2-0.17.rc1.0.2.el6_4.6.x86_64
--> Processing Dependency: libbind9.so.80()(64bit) for package: 32:bind-utils-9.8.2-0.17.rc1.0.2.el6_4.6.x86_64
--> Processing Dependency: libisc.so.83()(64bit) for package: 32:bind-utils-9.8.2-0.17.rc1.0.2.el6_4.6.x86_64
--> Processing Dependency: libisccc.so.80()(64bit) for package: 32:bind-utils-9.8.2-0.17.rc1.0.2.el6_4.6.x86_64
--> Processing Dependency: liblwres.so.80()(64bit) for package: 32:bind-utils-9.8.2-0.17.rc1.0.2.el6_4.6.x86_64
--> Processing Dependency: libisccfg.so.82()(64bit) for package: 32:bind-utils-9.8.2-0.17.rc1.0.2.el6_4.6.x86_64
--> Running transaction check
---> Package bind-libs.x86_64 32:9.8.2-0.17.rc1.0.2.el6_4.6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================
 Package                             Arch                            Version                                                   Repository                            Size
==========================================================================================================================================================================
Installing:
 bind-utils                          x86_64                          32:9.8.2-0.17.rc1.0.2.el6_4.6                             ol6_u5_base                          182 k
Installing for dependencies:
 bind-libs                           x86_64                          32:9.8.2-0.17.rc1.0.2.el6_4.6                             ol6_u5_base                          878 k

Transaction Summary
==========================================================================================================================================================================
Install       2 Package(s)

Total download size: 1.0 M
Installed size: 2.6 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): bind-libs-9.8.2-0.17.rc1.0.2.el6_4.6.x86_64.rpm                                                                                             | 878 kB     00:30    
(2/2): bind-utils-9.8.2-0.17.rc1.0.2.el6_4.6.x86_64.rpm                                                                                            | 182 kB     00:02    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                      29 kB/s | 1.0 MB     00:36    
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
Importing GPG key 0xEC551F03:
 Userid : Oracle OSS group (Open Source Software group) <build@oss.oracle.com>
 Package: 6:oraclelinux-release-6Server-5.0.2.x86_64 (@ol6_u5_base/$releasever)
 From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
Is this ok [y/N]: y
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : 32:bind-libs-9.8.2-0.17.rc1.0.2.el6_4.6.x86_64                                                                                                         1/2
  Installing : 32:bind-utils-9.8.2-0.17.rc1.0.2.el6_4.6.x86_64                                                                                                        2/2
lxc-patch: checking if updated pkgs need patching...
  Verifying  : 32:bind-libs-9.8.2-0.17.rc1.0.2.el6_4.6.x86_64                                                                                                         1/2
  Verifying  : 32:bind-utils-9.8.2-0.17.rc1.0.2.el6_4.6.x86_64                                                                                                        2/2

Installed:
  bind-utils.x86_64 32:9.8.2-0.17.rc1.0.2.el6_4.6                                                                                                                        

Dependency Installed:
  bind-libs.x86_64 32:9.8.2-0.17.rc1.0.2.el6_4.6                                                                                                                         

Complete!
[root@lxcora01 ~]#

Run nslookup Tests in LXC Container to Verify DNS Resolutions

Now run nslookup as shown below to verify automatic DNS registration of DHCP-issued IP addresses and to test that various configured dynamic and static forward and reverse lookups are working correctly from the LXC container.


[root@lxcora01 ~]# nslookup lxcora01
Server:        10.207.39.1
Address:    10.207.39.1#53

Name:    lxcora01.vmem.org
Address: 10.207.39.70

[root@lxcora01 ~]# nslookup 10.207.39.70
Server:        10.207.39.1
Address:    10.207.39.1#53

70.39.207.10.in-addr.arpa    name = lxcora01.vmem.org.

[root@lxcora01 ~]# nslookup w520
Server:        10.207.39.1
Address:    10.207.39.1#53

Name:    W520.vmem.org
Address: 10.207.39.1

[root@lxcora01 ~]# nslookup 10.207.39.1
Server:        10.207.39.1
Address:    10.207.39.1#53

1.39.207.10.in-addr.arpa    name = W520.vmem.org.

[root@lxcora01 ~]# nslookup lxc1-gns-vip
Server:        10.207.39.1
Address:    10.207.39.1#53

Name:    lxc1-gns-vip.vmem.org
Address: 10.207.39.3

[root@lxcora01 ~]# nslookup 10.207.39.3
Server:        10.207.39.1
Address:    10.207.39.1#53

3.39.207.10.in-addr.arpa    name = lxc1-gns-vip.vmem.org.

Run Various Status Commands on Container

The LXC container is now on the OpenvSwitch network and ready for configuration of pre-requisites for Oracle Enterprise Database product installations. Various checks on the running container can be run as shown below.

gstanden@w520:~$ sudo lxc-info -n lxcora01
[sudo] password for gstanden:
Name:           lxcora01
State:          RUNNING
PID:            5288
IP:             10.207.39.70
CPU use:        4.99 seconds
BlkIO use:      2.23 MiB
Memory use:     33.65 MiB
KMem use:       0 bytes
Link:           lxcora01-pub
 TX bytes:      107.54 KiB
 RX bytes:      4.54 MiB
 Total bytes:   4.65 MiB
gstanden@w520:~$ sudo lxc-ls -f
NAME      STATE    IPV4          IPV6  GROUPS  AUTOSTART 
--------------------------------------------------------
lxcora01  RUNNING  10.207.39.70  -     -       NO        
gstanden@w520:~$

Create File Management Links

Several files located in various directories are used to configure and manage this environment.  It is useful to build links off of the "/home/username" directory to act as pointers to various required files for centralized managment and control. Use the "ln -s" command to create links as shown below.  Note that links to files inside the rootfs of the LXC container will show as red due to access permissions when connected as non-root account, but are useful nevertheless for reminding location of these files.

root@vmem1:/home/gstanden/Networking# cat crt_links.sh

ln -s /etc/dhcp/dhcpd.conf .
ln -s /etc/dhcp/dhclient.conf .
ln -s /etc/init/my-network-up.sh .
ln -s /var/lib/bind/fwd.vmem.org .
ln -s /var/lib/bind/rev.vmem.org .
ln -s /var/lib/bind/fwd.mccc.org .
ln -s /var/lib/bind/rev.mccc.org .
ln -s /etc/network/if-up.d/lxc-ifup-sw1 .
ln -s /etc/network/if-down.d/lxc-ifdown-sw1 .
ln -s /etc/bind/named.conf.options .
ln -s /etc/bind/named.conf.local .
ln -s /etc/bind/rndc.key .
ln -s /var/lib/lxc/lxcora02/config lxcora02-config
ln -s /var/lib/lxc/lxcora03/config lxcora03-config
ln -s /etc/NetworkManager/dnsmasq.d/local .

root@vmem1:/home/gstanden/Networking#


Contents of the /home/gstanden/OpenvSwitch directory shown below.

gstanden@w520:~$ cd OpenvSwitch
gstanden@w520:~/OpenvSwitch$ ls -lrt
total 48
-rwxr-xr-x 1 gstanden gstanden 2153 Dec 30 19:34 crt_ovs_sw1.sh
-rwxr-xr-x 1 gstanden gstanden 1016 Dec 30 19:34 crt_ovs_sw2.sh
-rwxr-xr-x 1 gstanden gstanden 1019 Dec 30 19:34 crt_ovs_sw3.sh
-rwxr-xr-x 1 gstanden gstanden  704 Dec 30 19:34 crt_ovs_sw4.sh
-rwxr-xr-x 1 gstanden gstanden  704 Dec 30 19:34 crt_ovs_sw5.sh
-rwxr-xr-x 1 gstanden gstanden  704 Dec 30 19:34 crt_ovs_sw6.sh
-rwxr-xr-x 1 gstanden gstanden  704 Dec 30 19:34 crt_ovs_sw7.sh
-rwxr-xr-x 1 gstanden gstanden  704 Dec 30 19:34 crt_ovs_sw8.sh
-rwxr-xr-x 1 gstanden gstanden  704 Dec 30 19:34 crt_ovs_sw9.sh
lrwxrwxrwx 1 gstanden gstanden   28 Dec 30 19:38 my-network-up.conf -> /etc/init/my-network-up.conf
-rw-r--r-- 1 gstanden gstanden  414 Dec 30 20:50 crt_ovs_sw1.log
-rw-r--r-- 1 gstanden gstanden  195 Dec 30 20:50 crt_ovs_sw2.log
-rw-r--r-- 1 gstanden gstanden  195 Dec 30 20:50 crt_ovs_sw3.log
-rw-r--r-- 1 gstanden gstanden    0 Dec 30 20:50 crt_ovs_sw4.log
-rw-r--r-- 1 gstanden gstanden    0 Dec 30 20:50 crt_ovs_sw5.log
-rw-r--r-- 1 gstanden gstanden    0 Dec 30 20:50 crt_ovs_sw6.log
-rw-r--r-- 1 gstanden gstanden    0 Dec 30 20:50 crt_ovs_sw7.log
-rw-r--r-- 1 gstanden gstanden    0 Dec 30 20:50 crt_ovs_sw8.log
-rw-r--r-- 1 gstanden gstanden    0 Dec 30 20:50 crt_ovs_sw9.log
gstanden@w520:~/OpenvSwitch$ cd ..
gstanden@w520:~$ cd Networking
root@vmem1:/home/gstanden/Networking# ls -lrt

total 4
lrwxrwxrwx 1 root     root      20 May 16 17:38 dhcpd.conf -> /etc/dhcp/dhcpd.conf
lrwxrwxrwx 1 root     root      23 May 16 17:38 dhclient.conf -> /etc/dhcp/dhclient.conf
lrwxrwxrwx 1 root     root      26 May 16 17:38 rev.vmem.org -> /var/lib/bind/rev.vmem.org
lrwxrwxrwx 1 root     root      26 May 16 17:38 fwd.vmem.org -> /var/lib/bind/fwd.vmem.org
lrwxrwxrwx 1 root     root      26 May 16 17:38 rev.mccc.org -> /var/lib/bind/rev.mccc.org
lrwxrwxrwx 1 root     root      26 May 16 17:38 fwd.mccc.org -> /var/lib/bind/fwd.mccc.org
lrwxrwxrwx 1 root     root      33 May 16 17:38 lxc-ifup-sw1 -> /etc/network/if-up.d/lxc-ifup-sw1
lrwxrwxrwx 1 root     root      37 May 16 17:38 lxc-ifdown-sw1 -> /etc/network/if-down.d/lxc-ifdown-sw1
lrwxrwxrwx 1 root     root      18 May 16 17:38 rndc.key -> /etc/bind/rndc.key
lrwxrwxrwx 1 root     root      28 May 16 17:38 named.conf.options -> /etc/bind/named.conf.options
lrwxrwxrwx 1 root     root      26 May 16 17:38 named.conf.local -> /etc/bind/named.conf.local
lrwxrwxrwx 1 root     root      35 May 16 17:38 local -> /etc/NetworkManager/dnsmasq.d/local
lrwxrwxrwx 1 root     root      28 May 16 17:39 lxcora02-config -> /var/lib/lxc/lxcora02/config
lrwxrwxrwx 1 root     root      28 May 16 17:39 lxcora03-config -> /var/lib/lxc/lxcora03/config
-rwxr-xr-x 1 gstanden gstanden 569 May 16 17:40 crt_links.sh
lrwxrwxrwx 1 root     root      26 May 16 17:40 my-network-up.sh -> /etc/init/my-network-up.sh

root@vmem1:/home/gstanden/Networking#

gstanden@w520:~/Networking$ cd ..
gstanden@w520:~$ cd Templates
gstanden@w520:~/Templates$ ls -lrt
total 0

lrwxrwxrwx 1 gstanden gstanden 24 Dec 30 21:18 templates -> /usr/share/lxc/templates

gstanden@w520:~/Templates$

Add HAIP and ASM Networks to Container Configuation File

Additional network adapters are needed in the LXC Oracle Linux 6.5 containers.  Add those containers by adding additional network adapters to the /var/lib/lxc/lxcora01/config file.  The updated config file is shown below with the added code shown in bold.  Note that these additional adapters are for HAIP using four subnets and for the ASM flex cluster network using two subnets.  Note that the additional files needed in "/etc/network/if-up.d" and in "/etc/network/if-down.d" are found at the end of this blog in the downloadable files section for the lxcora01 container.

root@u1504:/var/lib/lxc/lxcora01# cat config
# Template used to create this container: /usr/share/lxc/templates/lxc-oracle
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)

# Filesystem
lxc.rootfs = /var/lib/lxc/lxcora01/rootfs

# Common configuration
lxc.include = /usr/share/lxc/config/oracle.common.conf

# Container configuration for Oracle Linux 6.5
lxc.arch = x86_64
lxc.utsname = lxcora01
lxc.cap.drop = sys_resource
lxc.cap.drop = setfcap setpcap

# OpenvSwitch Networking
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora01-pub-ifup-sw1
lxc.network.script.down = /etc/network/if-down.d/lxcora01-pub-ifdown-sw1
lxc.network.veth.pair = lxcora01-pub
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.hwaddr = fe:86:ab:24:50:25

lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora01-priv1-ifup-sw4
lxc.network.script.down = /etc/network/if-down.d/lxcora01-priv1-ifdown-sw4
lxc.network.veth.pair = lxcora01-priv1
lxc.network.name = eth1
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.210.39.10
lxc.network.hwaddr = 00:16:3e:28:e4:88

lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora01-priv2-ifup-sw5
lxc.network.script.down = /etc/network/if-down.d/lxcora01-priv2-ifdown-sw5
lxc.network.veth.pair = lxcora01-priv2
lxc.network.name = eth2
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.211.39.10
lxc.network.hwaddr = 00:16:3e:34:05:3f

lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora01-priv3-ifup-sw6
lxc.network.script.down = /etc/network/if-down.d/lxcora01-priv3-ifdown-sw6
lxc.network.veth.pair = lxcora01-priv3
lxc.network.name = eth3
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.212.39.10
lxc.network.hwaddr = 00:16:3e:38:4e:36

lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora01-priv4-ifup-sw7
lxc.network.script.down = /etc/network/if-down.d/lxcora01-priv4-ifdown-sw7
lxc.network.veth.pair = lxcora01-priv4
lxc.network.name = eth4
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.213.39.10
lxc.network.hwaddr = 00:16:3e:68:e8:35

lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora01-asm1-ifup-sw8
lxc.network.script.down = /etc/network/if-down.d/lxcora01-asm1-ifdown-sw8
lxc.network.veth.pair = lxcora01-asm1
lxc.network.name = eth5
lxc.network.mtu = 9000
lxc.network.ipv4 = 172.220.40.10
lxc.network.hwaddr = 00:16:3e:50:81:28

lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora01-asm2-ifup-sw9
lxc.network.script.down = /etc/network/if-down.d/lxcora01-asm2-ifdown-sw9
lxc.network.veth.pair = lxcora01-asm2
lxc.network.name = eth6
lxc.network.mtu = 9000
lxc.network.ipv4 = 172.221.40.10
lxc.network.hwaddr = 00:16:3e:24:c8:4b


# Linux Bridge Networking
# lxc.network.type = veth
# lxc.network.link = lxcbr0
# lxc.network.flags = up
# lxc.network.hwaddr = 00:16:3e:2e:aa:66
# lxc.network.name = eth0
# lxc.network.mtu = 1500

lxc.cgroup.devices.allow = c 10:236 rwm
lxc.cgroup.devices.allow = b 252:* rwm
# lxc.mount.entry = /dev/mapper /var/lib/lxc/lxcora10/rootfs/dev/mapper none defaults,bind,create=dir 0 0
lxc.mount.auto = proc sys cgroup
lxc.mount.auto = proc:rw sys:rw cgroup-full:rw
root@u1504:/var/lib/lxc/lxcora01#

Note that the lxc.network.script.up and the lxc.network.script.down require additional files to be installed under /etc/network/if-up.d and under /etc/network/if-down.d as shown above in the config file.  Those files are attached to this blog and can be downloaded.  A couple of examples of the files are shown below for reference.

root@u1504:/var/lib/lxc/lxcora01# cat /etc/network/if-up.d/lxcora01-asm2-ifup-sw9

#!/bin/bash
ovsBr='sw9'
sudo ovs-vsctl add-port ${ovsBr} $5
sudo ovs-vsctl set port lxcora01-asm2 tag=70
sudo ovs-vsctl set port ${ovsBr} tag=70
sudo ovs-vsctl set port ${ovsBr} trunks=60,70

root@u1504:/var/lib/lxc/lxcora01# cat /etc/network/if-down.d/lxcora01-asm2-ifdown-sw9

#!/bin/bash
ovsBr='sw9'
sudo ovs-vsctl del-port ${ovsBr} $5
function CheckPortExists
{
sudo ovs-vsctl show | grep \"$5\" | grep Port | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -f2 -d' ' | sed 's/"//g'
}
PortExists=$(CheckPortExists)
if [ "$PortExists" = "$5" ]
then
sudo ovs-vsctl del-port ${ovsBr} $5
fi

root@u1504:/var/lib/lxc/lxcora01#

Disable Transparent HugePages on Ubuntu Linux

Disable transparent huge pages as shown below.

gstanden@u1504:~$ sudo apt-get install hugepages

[sudo] password for gstanden:
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following packages were automatically installed and are no longer required:
  linux-headers-3.19.0-15 linux-headers-3.19.0-15-generic linux-image-3.19.0-15-generic linux-image-extra-3.19.0-15-generic
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  libhugetlbfs0
Suggested packages:
  libhugetlbfs-tests
The following NEW packages will be installed:
  hugepages libhugetlbfs0
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 104 kB of archives.
After this operation, 348 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://us.archive.ubuntu.com/ubuntu/ vivid/universe libhugetlbfs0 amd64 2.17-0ubuntu2 [55.4 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ vivid/universe hugepages amd64 2.17-0ubuntu2 [48.4 kB]
Fetched 104 kB in 0s (292 kB/s)    
Selecting previously unselected package libhugetlbfs0.
(Reading database ... 271909 files and directories currently installed.)
Preparing to unpack .../libhugetlbfs0_2.17-0ubuntu2_amd64.deb ...
Unpacking libhugetlbfs0 (2.17-0ubuntu2) ...
Selecting previously unselected package hugepages.
Preparing to unpack .../hugepages_2.17-0ubuntu2_amd64.deb ...
Unpacking hugepages (2.17-0ubuntu2) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up libhugetlbfs0 (2.17-0ubuntu2) ...
Setting up hugepages (2.17-0ubuntu2) ...
Processing triggers for libc-bin (2.21-0ubuntu4) ...
/sbin/ldconfig.real: /usr/lib/libbrcolm2.so.1 is not a symbolic link

/sbin/ldconfig.real: /usr/lib/libbrscandec2.so.1 is not a symbolic link

gstanden@u1504:~$ hugeadm

hugeadm [options]
options:
 --list-all-mounts          List all current hugetlbfs mount points
 --pool-list                List all pools
 --hard                     specified with --pool-pages-min to make
                            multiple attempts at adjusting the pool size to the
                            specified count on failure
 --pool-pages-min <size|DEFAULT>:[+|-]<pagecount|memsize<G|M|K>> 
                            Adjust pool 'size' lower bound
 --obey-mempolicy           Obey the NUMA memory policy when
                            adjusting the pool 'size' lower bound
 --thp-always               Enable transparent huge pages always
 --thp-madvise              Enable transparent huge pages with madvise
 --thp-never                Disable transparent huge pages
 --thp-khugepaged-pages <pages to scan>  Number of pages that khugepaged
                            should scan on each pass
 --thp-khugepaged-scan-sleep <milliseconds>  Time in ms to sleep between
                            khugepaged passes
 --thp-khugepages-alloc-sleep <milliseconds>  Time in ms for khugepaged
                            to wait if there was a huge page allocation failure
 --pool-pages-max <size|DEFAULT>:[+|-]<pagecount|memsize<G|M|K>> 
                            Adjust pool 'size' upper bound
 --set-recommended-min_free_kbytes 
                            Sets min_free_kbytes to a recommended value to improve availability of
                            huge pages at runtime
 --set-recommended-shmmax   Sets shmmax to a recommended value to
                            maximise the size possible for shared memory pools
 --set-shm-group <gid|groupname>  Sets hugetlb_shm_group to the
                            specified group, which has permission to use hugetlb shared memory pools
 --add-temp-swap[=count]    Specified with --pool-pages-min to create
                            temporary swap space for the duration of the pool resize. Default swap
                            size is 5 huge pages. Optional arg sets size to 'count' huge pages
 --add-ramdisk-swap         Specified with --pool-pages-min to create
                            swap space on ramdisks. By default, swap is removed after the resize.
 --persist                  Specified with --add-temp-swap or --add-ramdisk-swap
                            options to make swap space persist after the resize.
 --enable-zone-movable      Use ZONE_MOVABLE for huge pages
 --disable-zone-movable     Do not use ZONE_MOVABLE for huge pages
 --create-mounts            Creates a mount point for each available
                            huge page size on this system under /var/lib/hugetlbfs
 --create-user-mounts <user> 
                            Creates a mount point for each available huge
                            page size under /var/lib/hugetlbfs/<user>
                            usable by user <user>
 --create-group-mounts <group> 
                            Creates a mount point for each available huge
                            page size under /var/lib/hugetlbfs/<group>
                            usable by group <group>
 --create-global-mounts    
                            Creates a mount point for each available huge
                            page size under /var/lib/hugetlbfs/global
                            usable by anyone
 --max-size <size<G|M|K>>   Limit the filesystem size of a new mount point
 --max-inodes <number>      Limit the number of inodes on a new mount point
 --page-sizes               Display page sizes that a configured pool
 --page-sizes-all           Display page sizes support by the hardware
 --dry-run                  Print the equivalent shell commands for what
                            the specified options would have done without
                            taking any action
 --explain                  Gives a overview of the status of the system
                            with respect to huge page availability
 --verbose <level>, -v      Increases/sets tracing levels
 --help, -h                 Prints this message

gstanden@u1504:~$ cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

gstanden@u1504:~$ sudo hugeadm --thp-never

gstanden@u1504:~$ cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]

gstanden@u1504:~$

Configure Users and Packages In LXC Container

Run the following scripts as shown below.  The create_users.sh script installs the users and creates some directories.  The packages.sh script installs all packages required for Oracle RDBMS RAC.  Run these scripts inside the LXC container.

[root@lxcora01 ~]# ls -lrt

total 12
-rwxr-xr-x 1 root root  416 Sep 18  2014 create_users.sh
-rwxr-xr-x 1 root root 1273 Sep 18  2014 packages.sh
-rwxr-xr-x 1 root root 1176 Sep 21  2014 hugepages_setting.sh


[root@lxcora01 ~]# pwd
/root
[root@lxcora01 ~]#

Add the Following Lines to the end of /etc/security/limits.conf File

Add the lines below to /etc/security/limits.conf as shown below.

# Oracle Grid Infrastructure

 grid    hard    nofile     65536
 grid    soft    nofile        65536
 grid    soft    nproc       2047

# Oracle Database

# *     soft     memlock  9873408
# *     hard     memlock  9873408

Add Oracle Configurations to /etc/sysctl.conf

Add the following lines to /etc/sysctl.conf as shown below (new lines are in bold).

[root@lxcora01 ~]# cat /etc/sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 65536

# Controls the maximum size of a message, in bytes
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296

# Oracle

kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576

net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
kernel.panic_on_oops = 1

net.ipv4.conf.eth1.rp_filter = 2
net.ipv4.conf.eth2.rp_filter = 2
net.ipv4.conf.eth3.rp_filter = 2
net.ipv4.conf.eth4.rp_filter = 2
net.ipv4.conf.eth5.rp_filter = 2
net.ipv4.conf.eth6.rp_filter = 2

[root@lxcora01 ~]#

Apply Updates to "/etc/sysctl.conf"

Apply the updates to the /etc/sysctl.conf file as shown below.

[root@lxcora01 ~]# sysctl -p

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
error: permission denied on key 'kernel.sysrq'
error: permission denied on key 'kernel.core_uses_pid'
error: "net.ipv4.tcp_syncookies" is an unknown key
error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
error: "net.bridge.bridge-nf-call-iptables" is an unknown key
error: "net.bridge.bridge-nf-call-arptables" is an unknown key
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
error: permission denied on key 'fs.file-max'
error: permission denied on key 'fs.aio-max-nr'
net.ipv4.ip_local_port_range = 9000 65500
error: "net.core.rmem_default" is an unknown key
error: "net.core.rmem_max" is an unknown key
error: "net.core.wmem_default" is an unknown key
error: "net.core.wmem_max" is an unknown key
error: permission denied on key 'kernel.panic_on_oops'
net.ipv4.conf.eth1.rp_filter = 2
net.ipv4.conf.eth2.rp_filter = 2
net.ipv4.conf.eth3.rp_filter = 2
net.ipv4.conf.eth4.rp_filter = 2
net.ipv4.conf.eth5.rp_filter = 2
net.ipv4.conf.eth6.rp_filter = 2

[root@lxcora01 ~]#

Set umask for "oracle" and "grid" Users

Set the umask to 022 as shown below in the .bashrc files for "oracle" and "grid" users.

[oracle@lxcora01 ~]$ cat .bashrc
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

# User specific aliases and functions

umask 022


[oracle@lxcora01 ~]$ exit
logout
[root@lxcora01 ~]# su - grid
[grid@lxcora01 ~]$ cat .bashrc

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

# User specific aliases and functions

umask 022


[grid@lxcora01 ~]$

Set Password for "grid" Linux User

As the root user, set the password for the grid user as shown below.

[root@lxcora01 ~]# passwd grid
Changing password for user grid.
New password:
BAD PASSWORD: it is too short
BAD PASSWORD: is too simple
Retype new password:
passwd: all authentication tokens updated successfully.
[root@lxcora01 ~]#

Edit "/etc/rc.local" File

Add the following lines to the /etc/rc.local file in the LXC containers as shown below and configure the /etc/sysconfig/ntpd file as shown below.

# Oracle ASM Storage
chown grid:asmadmin /dev/mapper/asm_disk*
chmod 0660 /dev/mapper/asm_disk*

# NTP
ntpd

Configure "/etc/sysconfig/ntpd" File

Configure the /etc/sysconfig/ntpd file as shown below.

[root@lxcora01 ~]# cat /etc/sysconfig/ntpd
# Drop root to id 'ntp:ntp' by default.
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid -g"
[root@lxcora01 ~]#

Install NFS Server on Ubuntu Base OS Host

Install nfs server on the Ubuntu host as shown below.

gstanden@u1504:~$ sudo apt-get install nfs-kernel-server nfs-common portmap
[sudo] password for gstanden:
Reading package lists... Done
Building dependency tree      
Reading state information... Done
Note, selecting 'rpcbind' instead of 'portmap'
The following packages were automatically installed and are no longer required:
  linux-headers-3.19.0-15 linux-headers-3.19.0-15-generic linux-image-3.19.0-15-generic linux-image-extra-3.19.0-15-generic
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  keyutils libnfsidmap2 libtirpc1
Suggested packages:
  open-iscsi watchdog
The following NEW packages will be installed:
  keyutils libnfsidmap2 libtirpc1 nfs-common nfs-kernel-server rpcbind
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 468 kB of archives.
After this operation, 2,066 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://us.archive.ubuntu.com/ubuntu/ vivid/main libnfsidmap2 amd64 0.25-5 [32.2 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ vivid/main libtirpc1 amd64 0.2.5-1 [75.6 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ vivid/main keyutils amd64 1.5.9-5ubuntu1 [47.9 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu/ vivid/main rpcbind amd64 0.2.1-6ubuntu3 [38.3 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu/ vivid-updates/main nfs-common amd64 1:1.2.8-9ubuntu8.1 [186 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu/ vivid-updates/main nfs-kernel-server amd64 1:1.2.8-9ubuntu8.1 [88.0 kB]
Fetched 468 kB in 0s (818 kB/s)            
Selecting previously unselected package libnfsidmap2:amd64.
(Reading database ... 271938 files and directories currently installed.)
Preparing to unpack .../libnfsidmap2_0.25-5_amd64.deb ...
Unpacking libnfsidmap2:amd64 (0.25-5) ...
Selecting previously unselected package libtirpc1:amd64.
Preparing to unpack .../libtirpc1_0.2.5-1_amd64.deb ...
Unpacking libtirpc1:amd64 (0.2.5-1) ...
Selecting previously unselected package keyutils.
Preparing to unpack .../keyutils_1.5.9-5ubuntu1_amd64.deb ...
Unpacking keyutils (1.5.9-5ubuntu1) ...
Selecting previously unselected package rpcbind.
Preparing to unpack .../rpcbind_0.2.1-6ubuntu3_amd64.deb ...
Unpacking rpcbind (0.2.1-6ubuntu3) ...
Selecting previously unselected package nfs-common.
Preparing to unpack .../nfs-common_1%3a1.2.8-9ubuntu8.1_amd64.deb ...
Unpacking nfs-common (1:1.2.8-9ubuntu8.1) ...
Selecting previously unselected package nfs-kernel-server.
Preparing to unpack .../nfs-kernel-server_1%3a1.2.8-9ubuntu8.1_amd64.deb ...
Unpacking nfs-kernel-server (1:1.2.8-9ubuntu8.1) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for systemd (219-7ubuntu6) ...
Processing triggers for ureadahead (0.100.0-19) ...
ureadahead will be reprofiled on next reboot
Setting up libnfsidmap2:amd64 (0.25-5) ...
Setting up libtirpc1:amd64 (0.2.5-1) ...
Setting up keyutils (1.5.9-5ubuntu1) ...
Setting up rpcbind (0.2.1-6ubuntu3) ...
Setting up nfs-common (1:1.2.8-9ubuntu8.1) ...

Creating config file /etc/idmapd.conf with new version

Creating config file /etc/default/nfs-common with new version
Adding system user `statd' (UID 126) ...
Adding new user `statd' (UID 126) with group `nogroup' ...
Not creating home directory `/var/lib/nfs'.
invoke-rc.d: gssd.service doesn't exist but the upstart job does. Nothing to start or stop until a systemd or init job is present.
invoke-rc.d: idmapd.service doesn't exist but the upstart job does. Nothing to start or stop until a systemd or init job is present.
nfs-utils.service is a disabled or a static unit, not starting it.
Setting up nfs-kernel-server (1:1.2.8-9ubuntu8.1) ...

Creating config file /etc/exports with new version

Creating config file /etc/default/nfs-kernel-server with new version
Processing triggers for libc-bin (2.21-0ubuntu4) ...
/sbin/ldconfig.real: /usr/lib/libbrcolm2.so.1 is not a symbolic link

/sbin/ldconfig.real: /usr/lib/libbrscandec2.so.1 is not a symbolic link

Processing triggers for systemd (219-7ubuntu6) ...
Processing triggers for ureadahead (0.100.0-19) ...
gstanden@u1504:~$ cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
#        to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#
gstanden@u1504:~$

Create ASM LUNs for Oracle Grid Infrastructure

Create ASM LUNs for the Oracle grid infrastructure install as shown below.

gstanden@u1504:~$ sudo su -
root@u1504:~# cd /root
root@u1504:~# ls -lrt
total 4
-rwxr-xr-x 1 root root 219 Jun 27 21:55 create-oracle-luns.sh
root@u1504:~# cat create-oracle-luns.sh
dd if=/dev/zero of=/oracle_data/AsmSystem_01.img bs=512 count=0 seek=15M
dd if=/dev/zero of=/oracle_data/AsmSystem_02.img bs=512 count=0 seek=15M
dd if=/dev/zero of=/oracle_data/AsmSystem_03.img bs=512 count=0 seek=15M
root@u1504:~# ls -lrt /oracle_data/
total 0
-rw-r--r-- 1 root root 8053063680 Jun 27 21:55 AsmSystem_01.img
-rw-r--r-- 1 root root 8053063680 Jun 27 21:55 AsmSystem_03.img
-rw-r--r-- 1 root root 8053063680 Jun 27 21:55 AsmSystem_02.img
root@u1504:~#

Configure "/etc/exports" File on Ubuntu Base OS Host

Configure the /etc/exports file on the Ubuntu host as shown below.

gstanden@u1504:~$ cat /etc/exports

# /etc/exports: the access control list for filesystems which may be exported
#        to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#
/oracle_data               *(rw,sync,no_wdelay,insecure_locks,no_root_squash)

gstanden@u1504:~$

Edit "/etc/apparmor.d/lxc/lxc-default" File for NFS

Add configuration lines to allow NFS to operate under the apparmor regime as shown below.  Then restart apparmor as shown below.

gstanden@u1504:~$ cat /etc/apparmor.d/lxc/lxc-default

# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>

  # the container may never be allowed to mount devpts.  If it does, it
  # will remount the host's devpts.  We could allow it to do it with
  # the newinstance option (but, right now, we don't).
  deny mount fstype=devpts,
  mount fstype=nfs,
  mount fstype=nfs4,
  mount fstype=rpc_pipefs,

}
gstanden@u1504:~$ sudo /etc/init.d/apparmor restart

[ ok ] Restarting apparmor (via systemctl): apparmor.service.

gstanden@u1504:~$

Configure /dev/shm in LXC Containers

The /dev/shm is configured using an entry in the /var/lib/lxc/lxcora01/config file (and similar line in the lxcora02/config file) as shown below.

lxc.mount.entry = shm dev/shm tmpfs size=3500m,nosuid,nodev,noexec,create=dir 0 0

Edit "/etc/fstab" File

Also, an entry is needed in the /etc/fstab inside each LXC containers as shown below.

[root@lxcora01 ~]# cat /etc/fstab
tmpfs /dev/shm tmpfs rw,exec,size=3500m 0 0
u1504:/oracle_data   /oradata  nfs  rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0  0 0
[root@lxcora01 ~]#

Install cvuqdisk RPM

Install the cvuqdisk rpm as root, go to /home/grid/grid/rpm and install the rpm on the LXC container.

[root@lxcora01 ~]# cd /home/grid/grid/rpm
[root@lxcora01 rpm]# rpm -Uvh cvuqdisk-1.0.9-1.rpm

Configure Additional Linux Groups for Oracle 12c

Configure the additional groups for the oracle user on both LXC containers as shown below.

[root@lxcora02 ~]# ./create_users_12c_addon.sh
[root@lxcora02 ~]# id oracle
uid=500(oracle) gid=500(oracle) groups=500(oracle),1000(oinstall),1200(dba),1300(asmdba),1201(oper),1401(backupdba),1501(dgdba),1601(kmdba)
[root@lxcora02 ~]# cat create_users_12c_addon.sh
groupadd -g 1401 backupdba
groupadd -g 1501 dgdba
groupadd -g 1601 kmdba
groupadd -g 1701 osacfs
groupadd -g 1801 osaudit
usermod -a -G backupdba,dgdba,kmdba oracle
[root@lxcora02 ~]#

Configure ZEROCONF

Configure ZEROCONF as shown below on both LXC containers.

[root@lxcora02 ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=lxcora02
NOZEROCONF=yes
[root@lxcora02 ~]#

Verify SELINUX

Verify that SELINUX is set to disabled in both LXC containers as shown below.

[root@lxcora02 ~]# cat /etc/selinux/config
SELINUX=disabled
[root@lxcora02 ~]#

Clone the Container to Create Second RAC Node

Shutdown the LXC container and then clone it to create the second RAC node as shown below.

[root@lxcora01 ~]# shutdown -h now

Broadcast message from root@lxcora01
    (/dev/lxc/console) at 18:55 ...

The system is going down for halt NOW!
[root@lxcora01 ~]# Stopping sshd:                          [  OK  ]
Shutting down system logger:                               [  OK  ]
Shutting down interface eth0:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Sending all processes the TERM signal...                   [  OK  ]
Saving random seed:                                        [  OK  ]
Unmounting file systems:                                   [  OK  ]
umount2: Permission denied
umount: /proc/uptime: block devices not permitted on fs
umount2: Permission denied
umount: /proc/sys/fs/binfmt_misc: block devices not permitted on fs
umount2: Permission denied
umount2: Permission denied
umount: /proc/stat: block devices not permitted on fs
umount2: Permission denied
umount: /proc/meminfo: block devices not permitted on fs
umount2: Permission denied
umount: /proc/diskstats: block devices not permitted on fs
umount2: Permission denied
umount: /proc/cpuinfo: block devices not permitted on fs
umount2: Permission denied
umount: /dev/pts: block devices not permitted on fs
init: Re-executing /sbin/init
mount: you must specify the filesystem type
Halting system...
gstanden@u1504:~$ clear

Note on Startup and Shutdown of LXC Oracle Linux 6.5 Containers

There are some error and warning messages that occur during startup and shutdown of the OEL 6.5 LXC Containers that can be safely ignored for the purposes of installing Oracle Clusterware and RDBMS.  The startup and shutdown output of the LXC containers is shown below with the acceptable warnings and messages highlighted in bold.  These messages are due to known issues with the startup and shutdown of LXC containers currently.  These messages and warnings highlighted in bold can be safely ignored.

gstanden@u1504:~$ sudo lxc-start -n lxcora02 -F
[sudo] password for gstanden:
/etc/rc.d/rc.sysinit: line 30: 1mount: ambiguous redirect             <-- This message can be safely ignored
        Welcome to Oracle Linux Server
Setting hostname lxcora02:                                 [  OK  ]
Checking filesystems
                                                           [  OK  ]
Mounting local filesystems:                                [  OK  ]
Enabling /etc/fstab swaps:                                 [  OK  ]
Entering non-interactive startup
Calling the system activity data collector (sadc)...
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0: 
Determining IP information for eth0... done.
                                                           [  OK  ]
Starting system logger:                                    [  OK  ]
Starting rpcbind:                                          [  OK  ]
Starting NFS statd:                                        [  OK  ]
NFS filesystems queued to be mounted
Mounting filesystems:                                      [  OK  ]
Starting sshd:                                             [  OK  ]
Starting crond:                                            [  OK  ]

Oracle Linux Server release 6.5
Kernel 3.19.0-21-generic on an x86_64

lxcora02 login: root
Password:
Last login: Sun Jun 28 12:53:51 on lxc/console
[root@lxcora02 ~]# shutdown -h now

Broadcast message from root@lxcora02
    (/dev/lxc/console) at 14:07 ...

The system is going down for halt NOW!
[root@lxcora02 ~]# Stopping sshd:                          [  OK  ]
Stopping crond:                                            [  OK  ]
Unmounting NFS filesystems:                                [  OK  ]
Stopping rpcbind:                                          [  OK  ]
Shutting down system logger:                               [  OK  ]
Shutting down interface eth0:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Sending all processes the TERM signal...                   [  OK  ]
Sending all processes the KILL signal...                   [  OK  ]
Saving random seed:                                        [  OK  ]
Unmounting file systems:                                   [  OK  ]
umount2: Permission denied                                            <-- All mount and umount errors can be ignored
umount: /proc/uptime: block devices not permitted on fs
umount2: Permission denied
umount: /proc/sys/fs/binfmt_misc: block devices not permitted on fs
umount2: Permission denied
umount2: Permission denied
umount: /proc/stat: block devices not permitted on fs
umount2: Permission denied
umount: /proc/meminfo: block devices not permitted on fs
umount2: Permission denied
umount: /proc/diskstats: block devices not permitted on fs
umount2: Permission denied
umount: /proc/cpuinfo: block devices not permitted on fs
umount2: Permission denied
umount: /dev/shm: block devices not permitted on fs
umount2: Permission denied
umount2: Permission denied
umount: /dev/pts: block devices not permitted on fs
init: Re-executing /sbin/init
mount: you must specify the filesystem type
mount: you must specify the filesystem type

Halting system...
gstanden@u1504:~$

Verify that the LXC container is stopped as shown below.

gstanden@u1504:~$ sudo lxc-ls -f

NAME      STATE    IPV4  IPV6  GROUPS  AUTOSTART 
------------------------------------------------
lxcora01  STOPPED  -     -     -       NO
        

gstanden@u1504:~$

Clone the LXC container as shown below.

gstanden@u1504:~$ sudo lxc-clone -o lxcora01 -n lxcora02
Created container lxcora02 as copy of lxcora01
gstanden@u1504:~$

This will create a config file for the cloned container as shown below.  It is necessary to change the references in the OpenvSwitch stanzas from "lxcora01" to "lxcora02" because the lxc-clone does not know about OpenvSwitch networking by default.

root@u1504:/var/lib/lxc/lxcora02# cat config

# Template used to create this container: /usr/share/lxc/templates/lxc-oracle
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)
# Filesystem
# Common configuration
lxc.include = /usr/share/lxc/config/oracle.common.conf
# Container configuration for Oracle Linux 6.5
lxc.arch = x86_64
lxc.cap.drop = sys_resource
lxc.cap.drop = setfcap setpcap
# OpenvSwitch Networking
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxc-ifup-sw1
lxc.network.script.down = /etc/network/if-down.d/lxc-ifdown-sw1
lxc.network.veth.pair = lxcora01-pub            <-- Change to "lxcora01-pub" and similar for all other similar references 
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.hwaddr = 00:16:3e:73:1e:71
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora01-priv1-ifup-sw4
lxc.network.script.down = /etc/network/if-down.d/lxcora01-priv1-ifdown-sw4
lxc.network.veth.pair = lxcora01-priv1
lxc.network.name = eth1
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.210.39.10                <-- Change all ".10" static IP addresses to ".11" static IP addresses
lxc.network.hwaddr = 00:16:3e:f6:30:ca
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora01-priv2-ifup-sw5
lxc.network.script.down = /etc/network/if-down.d/lxcora01-priv2-ifdown-sw5
lxc.network.veth.pair = lxcora01-priv2
lxc.network.name = eth2
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.211.39.10
lxc.network.hwaddr = 00:16:3e:51:53:30
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora01-priv3-ifup-sw6
lxc.network.script.down = /etc/network/if-down.d/lxcora01-priv3-ifdown-sw6
lxc.network.veth.pair = lxcora01-priv3
lxc.network.name = eth3
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.212.39.10
lxc.network.hwaddr = 00:16:3e:38:9c:d7
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora01-priv4-ifup-sw7
lxc.network.script.down = /etc/network/if-down.d/lxcora01-priv4-ifdown-sw7
lxc.network.veth.pair = lxcora01-priv4
lxc.network.name = eth4
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.213.39.10
lxc.network.hwaddr = 00:16:3e:b5:65:3f
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora01-asm1-ifup-sw8
lxc.network.script.down = /etc/network/if-down.d/lxcora01-asm1-ifdown-sw8
lxc.network.veth.pair = lxcora01-asm1
lxc.network.name = eth5
lxc.network.mtu = 9000
lxc.network.ipv4 = 172.220.40.10
lxc.network.hwaddr = 00:16:3e:d4:a0:55
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora01-asm2-ifup-sw9
lxc.network.script.down = /etc/network/if-down.d/lxcora01-asm2-ifdown-sw9
lxc.network.veth.pair = lxcora01-asm2
lxc.network.name = eth6
lxc.network.mtu = 9000
lxc.network.ipv4 = 172.221.40.10
lxc.network.hwaddr = 00:16:3e:36:22:62
# Linux Bridge Networking
# lxc.network.type = veth
# lxc.network.link = lxcbr0
# lxc.network.flags = up
# lxc.network.hwaddr = 00:16:3e:2e:aa:66
# lxc.network.name = eth0
# lxc.network.mtu = 1500
lxc.cgroup.devices.allow = c 10:236 rwm
lxc.cgroup.devices.allow = b 252:* rwm
# lxc.mount.entry = /dev/mapper /var/lib/lxc/lxcora10/rootfs/dev/mapper none defaults,bind,create=dir 0 0
lxc.mount.auto = proc sys cgroup
lxc.mount.auto = proc:rw sys:rw cgroup-full:rw
lxc.rootfs = /var/lib/lxc/lxcora02/rootfs
lxc.utsname = lxcora02

root@u1504:/var/lib/lxc/lxcora02#

The edited config file for the cloned LXC container is shown below.

root@u1504:/var/lib/lxc/lxcora02# pwd
/var/lib/lxc/lxcora02

root@u1504:/var/lib/lxc/lxcora02# cat config

# Template used to create this container: /usr/share/lxc/templates/lxc-oracle
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)
# Filesystem
# Common configuration
lxc.include = /usr/share/lxc/config/oracle.common.conf
# Container configuration for Oracle Linux 6.5
lxc.arch = x86_64
lxc.cap.drop = sys_resource
lxc.cap.drop = setfcap setpcap
# OpenvSwitch Networking
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxc-ifup-sw1
lxc.network.script.down = /etc/network/if-down.d/lxc-ifdown-sw1
lxc.network.veth.pair = lxcora02-pub
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.hwaddr = 00:16:3e:73:1e:71
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora02-priv1-ifup-sw4
lxc.network.script.down = /etc/network/if-down.d/lxcora02-priv1-ifdown-sw4
lxc.network.veth.pair = lxcora02-priv1

lxc.network.name = eth1
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.210.39.11
lxc.network.hwaddr = 00:16:3e:f6:30:ca
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora02-priv2-ifup-sw5
lxc.network.script.down = /etc/network/if-down.d/lxcora02-priv2-ifdown-sw5
lxc.network.veth.pair = lxcora02-priv2

lxc.network.name = eth2
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.211.39.11
lxc.network.hwaddr = 00:16:3e:51:53:30
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora02-priv3-ifup-sw6
lxc.network.script.down = /etc/network/if-down.d/lxcora02-priv3-ifdown-sw6
lxc.network.veth.pair = lxcora02-priv3

lxc.network.name = eth3
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.212.39.11
lxc.network.hwaddr = 00:16:3e:38:9c:d7
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora02-priv4-ifup-sw7
lxc.network.script.down = /etc/network/if-down.d/lxcora02-priv4-ifdown-sw7
lxc.network.veth.pair = lxcora02-priv4

lxc.network.name = eth4
lxc.network.mtu = 9000
lxc.network.ipv4 = 192.213.39.11
lxc.network.hwaddr = 00:16:3e:b5:65:3f
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora02-asm1-ifup-sw8
lxc.network.script.down = /etc/network/if-down.d/lxcora02-asm1-ifdown-sw8
lxc.network.veth.pair = lxcora02-asm1

lxc.network.name = eth5
lxc.network.mtu = 9000
lxc.network.ipv4 = 172.220.40.11
lxc.network.hwaddr = 00:16:3e:d4:a0:55
lxc.network.type = veth
lxc.network.flags = up
lxc.network.script.up = /etc/network/if-up.d/lxcora02-asm2-ifup-sw9
lxc.network.script.down = /etc/network/if-down.d/lxcora02-asm2-ifdown-sw9
lxc.network.veth.pair = lxcora02-asm2

lxc.network.name = eth6
lxc.network.mtu = 9000
lxc.network.ipv4 = 172.221.40.11
lxc.network.hwaddr = 00:16:3e:36:22:62
# Linux Bridge Networking
# lxc.network.type = veth
# lxc.network.link = lxcbr0
# lxc.network.flags = up
# lxc.network.hwaddr = 00:16:3e:2e:aa:66
# lxc.network.name = eth0
# lxc.network.mtu = 1500
lxc.cgroup.devices.allow = c 10:236 rwm
lxc.cgroup.devices.allow = b 252:* rwm
# lxc.mount.entry = /dev/mapper /var/lib/lxc/lxcora10/rootfs/dev/mapper none defaults,bind,create=dir 0 0
lxc.mount.auto = proc sys cgroup
lxc.mount.auto = proc:rw sys:rw cgroup-full:rw
lxc.rootfs = /var/lib/lxc/lxcora02/rootfs
lxc.utsname = lxcora02

root@u1504:/var/lib/lxc/lxcora02#

Next it is necessary to create the required files to create and destroy the OpenvSwitch ports as shown below.  To create the additional files needed in the "/etc/network/if-up.d" and in the "/etc/network/if-down.d" directories a script is used, as shown below.

To use the script, as shown below, it is necessary to decide how many Oracle RAC nodes will be a part of this deployment.  In this blog, a 6-node RAC ASM Flex Cluster will be created.  Therefore, all the networking files needed for all six nodes will now be created, even though for now only nodes 1 (lxcora01) and node 2 (lxcora02) will be worked with.  Later on, farther down the blog, the additional nodes lxcora03, lxcora04, lxcora05, and lxcora06 will be cloned from lxcora02 and configured for the 6-node RAC cluster.  Note that the blog is written so that a 2-node RAC can be created and the section later in the blog to create the addtional 4 nodes to make a 6-node RAC can be skipped.  Now, below, the networking files are created using the script, as shown below.

root@u1504:/etc/network/if-up.d# ls -lrt lxcora01*

-rwxr-xr-x 1 root root  84 May 18 23:23 lxcora01-pub-ifup-sw1
-rwxr-xr-x 1 root root 180 Jun 28 17:32 lxcora01-asm1-ifup-sw8
-rwxr-xr-x 1 root root 180 Jun 28 17:32 lxcora01-asm2-ifup-sw9
-rwxr-xr-x 1 root root 186 Jun 28 17:33 lxcora01-priv1-ifup-sw4
-rwxr-xr-x 1 root root 186 Jun 28 17:33 lxcora01-priv2-ifup-sw5
-rwxr-xr-x 1 root root 186 Jun 28 17:34 lxcora01-priv3-ifup-sw6
-rwxr-xr-x 1 root root 186 Jun 28 17:34 lxcora01-priv4-ifup-sw7

root@u1504:/etc/network/if-up.d# cat create-ovs-sw-files-v2.sh

#! /bin/bash

# Usage
# sudo ./create-ovs-sw-files.sh ContainerBaseName ContainerCount

# Example
# sudo ./create-ovs-sw-files.sh lxcora0 6
# Above example would create networking files for 6 LXC containers with names of the form {lxcora01, lxcora02, lxcora03,lxcora04 lxcora05,lxcora06}

# Notes
# The ContainerBaseName must be passed in at the command line and is the prefix of the LXC container names
# The ContainerCount is the total number of LXC containers to be created
 
export DirectoryName='/etc/network/if-up.d /etc/network/if-down.d'
export ContainerBaseName=$1
export ContainerCount=$2

for j in $DirectoryName
do
cd $j

function GetFileNames {
ls "$ContainerBaseName"1* | more | grep "$1" | sed 's/$/ /' | tr -d '\n'
}
FileName=$(GetFileNames)

let n=2
    for i in $FileName
    do
        while [ $n -le $ContainerCount ]
        do
        function NewFileNames {
        echo $i | sed "s/"$ContainerBaseName"1/$ContainerBaseName$n/"
        }
        NewFile=$(NewFileNames)
        cp $i $NewFile
        ((n=n+1))
        done
    let n=2
    done
done

root@u1504:/etc/network/if-up.d# ./create-ovs-sw-files-v2.sh lxcora0 6

In the above script, "lxcora0" is the naming prefix of the LXC container RAC nodes, and "6" is the number of RAC nodes to be deployed.  If using a different naming scheme for the LXC RAC node containers, such as {racnode01, racnode01} then change the parameter from "lxcora0" to "racnode0", and similarly, if building a 4-node RAC, then change the "6" parameter to a "4" parameter.  Running this script will create all the files needed for both the "/etc/network/if-up.d" and the "/etc/network/if-down.d" directories for all 6 of the RAC nodes.  The results of running the script are shown below.

root@u1504:/etc/network/if-up.d# ls -lo lxcora0*

-rwxr-xr-x 1 root 180 Jun 28 17:32 lxcora01-asm1-ifup-sw8
-rwxr-xr-x 1 root 180 Jun 28 17:32 lxcora01-asm2-ifup-sw9
-rwxr-xr-x 1 root 186 Jun 28 17:33 lxcora01-priv1-ifup-sw4
-rwxr-xr-x 1 root 186 Jun 28 17:33 lxcora01-priv2-ifup-sw5
-rwxr-xr-x 1 root 186 Jun 28 17:34 lxcora01-priv3-ifup-sw6
-rwxr-xr-x 1 root 186 Jun 28 17:34 lxcora01-priv4-ifup-sw7
-rwxr-xr-x 1 root  84 May 18 23:23 lxcora01-pub-ifup-sw1
-rwxr-xr-x 1 root 180 Jun 28 18:04 lxcora02-asm1-ifup-sw8
-rwxr-xr-x 1 root 180 Jun 28 18:04 lxcora02-asm2-ifup-sw9
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora02-priv1-ifup-sw4
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora02-priv2-ifup-sw5
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora02-priv3-ifup-sw6
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora02-priv4-ifup-sw7
-rwxr-xr-x 1 root  84 Jun 28 18:04 lxcora02-pub-ifup-sw1
-rwxr-xr-x 1 root 180 Jun 28 18:04 lxcora03-asm1-ifup-sw8
-rwxr-xr-x 1 root 180 Jun 28 18:04 lxcora03-asm2-ifup-sw9
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora03-priv1-ifup-sw4
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora03-priv2-ifup-sw5
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora03-priv3-ifup-sw6
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora03-priv4-ifup-sw7
-rwxr-xr-x 1 root  84 Jun 28 18:04 lxcora03-pub-ifup-sw1
-rwxr-xr-x 1 root 180 Jun 28 18:04 lxcora04-asm1-ifup-sw8
-rwxr-xr-x 1 root 180 Jun 28 18:04 lxcora04-asm2-ifup-sw9
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora04-priv1-ifup-sw4
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora04-priv2-ifup-sw5
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora04-priv3-ifup-sw6
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora04-priv4-ifup-sw7
-rwxr-xr-x 1 root  84 Jun 28 18:04 lxcora04-pub-ifup-sw1
-rwxr-xr-x 1 root 180 Jun 28 18:04 lxcora05-asm1-ifup-sw8
-rwxr-xr-x 1 root 180 Jun 28 18:04 lxcora05-asm2-ifup-sw9
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora05-priv1-ifup-sw4
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora05-priv2-ifup-sw5
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora05-priv3-ifup-sw6
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora05-priv4-ifup-sw7
-rwxr-xr-x 1 root  84 Jun 28 18:04 lxcora05-pub-ifup-sw1
-rwxr-xr-x 1 root 180 Jun 28 18:04 lxcora06-asm1-ifup-sw8
-rwxr-xr-x 1 root 180 Jun 28 18:04 lxcora06-asm2-ifup-sw9
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora06-priv1-ifup-sw4
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora06-priv2-ifup-sw5
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora06-priv3-ifup-sw6
-rwxr-xr-x 1 root 186 Jun 28 18:04 lxcora06-priv4-ifup-sw7
-rwxr-xr-x 1 root  84 Jun 28 18:04 lxcora06-pub-ifup-sw1

root@u1504:/etc/network/if-up.d# cd ..

root@u1504:/etc/network# cd if-down.d

root@u1504:/etc/network/if-down.d# ls -lo lxcora0*

-rwxr-xr-x 1 root 301 Jun 18 21:25 lxcora01-asm1-ifdown-sw8
-rwxr-xr-x 1 root 301 Jun 18 21:26 lxcora01-asm2-ifdown-sw9
-rwxr-xr-x 1 root 301 Jun 18 21:25 lxcora01-priv1-ifdown-sw4
-rwxr-xr-x 1 root 301 Jun 18 21:25 lxcora01-priv2-ifdown-sw5
-rwxr-xr-x 1 root 301 Jun 18 21:25 lxcora01-priv3-ifdown-sw6
-rwxr-xr-x 1 root 301 Jun 18 21:24 lxcora01-priv4-ifdown-sw7
-rwxr-xr-x 1 root 301 May 18 23:00 lxcora01-pub-ifdown-sw1
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora02-asm1-ifdown-sw8
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora02-asm2-ifdown-sw9
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora02-priv1-ifdown-sw4
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora02-priv2-ifdown-sw5
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora02-priv3-ifdown-sw6
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora02-priv4-ifdown-sw7
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora02-pub-ifdown-sw1
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora03-asm1-ifdown-sw8
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora03-asm2-ifdown-sw9
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora03-priv1-ifdown-sw4
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora03-priv2-ifdown-sw5
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora03-priv3-ifdown-sw6
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora03-priv4-ifdown-sw7
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora03-pub-ifdown-sw1
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora04-asm1-ifdown-sw8
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora04-asm2-ifdown-sw9
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora04-priv1-ifdown-sw4
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora04-priv2-ifdown-sw5
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora04-priv3-ifdown-sw6
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora04-priv4-ifdown-sw7
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora04-pub-ifdown-sw1
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora05-asm1-ifdown-sw8
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora05-asm2-ifdown-sw9
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora05-priv1-ifdown-sw4
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora05-priv2-ifdown-sw5
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora05-priv3-ifdown-sw6
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora05-priv4-ifdown-sw7
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora05-pub-ifdown-sw1
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora06-asm1-ifdown-sw8
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora06-asm2-ifdown-sw9
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora06-priv1-ifdown-sw4
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora06-priv2-ifdown-sw5
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora06-priv3-ifdown-sw6
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora06-priv4-ifdown-sw7
-rwxr-xr-x 1 root 301 Jun 28 18:04 lxcora06-pub-ifdown-sw1

root@u1504:/etc/network/if-down.d# 

Note that the ports, such as "lxcora01-pub", are referenced by an environmental variable that is available ("$5") and so the actual names of the switches to not need to be specified in the files, where "$5" is the value of "lxc.network.veth.pair" from the LXC /var/lib/lxc/lxcora01/config file.

Note that only the scripts for lxcora01 for the ip-up.d and the if-down.d directories are attached at the end of this blog, since all the other files can be generated as needed by the script.  The script is also attached at the end of the blog.

Now start up the container and see what starts up leaving the original container lxcora01 down until verification has been done that the actual hostname and networking etc are correct (non-duplicate).  Since this is a first for the author too, all these steps will be carefully documented.

gstanden@u1504:~$ sudo lxc-ls -f
[sudo] password for gstanden:

NAME      STATE    IPV4  IPV6  GROUPS  AUTOSTART 
------------------------------------------------
lxcora01  STOPPED  -     -     -       NO        
lxcora02  STOPPED  -     -     -       NO     
  

gstanden@u1504:~$

Verify that the container starts up correctly and note that the IP address assigned to eth0 is different from the IP assigned to eth0 on the original LXC container from which the clone copy was made, as shown below.


gstanden@u1504:~$ sudo lxc-start -n lxcora02 -F

/etc/rc.d/rc.sysinit: line 30: 1mount: ambiguous redirect
        Welcome to Oracle Linux Server
Setting hostname lxcora02:                                 [  OK  ]
Checking filesystems
                                                           [  OK  ]
Mounting local filesystems:                                [  OK  ]
Enabling /etc/fstab swaps:                                 [  OK  ]
Entering non-interactive startup
Calling the system activity data collector (sadc)...
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0: 
Determining IP information for eth0... done.
                                                           [  OK  ]
Starting system logger:                                    [  OK  ]
Starting rpcbind:                                          [  OK  ]
Starting NFS statd:                                        [  OK  ]
Mounting filesystems:                                      [  OK  ]
Starting sshd:                                             [  OK  ]
Starting crond:                                            [  OK  ]

Oracle Linux Server release 6.5
Kernel 3.19.0-21-generic on an x86_64

lxcora02 login: root
Password:
Last login: Sat Jun 27 18:55:55 on lxc/console

[root@lxcora02 ~]# ifconfig eth0

eth0      Link encap:Ethernet  HWaddr 00:16:3E:73:1E:71 
          inet addr:10.207.39.71  Bcast:10.207.39.255  Mask:255.255.255.0
          inet6 addr: fe80::216:3eff:fe73:1e71/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:32 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6119 (5.9 KiB)  TX bytes:1938 (1.8 KiB)

[root@lxcora02 ~]# uname -a
Linux lxcora02 3.19.0-21-generic #21-Ubuntu SMP Sun Jun 14 18:31:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@lxcora02 ~]#

Check that all other ethX are configured with correct static IP addresses and have correct MTU 9000 settings as shown below.

[root@lxcora02 ~]# ifconfig | grep -A2 eth

eth0      Link encap:Ethernet  HWaddr 00:16:3E:73:1E:71 
          inet addr:10.207.39.71  Bcast:10.207.39.255  Mask:255.255.255.0
          inet6 addr: fe80::216:3eff:fe73:1e71/64 Scope:Link
--
eth1      Link encap:Ethernet  HWaddr 00:16:3E:F6:30:CA 
          inet addr:192.210.39.11  Bcast:192.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::216:3eff:fef6:30ca/64 Scope:Link
--
eth2      Link encap:Ethernet  HWaddr 00:16:3E:51:53:30 
          inet addr:192.211.39.11  Bcast:192.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::216:3eff:fe51:5330/64 Scope:Link
--
eth3      Link encap:Ethernet  HWaddr 00:16:3E:38:9C:D7 
          inet addr:192.212.39.11  Bcast:192.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::216:3eff:fe38:9cd7/64 Scope:Link
--
eth4      Link encap:Ethernet  HWaddr 00:16:3E:B5:65:3F 
          inet addr:192.213.39.11  Bcast:192.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::216:3eff:feb5:653f/64 Scope:Link
--
eth5      Link encap:Ethernet  HWaddr 00:16:3E:D4:A0:55 
          inet addr:172.220.40.11  Bcast:172.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::216:3eff:fed4:a055/64 Scope:Link
--
eth6      Link encap:Ethernet  HWaddr 00:16:3E:36:22:62 
          inet addr:172.221.40.11  Bcast:172.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::216:3eff:fe36:2262/64 Scope:Link

[root@lxcora02 ~]# ifconfig eth1

eth1      Link encap:Ethernet  HWaddr 00:16:3E:F6:30:CA 
          inet addr:192.210.39.11  Bcast:192.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::216:3eff:fef6:30ca/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:116 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:24599 (24.0 KiB)  TX bytes:828 (828.0 b)

[root@lxcora02 ~]#


Startup the original LXC container (lxcora01) and run ping tests between the various ethX.

Copy the Oracle Grid install media into the /var/lib/lxc/lxcora01/rootfs/home/grid directory as shown below.

root@u1504:/home/gstanden/Downloads# cp -p linuxamd64_12102_grid_*.zip /var/lib/lxc/lxcora01/rootfs/home/grid/.

root@u1504:/home/gstanden/Downloads# ls -lrt linuxamd64_12102_grid*.zip

-rw-r--r-- 1 gstanden gstanden  646972897 May 16 23:26 linuxamd64_12102_grid_2of2.zip
-rw-r--r-- 1 gstanden gstanden 1747043545 May 16 23:50 linuxamd64_12102_grid_1of2.zip

root@u1504:/home/gstanden/Downloads#

Set the correct permissions on the files as shown below.

[root@lxcora01 grid]# ls -al
total 2337944
drwx------ 2 grid oinstall       4096 Jun 27 20:28 .
drwxr-xr-x 5 root root           4096 Jun 27 18:58 ..
-rw------- 1 grid oinstall         73 Jun 27 20:40 .bash_history
-rw-r--r-- 1 grid oinstall         18 Jul 18  2013 .bash_logout
-rw-r--r-- 1 grid oinstall        176 Jul 18  2013 .bash_profile
-rw-r--r-- 1 grid oinstall        135 Jun 27 19:30 .bashrc
-rw-r--r-- 1 1000 oinstall 1747043545 May 16 23:50 linuxamd64_12102_grid_1of2.zip
-rw-r--r-- 1 1000 oinstall  646972897 May 16 23:26 linuxamd64_12102_grid_2of2.zip
[root@lxcora01 grid]# chown grid:oinstall *.zip
[root@lxcora01 grid]# ls -lrt
total 2337920

-rw-r--r-- 1 grid oinstall  646972897 May 16 23:26 linuxamd64_12102_grid_2of2.zip
-rw-r--r-- 1 grid oinstall 1747043545 May 16 23:50 linuxamd64_12102_grid_1of2.zip


[root@lxcora01 grid]#

As the "grid" linux user, unzip the files in order (1of2 then 2of2) as shown below.

[grid@lxcora01 ~]$ unzip linuxamd64_12102_grid_1of2.zip

Do a pre-check of the GNS configuration as shown below.

[grid@lxcora01 grid]$ pwd
/home/grid/grid
[grid@lxcora01 grid]$

[grid@lxcora01 grid]$ ./runcluvfy.sh comp gns -precrsinst -domain vmem.org -vip 10.207.39.3 -verbose -n lxcora01

Verifying GNS integrity


Checking GNS integrity...
Checking if the GNS subdomain name is valid...
The GNS subdomain name "vmem.org" is a valid domain name
Checking if the GNS VIP is a valid address...
GNS VIP "10.207.39.3" resolves to a valid IP address
Checking the status of GNS VIP...

GNS integrity check passed

Verification of GNS integrity was successful.
[grid@lxcora01 grid]$

Verify that lxcora01 and lxcora02 are resolved by DNS as shown below.

[grid@lxcora01 grid]$ nslookup lxcora02

Server:        10.207.39.1
Address:    10.207.39.1#53

Name:    lxcora02.vmem.org
Address: 10.207.39.71

[grid@lxcora01 grid]$ nslookup lxcora01

Server:        10.207.39.1
Address:    10.207.39.1#53

Name:    lxcora01.vmem.org
Address: 10.207.39.70

[grid@lxcora01 grid]$

Create no-password ssh between the nodes for the grid user as shown below.

[grid@lxcora01 grid]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_rsa):
Created directory '/home/grid/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
The key fingerprint is:
ec:a0:2f:54:58:77:b1:19:94:37:d1:8c:48:06:97:92 grid@lxcora01
The key's randomart image is:
+--[ RSA 2048]----+
|        .=B=.=   |
|      . Eo=++ o  |
|     o . oo. .   |
|    . ..         |
|     .. S        |
|    .. o         |
|   ..   .        |
|    ..           |
|     ..          |
+-----------------+
[grid@lxcora01 grid]$

As the root user, set the password for the grid user as shown below.

[root@lxcora01 ~]# passwd grid
Changing password for user grid.
New password:
BAD PASSWORD: it is too short
BAD PASSWORD: is too simple
Retype new password:
passwd: all authentication tokens updated successfully.
[root@lxcora01 ~]#

Add the following lines to the /etc/rc.local file in the LXC containers as shown below and configure the /etc/sysconfig/ntpd file as shown below.

# Oracle ASM Storage
chown grid:asmadmin /dev/mapper/asm_disk*
chmod 0660 /dev/mapper/asm_disk*

# NTP
ntpd

Configure the /etc/sysconfig/ntpd file as shown below.

[root@lxcora01 ~]# cat /etc/sysconfig/ntpd
# Drop root to id 'ntp:ntp' by default.
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid -g"
[root@lxcora01 ~]#

Install nfs server on the Ubuntu host as shown below.

gstanden@u1504:~$ sudo apt-get install nfs-kernel-server nfs-common portmap
[sudo] password for gstanden:
Reading package lists... Done
Building dependency tree      
Reading state information... Done
Note, selecting 'rpcbind' instead of 'portmap'
The following packages were automatically installed and are no longer required:
  linux-headers-3.19.0-15 linux-headers-3.19.0-15-generic linux-image-3.19.0-15-generic linux-image-extra-3.19.0-15-generic
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  keyutils libnfsidmap2 libtirpc1
Suggested packages:
  open-iscsi watchdog
The following NEW packages will be installed:
  keyutils libnfsidmap2 libtirpc1 nfs-common nfs-kernel-server rpcbind
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 468 kB of archives.
After this operation, 2,066 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://us.archive.ubuntu.com/ubuntu/ vivid/main libnfsidmap2 amd64 0.25-5 [32.2 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ vivid/main libtirpc1 amd64 0.2.5-1 [75.6 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ vivid/main keyutils amd64 1.5.9-5ubuntu1 [47.9 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu/ vivid/main rpcbind amd64 0.2.1-6ubuntu3 [38.3 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu/ vivid-updates/main nfs-common amd64 1:1.2.8-9ubuntu8.1 [186 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu/ vivid-updates/main nfs-kernel-server amd64 1:1.2.8-9ubuntu8.1 [88.0 kB]
Fetched 468 kB in 0s (818 kB/s)            
Selecting previously unselected package libnfsidmap2:amd64.
(Reading database ... 271938 files and directories currently installed.)
Preparing to unpack .../libnfsidmap2_0.25-5_amd64.deb ...
Unpacking libnfsidmap2:amd64 (0.25-5) ...
Selecting previously unselected package libtirpc1:amd64.
Preparing to unpack .../libtirpc1_0.2.5-1_amd64.deb ...
Unpacking libtirpc1:amd64 (0.2.5-1) ...
Selecting previously unselected package keyutils.
Preparing to unpack .../keyutils_1.5.9-5ubuntu1_amd64.deb ...
Unpacking keyutils (1.5.9-5ubuntu1) ...
Selecting previously unselected package rpcbind.
Preparing to unpack .../rpcbind_0.2.1-6ubuntu3_amd64.deb ...
Unpacking rpcbind (0.2.1-6ubuntu3) ...
Selecting previously unselected package nfs-common.
Preparing to unpack .../nfs-common_1%3a1.2.8-9ubuntu8.1_amd64.deb ...
Unpacking nfs-common (1:1.2.8-9ubuntu8.1) ...
Selecting previously unselected package nfs-kernel-server.
Preparing to unpack .../nfs-kernel-server_1%3a1.2.8-9ubuntu8.1_amd64.deb ...
Unpacking nfs-kernel-server (1:1.2.8-9ubuntu8.1) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for systemd (219-7ubuntu6) ...
Processing triggers for ureadahead (0.100.0-19) ...
ureadahead will be reprofiled on next reboot
Setting up libnfsidmap2:amd64 (0.25-5) ...
Setting up libtirpc1:amd64 (0.2.5-1) ...
Setting up keyutils (1.5.9-5ubuntu1) ...
Setting up rpcbind (0.2.1-6ubuntu3) ...
Setting up nfs-common (1:1.2.8-9ubuntu8.1) ...

Creating config file /etc/idmapd.conf with new version

Creating config file /etc/default/nfs-common with new version
Adding system user `statd' (UID 126) ...
Adding new user `statd' (UID 126) with group `nogroup' ...
Not creating home directory `/var/lib/nfs'.
invoke-rc.d: gssd.service doesn't exist but the upstart job does. Nothing to start or stop until a systemd or init job is present.
invoke-rc.d: idmapd.service doesn't exist but the upstart job does. Nothing to start or stop until a systemd or init job is present.
nfs-utils.service is a disabled or a static unit, not starting it.
Setting up nfs-kernel-server (1:1.2.8-9ubuntu8.1) ...

Creating config file /etc/exports with new version

Creating config file /etc/default/nfs-kernel-server with new version
Processing triggers for libc-bin (2.21-0ubuntu4) ...
/sbin/ldconfig.real: /usr/lib/libbrcolm2.so.1 is not a symbolic link

/sbin/ldconfig.real: /usr/lib/libbrscandec2.so.1 is not a symbolic link

Processing triggers for systemd (219-7ubuntu6) ...
Processing triggers for ureadahead (0.100.0-19) ...
gstanden@u1504:~$ cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
#        to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#
gstanden@u1504:~$

Create ASM LUNs for the Oracle grid infrastructure install as shown below.

gstanden@u1504:~$ sudo su -
root@u1504:~# cd /root
root@u1504:~# ls -lrt
total 4
-rwxr-xr-x 1 root root 219 Jun 27 21:55 create-oracle-luns.sh
root@u1504:~# cat create-oracle-luns.sh
dd if=/dev/zero of=/oracle_data/AsmSystem_01.img bs=512 count=0 seek=15M
dd if=/dev/zero of=/oracle_data/AsmSystem_02.img bs=512 count=0 seek=15M
dd if=/dev/zero of=/oracle_data/AsmSystem_03.img bs=512 count=0 seek=15M
root@u1504:~# ls -lrt /oracle_data/
total 0
-rw-r--r-- 1 root root 8053063680 Jun 27 21:55 AsmSystem_01.img
-rw-r--r-- 1 root root 8053063680 Jun 27 21:55 AsmSystem_03.img
-rw-r--r-- 1 root root 8053063680 Jun 27 21:55 AsmSystem_02.img
root@u1504:~#

Configure the /etc/exports file on the Ubuntu host as shown below.

gstanden@u1504:~$ cat /etc/exports

# /etc/exports: the access control list for filesystems which may be exported
#        to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#
/oracle_data               *(rw,sync,no_wdelay,insecure_locks,no_root_squash)

gstanden@u1504:~$

Add configuration lines as shown below to allow NFS to operate under the apparmor regime as shown below.  Then restart apparmor as shown below.

gstanden@u1504:~$ cat /etc/apparmor.d/lxc/lxc-default

# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>

  # the container may never be allowed to mount devpts.  If it does, it
  # will remount the host's devpts.  We could allow it to do it with
  # the newinstance option (but, right now, we don't).
  deny mount fstype=devpts,
  mount fstype=nfs,
  mount fstype=nfs4,
  mount fstype=rpc_pipefs,

}
gstanden@u1504:~$ sudo /etc/init.d/apparmor restart

[ ok ] Restarting apparmor (via systemctl): apparmor.service.

gstanden@u1504:~$

Run Cluster Verification Check

The expected correct output should be similar to that shown below.  There are a few errors which are acceptable, and are due to known bugs in the CVU or are due to differences in the way that LXC containers return check data to CVU, and those acceptable errors are in bold.

[grid@lxcora01 grid]$ ./runcluvfy.sh stage -pre crsinst -n lxcora01,lxcora02 -r 12.1 -osdba oinstall -asm -asmgrp asmadmin -asmdev /oradata/AsmSystem_01.img,/oradata/AsmSystem_02.img,/oradata/AsmSystem_03.img -verbose | tee report7.txt

[grid@lxcora01 grid]$ more report7.txt


Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "lxcora01"
  Destination Node                      Reachable?             
  ------------------------------------  ------------------------
  lxcora01                              yes                    
  lxcora02                              yes                    
Result: Node reachability check passed from node "lxcora01"


Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Status                 
  ------------------------------------  ------------------------
  lxcora02                              passed                 
  lxcora01                              passed                 
Result: User equivalence check passed for user "grid"

Check: Package existence for "cvuqdisk"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed   
  lxcora01      cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed   
Result: Package existence check passed for "cvuqdisk"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status                 
  ------------------------------------  ------------------------
  lxcora01                              passed                 
  lxcora02                              passed                 

Verification of the hosts config file successful


Interface information for node "lxcora01"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU  
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   10.207.39.70    10.207.39.0     0.0.0.0         10.207.39.1     FE:86:AB:24:50:25 1500 
 eth1   192.210.39.10   192.0.0.0       0.0.0.0         10.207.39.1     00:16:3E:28:E4:88 9000 
 eth2   192.211.39.10   192.0.0.0       0.0.0.0         10.207.39.1     00:16:3E:34:05:3F 9000 
 eth3   192.212.39.10   192.0.0.0       0.0.0.0         10.207.39.1     00:16:3E:38:4E:36 9000 
 eth4   192.213.39.10   192.0.0.0       0.0.0.0         10.207.39.1     00:16:3E:68:E8:35 9000 
 eth5   172.220.40.10   172.0.0.0       0.0.0.0         10.207.39.1     00:16:3E:50:81:28 9000 
 eth6   172.221.40.10   172.0.0.0       0.0.0.0         10.207.39.1     00:16:3E:24:C8:4B 9000 


Interface information for node "lxcora02"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU  
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   10.207.39.71    10.207.39.0     0.0.0.0         10.207.39.1     00:16:3E:73:1E:71 1500 
 eth1   192.210.39.11   192.0.0.0       0.0.0.0         10.207.39.1     00:16:3E:F6:30:CA 9000 
 eth2   192.211.39.11   192.0.0.0       0.0.0.0         10.207.39.1     00:16:3E:51:53:30 9000 
 eth3   192.212.39.11   192.0.0.0       0.0.0.0         10.207.39.1     00:16:3E:38:9C:D7 9000 
 eth4   192.213.39.11   192.0.0.0       0.0.0.0         10.207.39.1     00:16:3E:B5:65:3F 9000 
 eth5   172.220.40.11   172.0.0.0       0.0.0.0         10.207.39.1     00:16:3E:D4:A0:55 9000 
 eth6   172.221.40.11   172.0.0.0       0.0.0.0         10.207.39.1     00:16:3E:36:22:62 9000 


Check: Node connectivity of subnet "10.207.39.0"
  Source                          Destination                     Connected?     
  ------------------------------  ------------------------------  ----------------
  lxcora01[10.207.39.70]          lxcora02[10.207.39.71]          yes            
Result: Node connectivity passed for subnet "10.207.39.0" with node(s) lxcora01,lxcora02


Check: TCP connectivity of subnet "10.207.39.0"
  Source                          Destination                     Connected?     
  ------------------------------  ------------------------------  ----------------
  lxcora01 : 10.207.39.70         lxcora01 : 10.207.39.70         passed         
  lxcora02 : 10.207.39.71         lxcora01 : 10.207.39.70         passed         
  lxcora01 : 10.207.39.70         lxcora02 : 10.207.39.71         passed         
  lxcora02 : 10.207.39.71         lxcora02 : 10.207.39.71         passed         
Result: TCP connectivity check passed for subnet "10.207.39.0"


Check: Node connectivity of subnet "192.0.0.0"
  Source                          Destination                     Connected?     
  ------------------------------  ------------------------------  ----------------
  lxcora01[192.210.39.10]         lxcora01[192.211.39.10]         yes            
  lxcora01[192.210.39.10]         lxcora01[192.212.39.10]         yes            
  lxcora01[192.210.39.10]         lxcora01[192.213.39.10]         yes            
  lxcora01[192.210.39.10]         lxcora02[192.210.39.11]         yes            
  lxcora01[192.210.39.10]         lxcora02[192.211.39.11]         yes            
  lxcora01[192.210.39.10]         lxcora02[192.212.39.11]         yes            
  lxcora01[192.210.39.10]         lxcora02[192.213.39.11]         yes            
  lxcora01[192.211.39.10]         lxcora01[192.212.39.10]         yes            
  lxcora01[192.211.39.10]         lxcora01[192.213.39.10]         yes            
  lxcora01[192.211.39.10]         lxcora02[192.210.39.11]         yes            
  lxcora01[192.211.39.10]         lxcora02[192.211.39.11]         yes            
  lxcora01[192.211.39.10]         lxcora02[192.212.39.11]         yes            
  lxcora01[192.211.39.10]         lxcora02[192.213.39.11]         yes            
  lxcora01[192.212.39.10]         lxcora01[192.213.39.10]         yes            
  lxcora01[192.212.39.10]         lxcora02[192.210.39.11]         yes            
  lxcora01[192.212.39.10]         lxcora02[192.211.39.11]         yes            
  lxcora01[192.212.39.10]         lxcora02[192.212.39.11]         yes            
  lxcora01[192.212.39.10]         lxcora02[192.213.39.11]         yes            
  lxcora01[192.213.39.10]         lxcora02[192.210.39.11]         yes            
  lxcora01[192.213.39.10]         lxcora02[192.211.39.11]         yes            
  lxcora01[192.213.39.10]         lxcora02[192.212.39.11]         yes            
  lxcora01[192.213.39.10]         lxcora02[192.213.39.11]         yes            
  lxcora02[192.210.39.11]         lxcora02[192.211.39.11]         yes            
  lxcora02[192.210.39.11]         lxcora02[192.212.39.11]         yes            
  lxcora02[192.210.39.11]         lxcora02[192.213.39.11]         yes            
  lxcora02[192.211.39.11]         lxcora02[192.212.39.11]         yes            
  lxcora02[192.211.39.11]         lxcora02[192.213.39.11]         yes            
  lxcora02[192.212.39.11]         lxcora02[192.213.39.11]         yes            
Result: Node connectivity passed for subnet "192.0.0.0" with node(s) lxcora01,lxcora02


Check: TCP connectivity of subnet "192.0.0.0"
  Source                          Destination                     Connected?     
  ------------------------------  ------------------------------  ----------------
  lxcora01 : 192.210.39.10        lxcora01 : 192.210.39.10        passed         
  lxcora01 : 192.211.39.10        lxcora01 : 192.210.39.10        passed         
  lxcora01 : 192.212.39.10        lxcora01 : 192.210.39.10        passed         
  lxcora01 : 192.213.39.10        lxcora01 : 192.210.39.10        passed         
  lxcora02 : 192.210.39.11        lxcora01 : 192.210.39.10        passed         
  lxcora02 : 192.211.39.11        lxcora01 : 192.210.39.10        passed         
  lxcora02 : 192.212.39.11        lxcora01 : 192.210.39.10        passed         
  lxcora02 : 192.213.39.11        lxcora01 : 192.210.39.10        passed         
  lxcora01 : 192.210.39.10        lxcora01 : 192.211.39.10        passed         
  lxcora01 : 192.211.39.10        lxcora01 : 192.211.39.10        passed         
  lxcora01 : 192.212.39.10        lxcora01 : 192.211.39.10        passed         
  lxcora01 : 192.213.39.10        lxcora01 : 192.211.39.10        passed         
  lxcora02 : 192.210.39.11        lxcora01 : 192.211.39.10        passed         
  lxcora02 : 192.211.39.11        lxcora01 : 192.211.39.10        passed         
  lxcora02 : 192.212.39.11        lxcora01 : 192.211.39.10        passed         
  lxcora02 : 192.213.39.11        lxcora01 : 192.211.39.10        passed         
  lxcora01 : 192.210.39.10        lxcora01 : 192.212.39.10        passed         
  lxcora01 : 192.211.39.10        lxcora01 : 192.212.39.10        passed         
  lxcora01 : 192.212.39.10        lxcora01 : 192.212.39.10        passed         
  lxcora01 : 192.213.39.10        lxcora01 : 192.212.39.10        passed         
  lxcora02 : 192.210.39.11        lxcora01 : 192.212.39.10        passed         
  lxcora02 : 192.211.39.11        lxcora01 : 192.212.39.10        passed         
  lxcora02 : 192.212.39.11        lxcora01 : 192.212.39.10        passed         
  lxcora02 : 192.213.39.11        lxcora01 : 192.212.39.10        passed         
  lxcora01 : 192.210.39.10        lxcora01 : 192.213.39.10        passed         
  lxcora01 : 192.211.39.10        lxcora01 : 192.213.39.10        passed         
  lxcora01 : 192.212.39.10        lxcora01 : 192.213.39.10        passed         
  lxcora01 : 192.213.39.10        lxcora01 : 192.213.39.10        passed         
  lxcora02 : 192.210.39.11        lxcora01 : 192.213.39.10        passed         
  lxcora02 : 192.211.39.11        lxcora01 : 192.213.39.10        passed         
  lxcora02 : 192.212.39.11        lxcora01 : 192.213.39.10        passed         
  lxcora02 : 192.213.39.11        lxcora01 : 192.213.39.10        passed         
  lxcora01 : 192.210.39.10        lxcora02 : 192.210.39.11        passed         
  lxcora01 : 192.211.39.10        lxcora02 : 192.210.39.11        passed         
  lxcora01 : 192.212.39.10        lxcora02 : 192.210.39.11        passed         
  lxcora01 : 192.213.39.10        lxcora02 : 192.210.39.11        passed         
  lxcora02 : 192.210.39.11        lxcora02 : 192.210.39.11        passed         
  lxcora02 : 192.211.39.11        lxcora02 : 192.210.39.11        passed         
  lxcora02 : 192.212.39.11        lxcora02 : 192.210.39.11        passed         
  lxcora02 : 192.213.39.11        lxcora02 : 192.210.39.11        passed         
  lxcora01 : 192.210.39.10        lxcora02 : 192.211.39.11        passed         
  lxcora01 : 192.211.39.10        lxcora02 : 192.211.39.11        passed         
  lxcora01 : 192.212.39.10        lxcora02 : 192.211.39.11        passed         
  lxcora01 : 192.213.39.10        lxcora02 : 192.211.39.11        passed         
  lxcora02 : 192.210.39.11        lxcora02 : 192.211.39.11        passed         
  lxcora02 : 192.211.39.11        lxcora02 : 192.211.39.11        passed         
  lxcora02 : 192.212.39.11        lxcora02 : 192.211.39.11        passed         
  lxcora02 : 192.213.39.11        lxcora02 : 192.211.39.11        passed         
  lxcora01 : 192.210.39.10        lxcora02 : 192.212.39.11        passed         
  lxcora01 : 192.211.39.10        lxcora02 : 192.212.39.11        passed         
  lxcora01 : 192.212.39.10        lxcora02 : 192.212.39.11        passed         
  lxcora01 : 192.213.39.10        lxcora02 : 192.212.39.11        passed         
  lxcora02 : 192.210.39.11        lxcora02 : 192.212.39.11        passed         
  lxcora02 : 192.211.39.11        lxcora02 : 192.212.39.11        passed         
  lxcora02 : 192.212.39.11        lxcora02 : 192.212.39.11        passed         
  lxcora02 : 192.213.39.11        lxcora02 : 192.212.39.11        passed         
  lxcora01 : 192.210.39.10        lxcora02 : 192.213.39.11        passed         
  lxcora01 : 192.211.39.10        lxcora02 : 192.213.39.11        passed         
  lxcora01 : 192.212.39.10        lxcora02 : 192.213.39.11        passed         
  lxcora01 : 192.213.39.10        lxcora02 : 192.213.39.11        passed         
  lxcora02 : 192.210.39.11        lxcora02 : 192.213.39.11        passed         
  lxcora02 : 192.211.39.11        lxcora02 : 192.213.39.11        passed         
  lxcora02 : 192.212.39.11        lxcora02 : 192.213.39.11        passed         
  lxcora02 : 192.213.39.11        lxcora02 : 192.213.39.11        passed         
Result: TCP connectivity check passed for subnet "192.0.0.0"


Check: Node connectivity of subnet "172.0.0.0"
  Source                          Destination                     Connected?     
  ------------------------------  ------------------------------  ----------------
  lxcora01[172.220.40.10]         lxcora01[172.221.40.10]         yes            
  lxcora01[172.220.40.10]         lxcora02[172.220.40.11]         yes            
  lxcora01[172.220.40.10]         lxcora02[172.221.40.11]         yes            
  lxcora01[172.221.40.10]         lxcora02[172.220.40.11]         yes            
  lxcora01[172.221.40.10]         lxcora02[172.221.40.11]         yes            
  lxcora02[172.220.40.11]         lxcora02[172.221.40.11]         yes            
Result: Node connectivity passed for subnet "172.0.0.0" with node(s) lxcora01,lxcora02


Check: TCP connectivity of subnet "172.0.0.0"
  Source                          Destination                     Connected?     
  ------------------------------  ------------------------------  ----------------
  lxcora01 : 172.220.40.10        lxcora01 : 172.220.40.10        passed         
  lxcora01 : 172.221.40.10        lxcora01 : 172.220.40.10        passed         
  lxcora02 : 172.220.40.11        lxcora01 : 172.220.40.10        passed         
  lxcora02 : 172.221.40.11        lxcora01 : 172.220.40.10        passed         
  lxcora01 : 172.220.40.10        lxcora01 : 172.221.40.10        passed         
  lxcora01 : 172.221.40.10        lxcora01 : 172.221.40.10        passed         
  lxcora02 : 172.220.40.11        lxcora01 : 172.221.40.10        passed         
  lxcora02 : 172.221.40.11        lxcora01 : 172.221.40.10        passed         
  lxcora01 : 172.220.40.10        lxcora02 : 172.220.40.11        passed         
  lxcora01 : 172.221.40.10        lxcora02 : 172.220.40.11        passed         
  lxcora02 : 172.220.40.11        lxcora02 : 172.220.40.11        passed         
  lxcora02 : 172.221.40.11        lxcora02 : 172.220.40.11        passed         
  lxcora01 : 172.220.40.10        lxcora02 : 172.221.40.11        passed         
  lxcora01 : 172.221.40.10        lxcora02 : 172.221.40.11        passed         
  lxcora02 : 172.220.40.11        lxcora02 : 172.221.40.11        passed         
  lxcora02 : 172.221.40.11        lxcora02 : 172.221.40.11        passed         
Result: TCP connectivity check passed for subnet "172.0.0.0"


Interfaces found on subnet "10.207.39.0" that are likely candidates for VIP are:
lxcora01 eth0:10.207.39.70
lxcora02 eth0:10.207.39.71

Interfaces found on subnet "192.0.0.0" that are likely candidates for VIP are:
lxcora01 eth1:192.210.39.10
lxcora02 eth1:192.210.39.11

Interfaces found on subnet "192.0.0.0" that are likely candidates for VIP are:
lxcora01 eth2:192.211.39.10
lxcora02 eth2:192.211.39.11

Interfaces found on subnet "192.0.0.0" that are likely candidates for VIP are:
lxcora01 eth3:192.212.39.10
lxcora02 eth3:192.212.39.11

Interfaces found on subnet "192.0.0.0" that are likely candidates for VIP are:
lxcora01 eth4:192.213.39.10
lxcora02 eth4:192.213.39.11

Interfaces found on subnet "172.0.0.0" that are likely candidates for VIP are:
lxcora01 eth5:172.220.40.10
lxcora02 eth5:172.220.40.11

Interfaces found on subnet "172.0.0.0" that are likely candidates for VIP are:
lxcora01 eth6:172.221.40.10
lxcora02 eth6:172.221.40.11

WARNING:
Could not find a suitable set of interfaces for the private interconnect
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "10.207.39.0".
Subnet mask consistency check passed for subnet "192.0.0.0".
Subnet mask consistency check passed for subnet "172.0.0.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "10.207.39.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "10.207.39.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.

Check: Total memory
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      30.2745GB (3.174514E7KB)  4GB (4194304.0KB)         passed   
  lxcora01      30.2745GB (3.174514E7KB)  4GB (4194304.0KB)         passed   
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      30.2627GB (3.1732696E7KB)  50MB (51200.0KB)          passed   
  lxcora01      30.0516GB (3.1511436E7KB)  50MB (51200.0KB)          passed   
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      30.832GB (3.2329724E7KB)  16GB (1.6777216E7KB)      passed   
  lxcora01      30.832GB (3.2329724E7KB)  16GB (1.6777216E7KB)      passed   
Result: Swap space check passed

Check: Free disk space for "lxcora02:/usr,lxcora02:/var,lxcora02:/etc,lxcora02:/sbin,lxcora02:/tmp"
  Path              Node Name     Mount point   Available     Required      Status     
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              lxcora02      /             849.4424GB    1.0635GB      passed     
  /var              lxcora02      /             849.4424GB    1.0635GB      passed     
  /etc              lxcora02      /             849.4424GB    1.0635GB      passed     
  /sbin             lxcora02      /             849.4424GB    1.0635GB      passed     
  /tmp              lxcora02      /             849.4424GB    1.0635GB      passed     
Result: Free disk space check passed for "lxcora02:/usr,lxcora02:/var,lxcora02:/etc,lxcora02:/sbin,lxcora02:/tmp"

Check: Free disk space for "lxcora01:/usr,lxcora01:/var,lxcora01:/etc,lxcora01:/sbin,lxcora01:/tmp"
  Path              Node Name     Mount point   Available     Required      Status     
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              lxcora01      /             849.4431GB    1.0635GB      passed     
  /var              lxcora01      /             849.4431GB    1.0635GB      passed     
  /etc              lxcora01      /             849.4431GB    1.0635GB      passed     
  /sbin             lxcora01      /             849.4431GB    1.0635GB      passed     
  /tmp              lxcora01      /             849.4431GB    1.0635GB      passed     
Result: Free disk space check passed for "lxcora01:/usr,lxcora01:/var,lxcora01:/etc,lxcora01:/sbin,lxcora01:/tmp"

Check: User existence for "grid"
  Node Name     Status                    Comment                
  ------------  ------------------------  ------------------------
  lxcora02      passed                    exists(1098)           
  lxcora01      passed                    exists(1098)           

Checking for multiple users with UID value 1098
Result: Check for multiple users with UID value 1098 passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment                
  ------------  ------------------------  ------------------------
  lxcora02      passed                    exists                 
  lxcora01      passed                    exists                 
Result: Group existence check passed for "oinstall"

Check: Group existence for "asmadmin"
  Node Name     Status                    Comment                
  ------------  ------------------------  ------------------------
  lxcora02      passed                    exists                 
  lxcora01      passed                    exists                 
Result: Group existence check passed for "asmadmin"

Check: Membership of user "grid" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status     
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora02          yes           yes           yes           yes           passed     
  lxcora01          yes           yes           yes           yes           passed     
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed

Check: Membership of user "grid" in group "asmadmin"
  Node Name         User Exists   Group Exists  User in Group  Status         
  ----------------  ------------  ------------  ------------  ----------------
  lxcora02          yes           yes           yes           passed         
  lxcora01          yes           yes           yes           passed         
Result: Membership check for user "grid" in group "asmadmin" passed

Check: Run level
  Node Name     run level                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      3                         3,5                       passed   
  lxcora01      3                         3,5                       passed   
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status         
  ----------------  ------------  ------------  ------------  ----------------
  lxcora02          hard          65536         65536         passed         
  lxcora01          hard          65536         65536         passed         
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status         
  ----------------  ------------  ------------  ------------  ----------------
  lxcora02          soft          65536         1024          passed         
  lxcora01          soft          65536         1024          passed         
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status         
  ----------------  ------------  ------------  ------------  ----------------
  lxcora02          hard          123913        16384         passed         
  lxcora01          hard          123913        16384         passed         
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status         
  ----------------  ------------  ------------  ------------  ----------------
  lxcora02          soft          2047          2047          passed         
  lxcora01          soft          2047          2047          passed         
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      x86_64                    x86_64                    passed   
  lxcora01      x86_64                    x86_64                    passed   
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      3.19.0-21-generic         2.6.39                    passed   
  lxcora01      3.19.0-21-generic         2.6.39                    passed   
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          250           250           250           passed         
  lxcora02          250           250           250           passed         
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          32000         32000         32000         passed         
  lxcora02          32000         32000         32000         passed         
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          100           100           100           passed         
  lxcora02          100           100           100           passed         
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          128           128           128           passed         
  lxcora02          128           128           128           passed         
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          68719476736   68719476736   16253511680   passed         
  lxcora02          68719476736   68719476736   16253511680   passed         
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          4096          4096          4096          passed         
  lxcora02          4096          4096          4096          passed         
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          4294967296    4294967296    3174514       passed         
  lxcora02          4294967296    4294967296    3174514       passed         
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          6815744       6815744       6815744       passed         
  lxcora02          6815744       6815744       6815744       passed         
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed         
  lxcora02          between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed         
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          unknown       262144        262144        failed        Current value incorrect.
  lxcora02          unknown       262144        262144        failed        Current value incorrect.

PRVF-7544 : Check cannot be performed for kernel parameter "rmem_default" on node "lxcora01"
PRVG-2044 : Command "/sbin/sysctl -a | grep rmem_default[[:space:]]*=" failed on node "lxcora01" and produced no output.
PRVF-7544 : Check cannot be performed for kernel parameter "rmem_default" on node "lxcora02"
PRVG-2044 : Command "/sbin/sysctl -a | grep rmem_default[[:space:]]*=" failed on node "lxcora02" and produced no output.

Result: Kernel parameter check failed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          unknown       4194304       4194304       failed        Current value incorrect.
  lxcora02          unknown       4194304       4194304       failed        Current value incorrect.

PRVF-7544 : Check cannot be performed for kernel parameter "rmem_max" on node "lxcora01"
PRVG-2044 : Command "/sbin/sysctl -a | grep rmem_max[[:space:]]*=" failed on node "lxcora01" and produced no output.
PRVF-7544 : Check cannot be performed for kernel parameter "rmem_max" on node "lxcora02"
PRVG-2044 : Command "/sbin/sysctl -a | grep rmem_max[[:space:]]*=" failed on node "lxcora02" and produced no output.

Result: Kernel parameter check failed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          unknown       262144        262144        failed        Current value incorrect.
  lxcora02          unknown       262144        262144        failed        Current value incorrect.

PRVF-7544 : Check cannot be performed for kernel parameter "wmem_default" on node "lxcora01"
PRVG-2044 : Command "/sbin/sysctl -a | grep wmem_default[[:space:]]*=" failed on node "lxcora01" and produced no output.
PRVF-7544 : Check cannot be performed for kernel parameter "wmem_default" on node "lxcora02"
PRVG-2044 : Command "/sbin/sysctl -a | grep wmem_default[[:space:]]*=" failed on node "lxcora02" and produced no output.

Result: Kernel parameter check failed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          unknown       1048576       1048576       failed        Current value incorrect.
  lxcora02          unknown       1048576       1048576       failed        Current value incorrect.

PRVF-7544 : Check cannot be performed for kernel parameter "wmem_max" on node "lxcora01"
PRVG-2044 : Command "/sbin/sysctl -a | grep wmem_max[[:space:]]*=" failed on node "lxcora01" and produced no output.
PRVF-7544 : Check cannot be performed for kernel parameter "wmem_max" on node "lxcora02"
PRVG-2044 : Command "/sbin/sysctl -a | grep wmem_max[[:space:]]*=" failed on node "lxcora02" and produced no output.

Result: Kernel parameter check failed for "wmem_max"


Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          1048576       1048576       1048576       passed         
  lxcora02          1048576       1048576       1048576       passed         
Result: Kernel parameter check passed for "aio-max-nr"

Check: Kernel parameter for "panic_on_oops"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          1             1             1             passed         
  lxcora02          1             1             1             passed         
Result: Kernel parameter check passed for "panic_on_oops"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed   
  lxcora01      binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed   
Result: Package existence check passed for "binutils"

Check: Package existence for "compat-libcap1"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      compat-libcap1-1.10-1     compat-libcap1-1.10       passed   
  lxcora01      compat-libcap1-1.10-1     compat-libcap1-1.10       passed   
Result: Package existence check passed for "compat-libcap1"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed   
  lxcora01      compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed   
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      libgcc(x86_64)-4.4.7-4.el6  libgcc(x86_64)-4.4.4      passed   
  lxcora01      libgcc(x86_64)-4.4.7-4.el6  libgcc(x86_64)-4.4.4      passed   
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      libstdc++(x86_64)-4.4.7-4.el6  libstdc++(x86_64)-4.4.4   passed   
  lxcora01      libstdc++(x86_64)-4.4.7-4.el6  libstdc++(x86_64)-4.4.4   passed   
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      libstdc++-devel(x86_64)-4.4.7-4.el6  libstdc++-devel(x86_64)-4.4.4  passed   
  lxcora01      libstdc++-devel(x86_64)-4.4.7-4.el6  libstdc++-devel(x86_64)-4.4.4  passed   
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      sysstat-9.0.4-22.el6      sysstat-9.0.4             passed   
  lxcora01      sysstat-9.0.4-22.el6      sysstat-9.0.4             passed   
Result: Package existence check passed for "sysstat"

Check: Package existence for "gcc"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      gcc-4.4.7-4.el6           gcc-4.4.4                 passed   
  lxcora01      gcc-4.4.7-4.el6           gcc-4.4.4                 passed   
Result: Package existence check passed for "gcc"

Check: Package existence for "gcc-c++"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      gcc-c++-4.4.7-4.el6       gcc-c++-4.4.4             passed   
  lxcora01      gcc-c++-4.4.7-4.el6       gcc-c++-4.4.4             passed   
Result: Package existence check passed for "gcc-c++"

Check: Package existence for "ksh"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      ksh                       ksh                       passed   
  lxcora01      ksh                       ksh                       passed   
Result: Package existence check passed for "ksh"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      make-3.81-20.el6          make-3.81                 passed   
  lxcora01      make-3.81-20.el6          make-3.81                 passed   
Result: Package existence check passed for "make"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      glibc(x86_64)-2.12-1.132.el6  glibc(x86_64)-2.12        passed   
  lxcora01      glibc(x86_64)-2.12-1.132.el6  glibc(x86_64)-2.12        passed   
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      glibc-devel(x86_64)-2.12-1.132.el6  glibc-devel(x86_64)-2.12  passed   
  lxcora01      glibc-devel(x86_64)-2.12-1.132.el6  glibc-devel(x86_64)-2.12  passed   
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed   
  lxcora01      libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed   
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed   
  lxcora01      libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed   
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "nfs-utils"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      nfs-utils-1.2.3-39.el6    nfs-utils-1.2.3-15        passed   
  lxcora01      nfs-utils-1.2.3-39.el6    nfs-utils-1.2.3-15        passed   
Result: Package existence check passed for "nfs-utils"

Checking availability of ports "6200,6100" required for component "Oracle Notification Service (ONS)"
  Node Name         Port Number   Protocol      Available     Status         
  ----------------  ------------  ------------  ------------  ----------------
  lxcora02          6200          TCP           yes           successful     
  lxcora01          6200          TCP           yes           successful     
  lxcora02          6100          TCP           yes           successful     
  lxcora01          6100          TCP           yes           successful     
Result: Port availability check passed for ports "6200,6100"

Checking availability of ports "42424" required for component "Oracle Cluster Synchronization Services (CSSD)"
  Node Name         Port Number   Protocol      Available     Status         
  ----------------  ------------  ------------  ------------  ----------------
  lxcora02          42424         TCP           yes           successful     
  lxcora01          42424         TCP           yes           successful     
Result: Port availability check passed for ports "42424"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status                 
  ------------------------------------  ------------------------
  lxcora02                              passed                 
  lxcora01                              passed                 

Check for consistency of root user's primary group passed


Checking Devices for ASM...

Check: Package existence for "cvuqdisk"
  Node Name     Available                 Required                  Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed   
  lxcora01      cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed   
Result: Package existence check passed for "cvuqdisk"

Checking for shared devices...

PRVG-5150 : could not determine if path /oradata/AsmSystem_01.img is a valid path on all nodes
PRVG-5150 : could not determine if path /oradata/AsmSystem_03.img is a valid path on all nodes
PRVG-5150 : could not determine if path /oradata/AsmSystem_02.img is a valid path on all nodes


Checking consistency of device owner across all nodes...
Consistency check of device owner for "/oradata/AsmSystem_03.img" PASSED
Consistency check of device owner for "/oradata/AsmSystem_02.img" PASSED
Consistency check of device owner for "/oradata/AsmSystem_01.img" PASSED


Checking consistency of device group across all nodes...
Consistency check of device group for "/oradata/AsmSystem_03.img" PASSED
Consistency check of device group for "/oradata/AsmSystem_02.img" PASSED
Consistency check of device group for "/oradata/AsmSystem_01.img" PASSED


Checking consistency of device permissions across all nodes...
Consistency check of device permissions for "/oradata/AsmSystem_03.img" PASSED
Consistency check of device permissions for "/oradata/AsmSystem_02.img" PASSED
Consistency check of device permissions for "/oradata/AsmSystem_01.img" PASSED


Checking consistency of device size across all nodes...
Consistency check of device size for "/oradata/AsmSystem_03.img" PASSED
Consistency check of device size for "/oradata/AsmSystem_02.img" PASSED
Consistency check of device size for "/oradata/AsmSystem_01.img" PASSED

UDev attributes check for ASM Disks started...
Result: UDev attributes check passed for ASM Disks


Checking that ASM devices are ASM Filter Driver capable.

WARNING:
PRVG-12805 : Device "/oradata/AsmSystem_01.img" cannot be managed by the ASM Filter Driver on node "lxcora02".
PRVG-12805 : Device "/oradata/AsmSystem_02.img" cannot be managed by the ASM Filter Driver on node "lxcora02".
PRVG-12805 : Device "/oradata/AsmSystem_03.img" cannot be managed by the ASM Filter Driver on node "lxcora02".
PRVG-12805 : Device "/oradata/AsmSystem_01.img" cannot be managed by the ASM Filter Driver on node "lxcora01".
PRVG-12805 : Device "/oradata/AsmSystem_02.img" cannot be managed by the ASM Filter Driver on node "lxcora01".
PRVG-12805 : Device "/oradata/AsmSystem_03.img" cannot be managed by the ASM Filter Driver on node "lxcora01".
Result: Devices check for ASM failed


Starting Clock synchronization checks using Network Time Protocol(NTP)...

Checking existence of NTP configuration file "/etc/ntp.conf" across nodes
  Node Name                             File exists?           
  ------------------------------------  ------------------------
  lxcora02                              yes                    
  lxcora01                              yes                    
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP configuration file "/etc/ntp.conf" existence check passed
No NTP Daemons or Services were found to be running
PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):
lxcora02,lxcora01
Result: Clock synchronization check using Network Time Protocol(NTP) failed


Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "grid" is not in "root" group
  Node Name     Status                    Comment                
  ------------  ------------------------  ------------------------
  lxcora02      passed                    does not exist         
  lxcora01      passed                    does not exist         
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment  
  ------------  ------------------------  ------------------------  ----------
  lxcora02      0022                      0022                      passed   
  lxcora01      0022                      0022                      passed   
Result: Default user file creation mask check passed
Checking integrity of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of 'domain' and 'search' entries is defined
"domain" and "search" entries do not coexist in any "/etc/resolv.conf" file
Checking if 'domain' entry in file "/etc/resolv.conf" is consistent across the nodes...
"domain" entry does not exist in any "/etc/resolv.conf" file
Checking if 'search' entry in file "/etc/resolv.conf" is consistent across the nodes...
Checking file "/etc/resolv.conf" to make sure that only one 'search' entry is defined
More than one "search" entry does not exist in any "/etc/resolv.conf" file
All nodes have same "search" order defined in file "/etc/resolv.conf"
Checking DNS response time for an unreachable node
  Node Name                             Status                 
  ------------------------------------  ------------------------
  lxcora01                              passed                 
  lxcora02                              passed                 
The DNS response time for an unreachable node is within acceptable limit on all nodes
checking DNS response from all servers in "/etc/resolv.conf"
checking response for name "lxcora02" from each of the name servers specified in "/etc/resolv.conf"
  Node Name     Source                    Comment                   Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora02      10.207.39.1               IPv4                      passed   
  lxcora02      8.8.8.8                   IPv4                      passed   
checking response for name "lxcora01" from each of the name servers specified in "/etc/resolv.conf"
  Node Name     Source                    Comment                   Status   
  ------------  ------------------------  ------------------------  ----------
  lxcora01      10.207.39.1               IPv4                      passed   
  lxcora01      8.8.8.8                   IPv4                      passed   

Check for integrity of file "/etc/resolv.conf" failed

Check: Time zone consistency
Result: Time zone consistency check passed

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...
Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined
More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed


Checking daemon "avahi-daemon" is not configured and running

Check: Daemon "avahi-daemon" not configured
  Node Name     Configured                Status                 
  ------------  ------------------------  ------------------------
  lxcora02      no                        passed                 
  lxcora01      no                        passed                 
Daemon not configured check passed for process "avahi-daemon"

Check: Daemon "avahi-daemon" not running
  Node Name     Running?                  Status                 
  ------------  ------------------------  ------------------------
  lxcora02      no                        passed                 
  lxcora01      no                        passed                 
Daemon not running check passed for process "avahi-daemon"

Starting check for /dev/shm mounted as temporary file system ...

Check for /dev/shm mounted as temporary file system passed

Starting check for /boot mount ...

ERROR:

PRVE-10073 : Required /boot data is not available on node "lxcora02"
PRVE-10073 : Required /boot data is not available on node "lxcora02"
PRVE-10073 : Required /boot data is not available on node "lxcora01"
PRVE-10073 : Required /boot data is not available on node "lxcora01"

Check for /boot mount failed


Starting check for zeroconf check ...

Check for zeroconf check passed

Pre-check for cluster services setup was unsuccessful on all the nodes.
[grid@lxcora01 grid]$

Explanation of Cluster Verification Exceptions

The /boot Mount Exception

The /boot mount error arises because /boot for an LXC Oracle Enterprise Linux 6.5 container is very different than what would be found in an actual bare metal OEL 6.5 server.  This exception can be safely ignored.

The NTP Exception

There are issues with running NTP in a container.  For one thing, NTP cannot be started in the normal way using "chkconfig" for unknown reasons.  One point to understand is that NTP runs on the base Ubuntu OS.  However, during install of Oracle Clusterware, the installer will be looking for NTP so it is useful to have NTP running on the LXC containers. 

The workaround is to start NTP in /etc/rc.local as was previously described in an earlier section.  Therefore, to validate this check, manually check to ensure that NTP is up and running as shown below.  If NTP passes the below manual checks, it is safe to proceed with the Oracle Clusterware install.

gstanden@u1504:~$ ssh root@lxcora01
root@lxcora01's password:
Last login: Sun Jun 28 12:55:10 2015

[root@lxcora01 ~]# ps -ef | grep ntp

root       360     1  0 12:55 ?        00:00:00 ntpd                            <-- Verify ntpd is running on both nodes
root      3429  3417  0 13:39 pts/4    00:00:00 grep ntp

[root@lxcora01 ~]# ntpq -p                                                      <-- Verify ntpq output on both nodes
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*pool-test.ntp.o 216.218.254.202  2 u  123  128  377   83.394    0.016  37.793
-pawn.slash31.co 128.227.205.3    2 u   58  128  277   39.094   -4.702 194.382
+142.54.181.202  64.113.32.5      2 u   90  128  317   47.164    1.978  68.609
+ntp.mocana.com  204.26.59.157    3 u  225  128  376   84.518   -1.296 273.107

[root@lxcora01 ~]# su - grid
[grid@lxcora01 ~]$ ./no-passwd.sh                                               <-- Verify that time is synchronized on nodes

Sun Jun 28 13:39:42 EDT 2015
Sun Jun 28 13:39:42 EDT 2015
Sun Jun 28 13:39:42 EDT 2015
Sun Jun 28 13:39:42 EDT 2015
Sun Jun 28 13:39:43 EDT 2015
Sun Jun 28 13:39:43 EDT 2015
Sun Jun 28 13:39:43 EDT 2015
Sun Jun 28 13:39:43 EDT 2015

[grid@lxcora01 ~]$ exit
logout
[root@lxcora01 ~]# ssh lxcora02
root@lxcora02's password:
Last login: Sun Jun 28 12:57:00 2015

[root@lxcora02 ~]# ps -ef | grep ntp

root       360     1  0 12:56 ?        00:00:00 ntpd
root      3086  3074  0 13:39 pts/4    00:00:00 grep ntp

[root@lxcora02 ~]# ntpq -p

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-108.61.194.85.v 200.23.51.102    2 u   58   64  337   89.653    6.003 195.037
*pawn.slash31.co 128.227.205.3    2 u  159   64  324   39.666   -4.087   1.255
+clock.trit.net  43.77.130.254    2 u   41   64  367   75.905   -4.368 136.249
+kahuna.ruselabs 199.102.46.70    2 u   10   64  207   20.295    2.602   0.701

[root@lxcora02 ~]# su - grid
[grid@lxcora02 ~]$ cat no-passwd.sh                                            <-- no-passwd.sh script checks both nodes

ssh lxcora02 date
ssh lxcora01 date
ssh lxcora01.vmem.org date
ssh lxcora02.vmem.org date
ssh 10.207.39.70 date
ssh 10.207.39.71 date
ssh localhost date
ssh 127.0.0.1 date

[grid@lxcora02 ~]$ ./no-passwd.sh 

Sun Jun 28 13:40:23 EDT 2015
Sun Jun 28 13:40:23 EDT 2015
Sun Jun 28 13:40:23 EDT 2015
Sun Jun 28 13:40:23 EDT 2015
Sun Jun 28 13:40:23 EDT 2015
Sun Jun 28 13:40:24 EDT 2015
Sun Jun 28 13:40:24 EDT 2015
Sun Jun 28 13:40:24 EDT 2015

[grid@lxcora02 ~]$

The ASM Filter Driver Warning Exception

The ASM Filter Driver is an optional storage feature which does not have to be used and is only a warning.  It can be safely ignored if not electing to use the ASM Filter Driver.

Checking that ASM devices are ASM Filter Driver capable.

WARNING:
PRVG-12805 : Device "/oradata/AsmSystem_01.img" cannot be managed by the ASM Filter Driver on node "lxcora02".
PRVG-12805 : Device "/oradata/AsmSystem_02.img" cannot be managed by the ASM Filter Driver on node "lxcora02".
PRVG-12805 : Device "/oradata/AsmSystem_03.img" cannot be managed by the ASM Filter Driver on node "lxcora02".
PRVG-12805 : Device "/oradata/AsmSystem_01.img" cannot be managed by the ASM Filter Driver on node "lxcora01".
PRVG-12805 : Device "/oradata/AsmSystem_02.img" cannot be managed by the ASM Filter Driver on node "lxcora01".
PRVG-12805 : Device "/oradata/AsmSystem_03.img" cannot be managed by the ASM Filter Driver on node "lxcora01".
Result: Devices check for ASM failed

The ASM Shared Devices Exception

This is a known-issue with CVU sometimes not being able to execute this check correctly, and can safely be ignored.

Checking for shared devices...

PRVG-5150 : could not determine if path /oradata/AsmSystem_01.img is a valid path on all nodes
PRVG-5150 : could not determine if path /oradata/AsmSystem_03.img is a valid path on all nodes
PRVG-5150 : could not determine if path /oradata/AsmSystem_02.img is a valid path on all nodes

The rmem and wmem Kernel Parameter Exceptions

The rmem_default, rmem_max, wmem_default, and wmem_max are not supported inside the LXC containers.  However, note that LXC containers all share the same kernel as the base OS, in this case, Ubuntu, so if these are set to correct values in the Ubuntu Base OS kernel, then they will in fact be correct, even though the "sysctl -a | grep wmem" checks will fail for CVU, as shown below.

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          unknown       262144        262144        failed        Current value incorrect.
  lxcora02          unknown       262144        262144        failed        Current value incorrect.

PRVF-7544 : Check cannot be performed for kernel parameter "rmem_default" on node "lxcora01"
PRVG-2044 : Command "/sbin/sysctl -a | grep rmem_default[[:space:]]*=" failed on node "lxcora01" and produced no output.
PRVF-7544 : Check cannot be performed for kernel parameter "rmem_default" on node "lxcora02"
PRVG-2044 : Command "/sbin/sysctl -a | grep rmem_default[[:space:]]*=" failed on node "lxcora02" and produced no output.

Result: Kernel parameter check failed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          unknown       4194304       4194304       failed        Current value incorrect.
  lxcora02          unknown       4194304       4194304       failed        Current value incorrect.

PRVF-7544 : Check cannot be performed for kernel parameter "rmem_max" on node "lxcora01"
PRVG-2044 : Command "/sbin/sysctl -a | grep rmem_max[[:space:]]*=" failed on node "lxcora01" and produced no output.
PRVF-7544 : Check cannot be performed for kernel parameter "rmem_max" on node "lxcora02"
PRVG-2044 : Command "/sbin/sysctl -a | grep rmem_max[[:space:]]*=" failed on node "lxcora02" and produced no output.

Result: Kernel parameter check failed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          unknown       262144        262144        failed        Current value incorrect.
  lxcora02          unknown       262144        262144        failed        Current value incorrect.

PRVF-7544 : Check cannot be performed for kernel parameter "wmem_default" on node "lxcora01"
PRVG-2044 : Command "/sbin/sysctl -a | grep wmem_default[[:space:]]*=" failed on node "lxcora01" and produced no output.
PRVF-7544 : Check cannot be performed for kernel parameter "wmem_default" on node "lxcora02"
PRVG-2044 : Command "/sbin/sysctl -a | grep wmem_default[[:space:]]*=" failed on node "lxcora02" and produced no output.

Result: Kernel parameter check failed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment    
  ----------------  ------------  ------------  ------------  ------------  ------------
  lxcora01          unknown       1048576       1048576       failed        Current value incorrect.
  lxcora02          unknown       1048576       1048576       failed        Current value incorrect.

PRVF-7544 : Check cannot be performed for kernel parameter "wmem_max" on node "lxcora01"
PRVG-2044 : Command "/sbin/sysctl -a | grep wmem_max[[:space:]]*=" failed on node "lxcora01" and produced no output.
PRVF-7544 : Check cannot be performed for kernel parameter "wmem_max" on node "lxcora02"
PRVG-2044 : Command "/sbin/sysctl -a | grep wmem_max[[:space:]]*=" failed on node "lxcora02" and produced no output.

Result: Kernel parameter check failed for "wmem_max"

The values can be checked manually at the Ubuntu Host OS level as shown below.  Also, note in the above output that the "Configured" value should match the "Required" value as listed in the CVU output (shown in bold above).  As long as "Configured" value matches the "Required" value for these kernel parameter, and as long as they are set in the Ubuntu Host OS as shown below, this check is passed.

gstanden@u1504:~$ cat /etc/sysctl.conf | egrep 'wmem|rmem'

net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

gstanden@u1504:~$ sudo sysctl -a | egrep 'wmem_|rmem_' | egrep 'default|max'

net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

gstanden@u1504:~$

Create Additional LXC Containers for n-Node Oracle RAC

One of the great things about building Oracle RAC in LXC Containers is that it's feasible to run n-Node RAC because resource-intensive fully-virtualized machines do not need to be created (i.e. only containers are being used) so additional LXC containers can be cloned up to the number of nodes desired for this RAC deployment on Ubuntu 15.04 or whatever LXC base OS is being used.  In this example, six RAC LXC nodes are created for a six-node Oracle RAC deployment on LXC containers.  It's important to use lxcora02 as the source for the additional node clones, because lxcora01 has the install media under /home/grid, which does not need to be cloned as it only needs to exist on the first node, lxcora01, so as shown below, the additional nodes are quickly cloned.

gstanden@u1504:~$ sudo lxc-clone -o lxcora02 -n lxcora03
Created container lxcora03 as copy of lxcora02
gstanden@u1504:~$ sudo lxc-clone -o lxcora02 -n lxcora04
Created container lxcora04 as copy of lxcora02
gstanden@u1504:~$ sudo lxc-clone -o lxcora02 -n lxcora05
Created container lxcora05 as copy of lxcora02
gstanden@u1504:~$ sudo lxc-clone -o lxcora02 -n lxcora06
Created container lxcora06 as copy of lxcora02
gstanden@u1504:~$

Each of the newly created nodes will need all of the following completed before Oracle Clusterware can be installed:

  • Create "/etc/network/if-up.d" and "/etc/network/if-down.d" files for OpenvSwitch (as shown previsouly in section    )
  • Establish no-password SSH for all nodes (every node must be able to ssh to every other node with no password required)
  • Edit "/var/lib/lxc/lxcora0X/config" for OpenvSwitch config stanzas (as shown previously in section  )
  • Re-run "/home/grid/no-passwd.sh" script to ensure that no password is required
  • Re-run the CVU utility to ensure that all cluster verification checks are passed by the added LXC nodes

Create "if-up.d" and "if-down.d" Files for Cloned Nodes

I created a script that can be used to create the needed files for the additional nodes.  The script is not entirely generic so it would need some modification if the names of the LXC containers are not of the form "lxcora0X" so modify it as needed for other container names.

SCST Discovery Issue

If the following issue is encountered when attempting to connect to SCST and discover LUNS as shown below.

[root@oracle651 ~]# iscsiadm --mode discovery --type sendtargets --portal 10.207.40.76

Starting iscsid:                                           [  OK  ]
iscsiadm: Login response timeout. Waited 30 seconds and did not get reponse PDU.

iscsiadm: discovery login to 10.207.40.76 failed, giving up 2
iscsiadm: Could not perform SendTargets discovery: encountered non-retryable iSCSI login failure


[root@oracle651 ~]#

Run the following command as shown below.

[root@oracle651 ~]# scstadmin -set_drv_attr iscsi -attributes enabled=1

Collecting current configuration: done.

Performing this action may result in lost or corrupt data, are you sure you wish to continue (y/[n]) ? y

-> Making requested changes.
    -> Enabling driver 'iscsi': done.
    -> Done, 0 change(s) made.

All done.
[root@oracle651 ~]

Retry discovery as shown below first from the SCST SAN host itself.

[root@oracle651 ~]# iscsiadm -m discovery -t sendtargets -p 10.207.40.76

10.207.40.76:3260,1 iqn.2015-07.org.vmem:oracle651.san.asm.luns
10.207.39.76:3260,1 iqn.2015-07.org.vmem:oracle651.san.asm.luns
10.207.41.76:3260,1 iqn.2015-07.org.vmem:oracle651.san.asm.luns


[root@oracle651 ~]#

Retry discovery as shown below from the Ubuntu SAN client.

gstanden@u1504:~$ sudo iscsiadm -m discovery -t sendtargets -p 10.207.40.76
[sudo] password for gstanden:

10.207.40.76:3260,1 iqn.2015-07.org.vmem:oracle651.san.asm.luns
10.207.39.76:3260,1 iqn.2015-07.org.vmem:oracle651.san.asm.luns
10.207.41.76:3260,1 iqn.2015-07.org.vmem:oracle651.san.asm.luns


gstanden@u1504:~$

Note that the initiators were previously added to the SCST target group as shown below.

[root@oracle651 ~]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v3.0.2.


HANDLER vdisk_fileio {
    DEVICE systemdg_01 {
        filename /storage_0512/systemdg_01.img
    }

    DEVICE systemdg_02 {
        filename /storage_0512/systemdg_02.img
    }

    DEVICE systemdg_03 {
        filename /storage_0512/systemdg_03.img
    }
}

TARGET_DRIVER iscsi {
    enabled 0

    TARGET iqn.2015-07.org.vmem:oracle651.san.asm.luns {
        enabled 1
        rel_tgt_id 1

        GROUP ubuntu1504 {
            LUN 0 systemdg_01
            LUN 1 systemdg_02
            LUN 2 systemdg_03

            INITIATOR iqn.1988-12.com.oracle:b59b40c12ce

            INITIATOR iqn.1993-08.org.debian:01:359756c6aac

        }
    }
}

[root@oracle651 ~]#

Run orainstRoot.sh on All Nodes

Run the orainstRoot.sh as shown below.

[root@lxcora01 oraInventory]# ./orainstRoot.sh
Changing permissions of /u00/app/12.1.0/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u00/app/12.1.0/oraInventory to oinstall.
The execution of the script is complete.
[root@lxcora01 oraInventory]# ssh lxcora02
root@lxcora02's password:
Last login: Sun Jun 28 22:30:35 2015 from 10.207.39.70
[root@lxcora02 ~]# cd /u00/app/12.1.0/oraInventory
[root@lxcora02 oraInventory]# ./orainstRoot.sh
Changing permissions of /u00/app/12.1.0/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u00/app/12.1.0/oraInventory to oinstall.
The execution of the script is complete.
[root@lxcora02 oraInventory]# ssh lxcora03
root@lxcora03's password:
Last login: Sun Jun 28 22:31:11 2015 from 10.207.39.71
[root@lxcora03 ~]# cd /u00/app/12.1.0/oraInventory
[root@lxcora03 oraInventory]# ./orainstRoot.sh
Changing permissions of /u00/app/12.1.0/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u00/app/12.1.0/oraInventory to oinstall.
The execution of the script is complete.
[root@lxcora03 oraInventory]# ssh lxcora04
The authenticity of host 'lxcora04 (10.207.39.73)' can't be established.
RSA key fingerprint is 26:48:0f:ee:41:65:14:38:56:06:b3:14:a0:91:fc:85.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora04,10.207.39.73' (RSA) to the list of known hosts.
root@lxcora04's password:
Last login: Sun Jun 28 21:03:41 2015
[root@lxcora04 ~]# cd /u00/app/12.1.0/oraInventory
[root@lxcora04 oraInventory]# ./orainstRoot.sh
Changing permissions of /u00/app/12.1.0/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u00/app/12.1.0/oraInventory to oinstall.
The execution of the script is complete.
[root@lxcora04 oraInventory]# ssh lxcora05
The authenticity of host 'lxcora05 (10.207.39.74)' can't be established.
RSA key fingerprint is 26:48:0f:ee:41:65:14:38:56:06:b3:14:a0:91:fc:85.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora05,10.207.39.74' (RSA) to the list of known hosts.
root@lxcora05's password:
Last login: Sun Jun 28 21:06:02 2015
[root@lxcora05 ~]# cd /u00/app/12.1.0/oraInventory
[root@lxcora05 oraInventory]# ./orainstRoot.sh
Changing permissions of /u00/app/12.1.0/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u00/app/12.1.0/oraInventory to oinstall.
The execution of the script is complete.
[root@lxcora05 oraInventory]# ssh lxcora06
The authenticity of host 'lxcora06 (10.207.39.75)' can't be established.
RSA key fingerprint is 26:48:0f:ee:41:65:14:38:56:06:b3:14:a0:91:fc:85.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lxcora06,10.207.39.75' (RSA) to the list of known hosts.
root@lxcora06's password:
Last login: Sun Jun 28 21:06:35 2015
[root@lxcora06 ~]# cd /u00/app/12.1.0/oraInventory
[root@lxcora06 oraInventory]# ./orainstRoot.sh
Changing permissions of /u00/app/12.1.0/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u00/app/12.1.0/oraInventory to oinstall.
The execution of the script is complete.
[root@lxcora06 oraInventory]#

Create the /etc/network/if-up.d/openvswitch-net file

gstanden@u1504:~$ cat /etc/network/if-up.d/openvswitch-net
#!/bin/sh -e
 /home/gstanden/OpenvSwitch/crt_ovs_sw1.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw1.log
 /home/gstanden/OpenvSwitch/crt_ovs_sx1.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sx1.log
 /home/gstanden/OpenvSwitch/crt_ovs_sw2.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw2.log
 /home/gstanden/OpenvSwitch/crt_ovs_sw3.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw3.log
 /home/gstanden/OpenvSwitch/crt_ovs_sw4.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw4.log
 /home/gstanden/OpenvSwitch/crt_ovs_sw5.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw5.log
 /home/gstanden/OpenvSwitch/crt_ovs_sw6.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw6.log
 /home/gstanden/OpenvSwitch/crt_ovs_sw7.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw7.log
 /home/gstanden/OpenvSwitch/crt_ovs_sw8.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw8.log
 /home/gstanden/OpenvSwitch/crt_ovs_sw9.sh 2>&1 > /home/gstanden/OpenvSwitch/crt_ovs_sw9.log
exit 0
gstanden@u1504:~$






















ċ
config-lxcora01
(4k)
Gilbert Standen,
Jul 26, 2015, 12:53 PM
ċ
Gilbert Standen,
Jun 28, 2015, 4:37 PM
ċ
create-scst-ubuntu.tar.gz
(6k)
Gilbert Standen,
Aug 24, 2015, 6:55 PM
ċ
Gilbert Standen,
Jun 27, 2015, 3:52 PM
ċ
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
fwd.mccc.org
(0k)
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
fwd.vmem.org
(0k)
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
Gilbert Standen,
Jun 27, 2015, 3:52 PM
ċ
interfaces
(0k)
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
local
(0k)
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
Gilbert Standen,
Jun 28, 2015, 3:23 PM
ċ
Gilbert Standen,
Jun 28, 2015, 3:24 PM
ċ
Gilbert Standen,
Jun 28, 2015, 3:24 PM
ċ
Gilbert Standen,
Jun 28, 2015, 3:24 PM
ċ
Gilbert Standen,
Jun 28, 2015, 3:24 PM
ċ
Gilbert Standen,
Jun 28, 2015, 3:24 PM
ċ
Gilbert Standen,
Jun 28, 2015, 3:24 PM
ċ
Gilbert Standen,
Jun 28, 2015, 3:24 PM
ċ
Gilbert Standen,
Jun 28, 2015, 3:24 PM
ċ
Gilbert Standen,
Jun 28, 2015, 3:24 PM
ċ
Gilbert Standen,
Jun 28, 2015, 3:24 PM
ċ
Gilbert Standen,
Jun 28, 2015, 3:24 PM
ċ
Gilbert Standen,
Jun 28, 2015, 3:24 PM
ċ
Gilbert Standen,
Jun 28, 2015, 3:24 PM
ċ
lxcora01.log
(64k)
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
named.conf.local
(1k)
Gilbert Standen,
May 16, 2015, 8:00 PM
ċ
named.conf.options
(0k)
Gilbert Standen,
May 16, 2015, 8:01 PM
ċ
Gilbert Standen,
Jun 27, 2015, 3:52 PM
ċ
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
rev.mccc.org
(0k)
Gilbert Standen,
May 16, 2015, 5:49 PM
ċ
rev.vmem.org
(0k)
Gilbert Standen,
May 16, 2015, 5:49 PM
Comments