Oracle EE ASM 12c LXC Ubuntu 14.04

Install Oracle Enterprise Edition 12c ASM on Ubuntu 14.04.1 (Server or Desktop) in 15 minutes ?

Here's how, using LXC (Linux Containers)...

Overview

Installation of Oracle ASM and Oracle RDBMS have always been problematic on Ubuntu native OS. There are fundamental differences between the architecture of Ubuntu and the Red-Hat type Linuxes which require special unsupported techniques. Linux Containers fundamentally change this and make it possible to install Oracle ASM (11g or 12c) and Oracle RDBMS (11g or 12c) on Ubuntu easily and quickly. The installation of Oracle ASM and RDBMS on Linux Containers running on Ubuntu is not officially supported by Oracle afaik. Nevertheless, this guide shows how to deploy Oracle Enterprise Edition ASM and RDBMS products on Ubuntu as a proof-oc-concept (POC) and for personal use. Up until the advent of Linux Containers, the only version of Oracle which had any significant use on Ubuntu Linux was Oracle XE, and even installation of Oracle XE on Ubuntu was not straightforward, both because there is no Oracle-vended *.deb package ("alien" must be used to convert the Oracle-vended Oracle XE *.rpm package to *.deb format) and also because several non-standard tweaks are needed, as described here.

The advent of Linux Containers (in particular for this blog post LXC flavor of Linux Containers) alters the picture significantly and makes deployment of mainline Oracle Enterprise Editions products quick, easy and feasible, albeit officially unsupported by Oracle Corporation afaik, to run Oracle Enterprise Edition products on Linux Containers on Ubuntu.

Modify /etc/sysctl.conf File on Host Server

Modify the /etc/sysctl.conf fle on the host server as shown below. The parts to be added are shown in bold below.

gstanden@vmem1:~$ cat /etc/sysctl.conf

# /etc/sysctl.conf - Configuration file for setting system variables

# See /etc/sysctl.d/ for additional system variables.

# See sysctl.conf (5) for information.

#

#kernel.domainname = example.com

# Uncomment the following to stop low-level messages on console

#kernel.printk = 3 4 1 3

##############################################################3

# Functions previously found in netbase

#

# Uncomment the next two lines to enable Spoof protection (reverse-path filter)

# Turn on Source Address Verification in all interfaces to

# prevent some spoofing attacks

#net.ipv4.conf.default.rp_filter=1

#net.ipv4.conf.all.rp_filter=1

# Uncomment the next line to enable TCP/IP SYN cookies

# See http://lwn.net/Articles/277146/

# Note: This may impact IPv6 TCP sessions too

#net.ipv4.tcp_syncookies=1

# Uncomment the next line to enable packet forwarding for IPv4

#net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPv6

# Enabling this option disables Stateless Address Autoconfiguration

# based on Router Advertisements for this host

#net.ipv6.conf.all.forwarding=1

###################################################################

# Additional settings - these settings can improve the network

# security of the host and prevent against some network attacks

# including spoofing attacks and man in the middle attacks through

# redirection. Some network environments, however, require that these

# settings are disabled so review and enable them as needed.

#

# Do not accept ICMP redirects (prevent MITM attacks)

#net.ipv4.conf.all.accept_redirects = 0

#net.ipv6.conf.all.accept_redirects = 0

# _or_

# Accept ICMP redirects only for gateways listed in our default

# gateway list (enabled by default)

# net.ipv4.conf.all.secure_redirects = 1

#

# Do not send ICMP redirects (we are not a router)

#net.ipv4.conf.all.send_redirects = 0

#

# Do not accept IP source route packets (we are not a router)

#net.ipv4.conf.all.accept_source_route = 0

#net.ipv6.conf.all.accept_source_route = 0

#

# Log Martian Packets

#net.ipv4.conf.all.log_martians = 1

#

# Oracle

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 6815744

fs.aio-max-nr = 1048576

net.ipv4.ip_local_port_range = 9000 65500

net.core.rmem_default = 262144

net.core.rmem_max = 4194304

net.core.wmem_default = 262144

net.core.wmem_max = 1048576

vm.nr_hugepages = 2060

gstanden@vmem1:~

Edit /etc/security/limits.conf File on Host Server

Add the lines for Oracle at the end of the /etc/security/limits.conf file as shown in bold below.

gstanden@vmem1:~$ cat /etc/security/limits.conf

# /etc/security/limits.conf

#

#Each line describes a limit for a user in the form:

#

#<domain> <type> <item> <value>

#

#Where:

#<domain> can be:

# - a user name

# - a group name, with @group syntax

# - the wildcard *, for default entry

# - the wildcard %, can be also used with %group syntax,

# for maxlogin limit

# - NOTE: group and wildcard limits are not applied to root.

# To apply a limit to the root user, <domain> must be

# the literal username root.

#

#<type> can have the two values:

# - "soft" for enforcing the soft limits

# - "hard" for enforcing hard limits

#

#<item> can be one of the following:

# - core - limits the core file size (KB)

# - data - max data size (KB)

# - fsize - maximum filesize (KB)

# - memlock - max locked-in-memory address space (KB)

# - nofile - max number of open files

# - rss - max resident set size (KB)

# - stack - max stack size (KB)

# - cpu - max CPU time (MIN)

# - nproc - max number of processes

# - as - address space limit (KB)

# - maxlogins - max number of logins for this user

# - maxsyslogins - max number of logins on the system

# - priority - the priority to run user process with

# - locks - max number of file locks the user can hold

# - sigpending - max number of pending signals

# - msgqueue - max memory used by POSIX message queues (bytes)

# - nice - max nice priority allowed to raise to values: [-20, 19]

# - rtprio - max realtime priority

# - chroot - change root to directory (Debian-specific)

#

#<domain> <type> <item> <value>

#

#* soft core 0

#root hard core 100000

#* hard rss 10000

#@student hard nproc 20

#@faculty soft nproc 20

#@faculty hard nproc 50

#ftp hard nproc 0

#ftp - chroot /ftp

#@student - maxlogins 4

# Oracle Kernel Parameters

oracle soft nproc 2047

oracle hard nproc 16384

oracle soft nofile 1024

oracle hard nofile 65536

oracle soft stack 10240

oracle hard stack 10240

* soft memlock 9873408

* hard memlock 9873408

gstanden@vmem1:~$

Install Oracle Instant Client on Host Server

Install Oracle Instant Client on host server as described here.

Install LXC

Install LXC as shown below. There are some good howto guides here and here regarding LXC installation on Ubuntu. All that is really required is to install the lxc package in simplest form. Linux Containers LXC package is already installed on my system so your output will differ from that shown below, but the command is the same.

gstanden@vmem1:~$ sudo apt-get install lxc

[sudo] password for gstanden:

Reading package lists... Done

Building dependency tree

Reading state information... Done

lxc is already the newest version.

The following package was automatically installed and is no longer required:

linux-image-extra-3.13.0-32-generic

Use 'apt-get autoremove' to remove it.

0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.

gstanden@vmem1:~$

Create LXC Container for Oracle ASM

Create an Oracle Enterprise Linux 6.5 container as shown below. Note that at the end, information is provided about the just-installed LXC container.

gstanden@vmem1:~$ sudo lxc-create -t oracle -n lxcora10

Host is Ubuntu 14.04

No release specified with -R, defaulting to 6.5

Create configuration file /var/lib/lxc/lxcora10/config

Yum installing release 6.5 for x86_64

.

.

.

( output truncated for brevity )

.

.

.

Complete!

Rebuilding rpm database

Patching container rootfs /var/lib/lxc/lxcora10/rootfs for Oracle Linux 6.5

Configuring container for Oracle Linux 6.5

Added container user:oracle password:oracle

Added container user:root password:root

Container : /var/lib/lxc/lxcora10/rootfs

Config : /var/lib/lxc/lxcora10/config

Network : eth0 (veth) on virbr0

gstanden@vmem1:~$

Gather Information about LXC Container

Information about the running container can be gathered as shown below.

gstanden@vmem1:~$ sudo lxc-info -n lxcora10

Name: lxcora10

State: STOPPED

gstanden@vmem1:~$ sudo lxc-start -n lxcora10

gstanden@vmem1:~$ sudo lxc-info -n lxcora10

gstanden@vmem1:~$ sudo lxc-info -n lxcora10

Name: lxcora10

State: RUNNING

PID: 12383

IP: 10.0.3.110

CPU use: 0.75 seconds

BlkIO use: 15.57 MiB

Memory use: 10.25 MiB

KMem use: 0 bytes

Link: vethUNYH04

TX bytes: 1.42 KiB

RX bytes: 4.53 KiB

Total bytes: 5.96 KiB

gstanden@vmem1:~$

Connect to LXC Container via Console

The LXC container comes with a default console connection which can be used as shown below. Notice LXC container is running the same kernel as the host, literally. Not just the same version. The exact same kernel. All the Linux Containers share the one kernel.

gstanden@vmem1:~$ sudo lxc-console -n lxcora10

Connected to tty 1

Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself

Oracle Linux Server release 6.5

Kernel 3.13.11.6 on an x86_64

lxcora10 login: root

Password:

Last login: Wed Sep 17 16:10:08 on lxc/tty1

[root@lxcora10 ~]# uname -a

Linux lxcora10 3.13.11.6 #1 SMP Mon Sep 15 11:54:55 CDT 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@lxcora10 ~]#

Disconnect from LXC Console

Disconnect from the LXC container console by typing "exit" at the command prompt, then "<ctrl>+a" and then type "q" as shown below.

Oracle Linux Server release 6.5

Kernel 3.13.11.6 on an x86_64

lxcora10 login:

gstanden@vmem1:~$

Connect to LXC Containers vis SSH

The lxc-info command in a previous step provides the IP address. There is no DNS resolution (unless it has been setup separately using bind9 or dnsmasq) so the IP address is used as shown below.

gstanden@vmem1:~$ sudo lxc-info -n lxcora10

Name: lxcora10

State: RUNNING

PID: 12383

IP: 10.0.3.110

CPU use: 0.83 seconds

Memory use: 8.76 MiB

KMem use: 0 bytes

Link: vethUNYH04

TX bytes: 1.42 KiB

RX bytes: 5.16 KiB

Total bytes: 6.58 KiB

gstanden@vmem1:~$ ssh root@10.0.3.110

The authenticity of host '10.0.3.110 (10.0.3.110)' can't be established.

RSA key fingerprint is fc:4e:2d:4a:54:f5:f2:11:7f:d8:53:6c:a5:e5:1b:f5.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '10.0.3.110' (RSA) to the list of known hosts.

root@10.0.3.110's password:

Last login: Wed Sep 17 16:11:28 2014

[root@lxcora10 ~]# uname -a

Linux lxcora10 3.13.11.6 #1 SMP Mon Sep 15 11:54:55 CDT 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@lxcora10 ~]# hostname

lxcora10

[root@lxcora10 ~]#

Add LXC Container to /etc/hosts File

Add the LXC container to the /etc/hosts file as shown below.

gstanden@vmem1:~$ cat /etc/hosts

127.0.0.1 localhost

127.0.1.1 vmem1

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback

fe00::0 ip6-localnet

ff00::0 ip6-mcastprefix

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters

10.0.3.110 lxcora10.vmem.org lxcora10

gstanden@vmem1:~$

Create required Users and Groups for Oracle Install

Create the required users and groups for the Oracle ASM install as shown below. Note, the "oracle" LXC template comes with and "oracle" user installed. It has UID 500. Recommend not to change this and to just modify it as shown below for use. Changing the UID can lead to odd issues with "su -" and "ssh" in some cases.

gstanden@vmem1:~$ ssh root@lxcora10

root@lxcora10's password:

Last login: Thu Sep 18 00:57:04 2014 from 10.0.3.1

[root@lxcora10 ~]# vi create_users.sh

[root@lxcora10 ~]# chmod +x create_users.sh

[root@lxcora10 ~]# cat create_users.sh

groupadd -g 1000 oinstall

groupadd -g 1100 asmadmin

groupadd -g 1200 dba

groupadd -g 1300 asmdba

groupadd -g 1201 oper

groupadd -g 1301 asmoper

useradd -u 1098 -g oinstall -G asmadmin,asmdba,asmoper,dba grid

usermod -a -G dba,asmdba,oper,oinstall oracle

mkdir -p /u00/app/12.1.0/grid

mkdir -p /u00/app/grid

chown -R grid:oinstall /u00

mkdir -p /u00/app/oracle

chown oracle:oinstall /u00/app/oracle

chmod -R 775 /u00

[root@lxcora10 ~]# ./create_users.sh

[root@lxcora10 ~]#

Set Passwords and Test su for "grid" and "oracle" Linux Users

Set these as shown below.

[root@lxcora01 ~]# passwd oracle

Changing password for user oracle.

New password:

Retype new password:

passwd: all authentication tokens updated successfully.

[root@lxcora01 ~]# passwd grid

Changing password for user grid.

New password:

Retype new password:

passwd: all authentication tokens updated successfully.

[root@lxcora01 ~]# su - oracle

[oracle@lxcora01 ~]$ exit

logout

[root@lxcora01 ~]# su - grid

[grid@lxcora01 ~]$ exit

logout

[root@lxcora01 ~]#

Set "umask" for "grid" Linux User

Set umask for "grid" as shown below.

[root@lxcora01 ~]# cat .bashrc

# .bashrc

# User specific aliases and functions

alias rm='rm -i'

alias cp='cp -i'

alias mv='mv -i'

# Source global definitions

if [ -f /etc/bashrc ]; then

. /etc/bashrc

fi

umask 022

[root@lxcora01 ~]#

Set "umask" for "oracle" Linux User

Set "umask" for "oracle" as shown below.

[root@lxcora01 ~]# su - oracle

[oracle@lxcora01 ~]$ vi .bashrc

[oracle@lxcora01 ~]$ cat .bashrc

# .bashrc

# Source global definitions

if [ -f /etc/bashrc ]; then

. /etc/bashrc

fi

# User specific aliases and functions

umask 022

[oracle@lxcora01 ~]$

Install Required Packages for Oracle Install

root@vmem1:/var/lib/lxc/lxcora9/rootfs/root# more packages.sh

yum -y install binutils

yum -y install compat-libcap1

yum -y install compat-libstdc++-33

yum -y install compat-libstdc++-33.i686

yum -y install gcc

yum -y install gcc-c++

yum -y install glibc

yum -y install glibc.i686

yum -y install glibc-devel

yum -y install glibc-devel.i686

yum -y install ksh

yum -y install libgcc

yum -y install libgcc.i686

yum -y install libstdc++

yum -y install libstdc++.i686

yum -y install libstdc++-devel

yum -y install libstdc++-devel.i686

yum -y install libaio

yum -y install libaio.i686

yum -y install libaio-devel

yum -y install libaio-devel.i686

yum -y install libXext

yum -y install libXext.i686

yum -y install libXtst

yum -y install libXtst.i686

yum -y install libX11

yum -y install libX11.i686

yum -y install libXau

yum -y install libXau.i686

yum -y install libxcb

yum -y install libxcb.i686

yum -y install libXi

yum -y install libXi.i686

yum -y install make

yum -y install sysstat

yum -y install unixODBC

yum -y install unixODBC-devel

yum -y install xdpyinfo

yum -y install xorg-x11-apps

yum -y install pdksh

yum -y install libicu

yum -y remove elfutils-libelf-devel.i386

yum -y install ntp

yum -y install sg3-utils

yum -y install xauth

yum -y install xorg-x11-fonts*

yum -y install unzip

yum -y install nfs-utils

chkconfig sendmail off

root@vmem1:/var/lib/lxc/lxcora9/rootfs/root#

Chkconfig "sendmail" off

Sendmail is not needed and will cause LXC container to boot slowly. Disable it for boot as shown below.

root@lxcora10 # chkconfig sendmail off

Install the /etc/sysctl.conf File Required for Oracle

Install the following /etc/sysctl.conf file into the /etc/sysctl.conf file of container as required by Oracle as shown below. Add the bold section everything below "# Oracle" to the end of the /etc/sysctl.conf file as shown below.

# Kernel sysctl configuration file for Red Hat Linux

#

# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and

# sysctl.conf(5) for more details.

# Controls IP packet forwarding

net.ipv4.ip_forward = 0

# Controls source route verification

net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing

net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel

kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.

# Useful for debugging multi-threaded applications.

kernel.core_uses_pid = 1

# Controls the use of TCP syncookies

net.ipv4.tcp_syncookies = 1

# Disable netfilter on bridges.

net.bridge.bridge-nf-call-ip6tables = 0

net.bridge.bridge-nf-call-iptables = 0

net.bridge.bridge-nf-call-arptables = 0

# Controls the default maxmimum size of a mesage queue

kernel.msgmnb = 65536

# Controls the maximum size of a message, in bytes

kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes

kernel.shmmax = 5368709120

# Controls the maximum number of shared memory segments, in pages

kernel.shmall = 4294967296

# Oracle

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 6815744

fs.aio-max-nr = 1048576

net.ipv4.ip_local_port_range = 9000 65500

net.core.rmem_default = 262144

net.core.rmem_max = 4194304

net.core.wmem_default = 262144

net.core.wmem_max = 1048576

Edit the Container Configuration File

The container configuration file needs to be edited as shown below to support addition of block devices and other features needed for Oracle ASM. The last 5 lines at the end (in bold) are the ones that need to be added to the default configuration file for the LXC container.

root@vmem1:/var/lib/lxc/lxcora10# pwd

/var/lib/lxc/lxcora10

root@vmem1:/var/lib/lxc/lxcora10# vi config

root@vmem1:/var/lib/lxc/lxcora10# cat config

# Template used to create this container: /usr/share/lxc/templates/lxc-oracle

# Parameters passed to the template:

# For additional config options, please look at lxc.container.conf(5)

lxc.network.type = veth

lxc.network.flags = up

lxc.network.link = lxcbr0

lxc.network.hwaddr = 00:16:3e:fd:a0:55

lxc.rootfs = /var/lib/lxc/lxcora10/rootfs

# Common configuration

lxc.include = /usr/share/lxc/config/oracle.common.conf

# Container configuration for Oracle Linux 6.5

lxc.arch = x86_64

lxc.utsname = lxcora10

lxc.cap.drop = sys_resource

lxc.cap.drop = setfcap setpcap

# Networking

lxc.network.name = eth0

lxc.network.mtu = 1500

lxc.network.hwaddr = fe:7e:9e:e5:d6:7c

lxc.cgroup.devices.allow = c 10:236 rwm

lxc.cgroup.devices.allow = b 252:* rwm

lxc.mount.entry = /dev/mapper /var/lib/lxc/lxcora10/rootfs/dev/mapper none defaults,bind,create=dir 0 0

lxc.mount.auto = proc sys cgroup

lxc.mount.auto = proc:rw sys:rw cgroup-full:rw

root@vmem1:/var/lib/lxc/lxcora10#

Install device-mapper-multipath on Host Server

Device-mapper-multipath is used to provide naming services in /dev/mapper for the LUN partitions from a USB stick or an Expresscard. It is also possible to make some small partitions at the end of the Ubuntu partition, but this is more useful if your Ubuntu install used UEFI which means that an extended logical partition will not be necessary. Simplest is probably to use a USB stick or expresscard.

Prepare Storage for ASM Diskgroups

There are various ways to do this. For this example a Wintec Filemate 128 Gb all-flash expresscard is used. It has been partitioned into 17 x 2Gb GPT partitions as shown below. Recommend against using loopback devices. A USB stick should work also for "physical storage". Partition the USB stick into "LUNs" and then get the required information to add it to /etc/multipath.conf.

gstanden@vmem1:~$ ls -lrt /dev/mapper/asm*

brw-rw---- 1 root disk 252, 6 Sep 18 17:55 /dev/mapper/asm_disk

brw-rw---- 1 root disk 252, 9 Sep 18 17:55 /dev/mapper/asm_disk6

brw-rw---- 1 root disk 252, 8 Sep 18 17:55 /dev/mapper/asm_disk5

brw-rw---- 1 root disk 252, 7 Sep 18 17:55 /dev/mapper/asm_disk4

brw-rw---- 1 root disk 252, 5 Sep 18 17:55 /dev/mapper/asm_disk3

brw-rw---- 1 root disk 252, 2 Sep 18 17:55 /dev/mapper/asm_disk2

brw-rw---- 1 root disk 252, 1 Sep 18 17:55 /dev/mapper/asm_disk1

brw-rw---- 1 root disk 252, 12 Sep 18 17:55 /dev/mapper/asm_disk9

brw-rw---- 1 root disk 252, 11 Sep 18 17:55 /dev/mapper/asm_disk8

brw-rw---- 1 root disk 252, 10 Sep 18 17:55 /dev/mapper/asm_disk7

brw-rw---- 1 root disk 252, 16 Sep 18 17:55 /dev/mapper/asm_disk13

brw-rw---- 1 root disk 252, 15 Sep 18 17:55 /dev/mapper/asm_disk12

brw-rw---- 1 root disk 252, 14 Sep 18 17:55 /dev/mapper/asm_disk11

brw-rw---- 1 root disk 252, 13 Sep 18 17:55 /dev/mapper/asm_disk10

brw-rw---- 1 root disk 252, 20 Sep 18 17:55 /dev/mapper/asm_disk17

brw-rw---- 1 root disk 252, 19 Sep 18 17:55 /dev/mapper/asm_disk16

brw-rw---- 1 root disk 252, 18 Sep 18 17:55 /dev/mapper/asm_disk15

brw-rw---- 1 root disk 252, 17 Sep 18 17:55 /dev/mapper/asm_disk14

gstanden@vmem1:~$ sudo gdisk -l /dev/sdb

[sudo] password for gstanden:

GPT fdisk (gdisk) version 0.8.8

Partition table scan:

MBR: protective

BSD: not present

APM: not present

GPT: present

Found valid GPT with protective MBR; using GPT.

Disk /dev/sdb: 250069680 sectors, 119.2 GiB

Logical sector size: 512 bytes

Disk identifier (GUID): 9A4E5E70-D053-49C1-8217-4CECC833529F

Partition table holds up to 128 entries

First usable sector is 34, last usable sector is 250069646

Partitions will be aligned on 2048-sector boundaries

Total free space is 178766445 sectors (85.2 GiB)

Number Start (sector) End (sector) Size Code Name

1 2048 4196351 2.0 GiB 8300 Linux filesystem

2 4196352 8390655 2.0 GiB 8300 Linux filesystem

3 8390656 12584959 2.0 GiB 8300 Linux filesystem

4 12584960 16779263 2.0 GiB 8300 Linux filesystem

5 16779264 20973567 2.0 GiB 8300 Linux filesystem

6 20973568 25167871 2.0 GiB 8300 Linux filesystem

7 25167872 29362175 2.0 GiB 8300 Linux filesystem

8 29362176 33556479 2.0 GiB 8300 Linux filesystem

9 33556480 37750783 2.0 GiB 8300 Linux filesystem

10 37750784 41945087 2.0 GiB 8300 Linux filesystem

11 41945088 46139391 2.0 GiB 8300 Linux filesystem

12 46139392 50333695 2.0 GiB 8300 Linux filesystem

13 50333696 54527999 2.0 GiB 8300 Linux filesystem

14 54528000 58722303 2.0 GiB 8300 Linux filesystem

15 58722304 62916607 2.0 GiB 8300 Linux filesystem

16 62916608 67110911 2.0 GiB 8300 Linux filesystem

17 67110912 71305215 2.0 GiB 8300 Linux filesystem

gstanden@vmem1:~$

Add Command to /etc/rc.local to Set Ownership of ASM Block Device

[root@lxcora01 ~]# cat /etc/rc.local

#!/bin/sh

#

# This script will be executed *after* all the other init scripts.

# You can put your own initialization stuff in here if you don't

# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

chown grid:asmadmin /dev/mapper/asm_disk1

chmod 0660 /dev/mapper/asm_disk1

[root@lxcora01 ~]#

Restart the LXC Container

Restart the container as shown below so that the changes in the container configuration take effect.

root@vmem1:/var/lib/lxc/lxcora10# ssh root@lxcora10

The authenticity of host 'lxcora10 (10.0.3.110)' can't be established.

RSA key fingerprint is fc:4e:2d:4a:54:f5:f2:11:7f:d8:53:6c:a5:e5:1b:f5.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'lxcora10,10.0.3.110' (RSA) to the list of known hosts.

root@lxcora10's password:

Last login: Thu Sep 18 01:02:27 2014 from 10.0.3.1

[root@lxcora10 ~]# reboot

Broadcast message from root@lxcora10

(/dev/pts/1) at 1:17 ...

The system is going down for reboot NOW!

[root@lxcora10 ~]# Connection to lxcora10 closed by remote host.

Connection to lxcora10 closed.

root@vmem1:/var/lib/lxc/lxcora10#

Run the Command to Verify Oracle Kernel Parameter Settings

After reboot, /proc/sys is mounted RW so that settings in /etc/sysctl.conf can be read and applied. Use the "sysctl -a" command to verify that /proc/sys is actually rw and to verify the correct values as shown below. Note that required settings for Oracle ASM EE install are successfully applied. Not all parameters will configure, but ensure that output is similar to below.

[root@lxcora01 ~]# sysctl -p

net.ipv4.ip_forward = 0

net.ipv4.conf.default.rp_filter = 1

net.ipv4.conf.default.accept_source_route = 0

error: permission denied on key 'kernel.sysrq'

error: permission denied on key 'kernel.core_uses_pid'

error: "net.ipv4.tcp_syncookies" is an unknown key

error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key

error: "net.bridge.bridge-nf-call-iptables" is an unknown key

error: "net.bridge.bridge-nf-call-arptables" is an unknown key

kernel.msgmnb = 65536

kernel.msgmax = 65536

kernel.shmmax = 68719476736

kernel.shmall = 4294967296

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

error: permission denied on key 'fs.file-max'

error: permission denied on key 'fs.aio-max-nr'

net.ipv4.ip_local_port_range = 9000 65500

error: "net.core.rmem_default" is an unknown key

error: "net.core.rmem_max" is an unknown key

error: "net.core.wmem_default" is an unknown key

error: "net.core.wmem_max" is an unknown key

[root@lxcora01 ~]#

[root@lxcora01 ~]# sysctl -a | egrep 'shmmni|file-max|aio-max-nr|shmmax|shmall|sem|local_port_range'

fs.aio-max-nr = 1048576

fs.file-max = 6815744

kernel.sem = 250 32000 100 128

kernel.sem_next_id = -1

kernel.shmall = 4294967296

kernel.shmmax = 68719476736

kernel.shmmni = 4096

net.ipv4.ip_local_port_range = 9000 65500

[root@lxcora01 ~]#

Stage Oracle 12c EE ASM Installation Media

Stage the installation media for Oracle Grid Infrastructure install in /home/grid as shown below.

gstanden@vmem1:~$ sudo su -

root@vmem1:~# cd /var/lib/lxc/lxcora9

root@vmem1:/var/lib/lxc/lxcora9# cd rootfs/home/grid

root@vmem1:/var/lib/lxc/lxcora9/rootfs/home/grid# ls -lrt

total 2337900

drwxr-xr-x 7 1099 gstanden 4096 Jul 11 05:36 grid

-rwxrwxrwx 1 1099 gstanden 1747021093 Aug 10 17:23 p17694377_121020_Linux-x86-64_3of8.zip

-rwxrwxrwx 1 1099 gstanden 646969279 Aug 10 17:35 p17694377_121020_Linux-x86-64_4of8.zip

drwxr-xr-x 2 1099 gstanden 4096 Sep 16 22:57 oraInventory

root@vmem1:/var/lib/lxc/lxcora9/rootfs/home/grid# scp p*.zip ../../../../lxcora10/rootfs/home/grid/.

root@vmem1:/var/lib/lxc/lxcora9/rootfs/home/grid#

Set Permissions on Install Media

As root user, login to the LXC container and set the permissions on the install zip archives to "grid:oinstall" as shown below.

gstanden@vmem1:~$ ssh root@lxcora10

root@lxcora10's password:

Last login: Thu Sep 18 01:29:19 2014 from 10.0.3.1

[root@lxcora10 ~]# cd /home/grid

[root@lxcora10 grid]# ls -lrt

total 2337892

-rwxr-xr-x 1 root root 1747021093 Sep 18 01:34 p17694377_121020_Linux-x86-64_3of8.zip

-rwxr-xr-x 1 root root 646969279 Sep 18 01:34 p17694377_121020_Linux-x86-64_4of8.zip

[root@lxcora10 grid]# chown grid:oinstall p*.zip

[root@lxcora10 grid]# ls -lrt

total 2337892

-rwxr-xr-x 1 grid oinstall 1747021093 Sep 18 01:34 p17694377_121020_Linux-x86-64_3of8.zip

-rwxr-xr-x 1 grid oinstall 646969279 Sep 18 01:34 p17694377_121020_Linux-x86-64_4of8.zip

[root@lxcora10 grid]#

Unzip Install Media

Login as the "grid" user and unzip the install media and begin the install as shown below.

gstanden@vmem1:~$ ssh -Y -C grid@lxcora10

grid@lxcora10's password:

/usr/bin/xauth: creating new authority file /home/grid/.Xauthority

[grid@lxcora10 ~]$ xclock

[grid@lxcora10 ~]$ cd grid

-bash: cd: grid: No such file or directory

[grid@lxcora10 ~]$ ls -lrt

total 2337892

-rwxr-xr-x 1 grid oinstall 1747021093 Sep 18 01:34 p17694377_121020_Linux-x86-64_3of8.zip

-rwxr-xr-x 1 grid oinstall 646969279 Sep 18 01:34 p17694377_121020_Linux-x86-64_4of8.zip

[grid@lxcora10 ~]$ id

uid=1099(grid) gid=1000(oinstall) groups=1000(oinstall),1100(asmadmin),1200(dba),1300(asmdba),1301(asmoper)

[grid@lxcora10 ~]$ unzip p17694377_121020_Linux-x86-64_3of8.zip

[grid@lxcora10 ~]$ unzip p17694377_121020_Linux-x86-64_4of8.zip

Install the Oracle "cvuqdisk" rpm Package

Install the package as shown below while connected as the "root" user.

[root@lxcora01 ~]# cd /home/grid/grid/rpm

[root@lxcora01 rpm]# rpm -Uvh cvuqdisk-1.0.9-1.rpm

Preparing... ########################################### [100%]

Using default group oinstall to install package

1:cvuqdisk ########################################### [100%]

[root@lxcora01 rpm]#

Setup Public Key and "no-password" SSH for "grid" Linux user

Set this up as shown below.

[grid@lxcora01 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/grid/.ssh/id_rsa):

Created directory '/home/grid/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/grid/.ssh/id_rsa.

Your public key has been saved in /home/grid/.ssh/id_rsa.pub.

The key fingerprint is:

3e:88:42:64:fa:12:51:5f:58:89:7e:42:a3:16:64:7c grid@lxcora01

The key's randomart image is:

+--[ RSA 2048]----+

| o+ +o. |

| oo.E.. |

|. o*.. |

| =o o . |

|o.. o S |

| + . o |

|. o . . o |

| . . . |

| |

+-----------------+

[grid@lxcora01 ~]$ cd .ssh

[grid@lxcora01 .ssh]$ cp -p id_rsa.pub authorized_keys

[grid@lxcora01 .ssh]$ cd

[grid@lxcora01 ~]$ ssh lxcora01

The authenticity of host 'lxcora01 (127.0.0.1)' can't be established.

RSA key fingerprint is 55:bd:d9:85:c1:31:0d:8a:2b:76:e8:dd:f9:0a:55:02.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'lxcora01' (RSA) to the list of known hosts.

[grid@lxcora01 ~]$ exit

logout

Connection to lxcora01 closed.

[grid@lxcora01 ~]$ ssh localhost

The authenticity of host 'localhost (127.0.0.1)' can't be established.

RSA key fingerprint is 55:bd:d9:85:c1:31:0d:8a:2b:76:e8:dd:f9:0a:55:02.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'localhost' (RSA) to the list of known hosts.

Last login: Thu Sep 18 20:09:47 2014 from localhost

[grid@lxcora01 ~]$ exit

logout

Connection to localhost closed.

[grid@lxcora01 ~]$ ssh 127.0.0.1

The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.

RSA key fingerprint is 55:bd:d9:85:c1:31:0d:8a:2b:76:e8:dd:f9:0a:55:02.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '127.0.0.1' (RSA) to the list of known hosts.

Last login: Thu Sep 18 20:09:54 2014 from localhost

[grid@lxcora01 ~]$ exit

logout

Connection to 127.0.0.1 closed.

[grid@lxcora01 ~]$ ssh 10.0.3.144

The authenticity of host '10.0.3.144 (10.0.3.144)' can't be established.

RSA key fingerprint is 55:bd:d9:85:c1:31:0d:8a:2b:76:e8:dd:f9:0a:55:02.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '10.0.3.144' (RSA) to the list of known hosts.

Last login: Thu Sep 18 20:10:02 2014 from localhost

[grid@lxcora01 ~]$

Run "cluvfy" the Oracle Cluster Verification Utility

Run the "runcluvfy.sh" script as shown below. The output should be similar to that shown below.

[grid@lxcora01 grid]$ ./runcluvfy.sh stage -pre crsinst -n lxcora01 -r 12.1 -osdba oinstall -asm -asmgrp asmadmin -asmdev /dev/mapper/asm_disk1

Performing pre-checks for cluster services setup

Checking node reachability...

Node reachability check passed from node "lxcora01"

Checking user equivalence...

User equivalence check passed for user "grid"

Package existence check passed for "cvuqdisk"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Node connectivity passed for subnet "10.0.3.0" with node(s) lxcora01

TCP connectivity check passed for subnet "10.0.3.0"

Interfaces found on subnet "10.0.3.0" that are likely candidates for VIP are:

lxcora01 eth0:10.0.3.96

WARNING:

Could not find a suitable set of interfaces for the private interconnect

Node connectivity check passed

Checking multicast communication...

Checking subnet "10.0.3.0" for multicast communication with multicast group "224.0.0.251"...

Check of subnet "10.0.3.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.

Total memory check passed

Available memory check passed

Swap space check passed

Free disk space check passed for "lxcora01:/usr,lxcora01:/var,lxcora01:/etc,lxcora01:/sbin,lxcora01:/tmp"

Check for multiple users with UID value 1100 passed

User existence check passed for "grid"

Group existence check passed for "oinstall"

Group existence check passed for "asmadmin"

Membership check for user "grid" in group "oinstall" [as Primary] passed

Membership check for user "grid" in group "asmadmin" passed

Run level check passed

Hard limits check failed for "maximum open file descriptors"

Check failed on nodes:

lxcora01

Soft limits check passed for "maximum open file descriptors"

Hard limits check passed for "maximum user processes"

Soft limits check failed for "maximum user processes"

Check failed on nodes:

lxcora01

System architecture check passed

Kernel version check passed

Kernel parameter check passed for "semmsl"

Kernel parameter check passed for "semmns"

Kernel parameter check passed for "semopm"

Kernel parameter check passed for "semmni"

Kernel parameter check passed for "shmmax"

Kernel parameter check passed for "shmmni"

Kernel parameter check passed for "shmall"

Kernel parameter check passed for "file-max"

Kernel parameter check passed for "ip_local_port_range"

PRVF-7544 : Check cannot be performed for kernel parameter "rmem_default" on node "lxcora01"

PRVG-2044 : Command "/sbin/sysctl -a | grep rmem_default[[:space:]]*=" failed on node "lxcora01" and produced no output.

Kernel parameter check failed for "rmem_default"

Check failed on nodes:

lxcora01

PRVF-7544 : Check cannot be performed for kernel parameter "rmem_max" on node "lxcora01"

PRVG-2044 : Command "/sbin/sysctl -a | grep rmem_max[[:space:]]*=" failed on node "lxcora01" and produced no output.

Kernel parameter check failed for "rmem_max"

Check failed on nodes:

lxcora01

PRVF-7544 : Check cannot be performed for kernel parameter "wmem_default" on node "lxcora01"

PRVG-2044 : Command "/sbin/sysctl -a | grep wmem_default[[:space:]]*=" failed on node "lxcora01" and produced no output.

Kernel parameter check failed for "wmem_default"

Check failed on nodes:

lxcora01

PRVF-7544 : Check cannot be performed for kernel parameter "wmem_max" on node "lxcora01"

PRVG-2044 : Command "/sbin/sysctl -a | grep wmem_max[[:space:]]*=" failed on node "lxcora01" and produced no output.

Kernel parameter check failed for "wmem_max"

Check failed on nodes:

lxcora01

Kernel parameter check passed for "aio-max-nr"

PRVG-1206 : Check cannot be performed for configured value of kernel parameter "panic_on_oops" on node "lxcora01"

Kernel parameter check failed for "panic_on_oops"

Check failed on nodes:

lxcora01

Package existence check passed for "binutils"

Package existence check passed for "compat-libcap1"

Package existence check passed for "compat-libstdc++-33(x86_64)"

Package existence check passed for "libgcc(x86_64)"

Package existence check passed for "libstdc++(x86_64)"

Package existence check passed for "libstdc++-devel(x86_64)"

Package existence check passed for "sysstat"

Package existence check passed for "gcc"

Package existence check passed for "gcc-c++"

Package existence check passed for "ksh"

Package existence check passed for "make"

Package existence check passed for "glibc(x86_64)"

Package existence check passed for "glibc-devel(x86_64)"

Package existence check passed for "libaio(x86_64)"

Package existence check passed for "libaio-devel(x86_64)"

Package existence check passed for "nfs-utils"

Checking availability of ports "6200,6100" required for component "Oracle Notification Service (ONS)"

Port availability check passed for ports "6200,6100"

Checking availability of ports "42424" required for component "Oracle Cluster Synchronization Services (CSSD)"

Port availability check passed for ports "42424"

Check for multiple users with UID value 0 passed

Current group ID check passed

Starting check for consistency of primary group of root user

Check for consistency of root user's primary group passed

Checking Devices for ASM...

Package existence check passed for "cvuqdisk"

Checking for shared devices...

PRVG-5150 : could not determine if path /dev/mapper/asm_disk3 is a valid path on all nodes

PRVG-5150 : could not determine if path /dev/mapper/asm_disk2 is a valid path on all nodes

PRVG-5150 : could not determine if path /dev/mapper/asm_disk1 is a valid path on all nodes

Checking consistency of device owner across all nodes...

Consistency check of device owner for "/dev/mapper/asm_disk3" PASSED

Consistency check of device owner for "/dev/mapper/asm_disk2" PASSED

Consistency check of device owner for "/dev/mapper/asm_disk1" PASSED

Checking consistency of device group across all nodes...

Consistency check of device group for "/dev/mapper/asm_disk3" PASSED

Consistency check of device group for "/dev/mapper/asm_disk2" PASSED

Consistency check of device group for "/dev/mapper/asm_disk1" PASSED

Checking consistency of device permissions across all nodes...

Consistency check of device permissions for "/dev/mapper/asm_disk3" PASSED

Consistency check of device permissions for "/dev/mapper/asm_disk2" PASSED

Consistency check of device permissions for "/dev/mapper/asm_disk1" PASSED

Checking consistency of device size across all nodes...

Consistency check of device size for "/dev/mapper/asm_disk3" PASSED

Consistency check of device size for "/dev/mapper/asm_disk2" PASSED

Consistency check of device size for "/dev/mapper/asm_disk1" PASSED

Checking that ASM devices are ASM Filter Driver capable.

ASM Filter Driver compatibility check passed.

Devices check for ASM failed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP configuration file "/etc/ntp.conf" existence check passed

Liveness check passed for "ntpd"

Check for NTP daemon or service alive passed on all nodes

PRVF-5410 : Check of common NTP Time Server failed

PRVF-5416 : Query of NTP daemon failed on all nodes

Clock synchronization check using Network Time Protocol(NTP) failed

Core file name pattern consistency check passed.

User "grid" is not part of "root" group. Check passed

Default user file creation mask check passed

Checking integrity of file "/etc/resolv.conf" across nodes

"domain" and "search" entries do not coexist in any "/etc/resolv.conf" file

The DNS response time for an unreachable node is within acceptable limit on all nodes

Check for integrity of file "/etc/resolv.conf" failed

Time zone consistency check passed

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...

Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed

Checking daemon "avahi-daemon" is not configured and running

Daemon not configured check passed for process "avahi-daemon"

Daemon not running check passed for process "avahi-daemon"

Starting check for /dev/shm mounted as temporary file system ...

ERROR:

PRVE-0420 : /dev/shm is not found mounted on node ""

PRVE-0420 : /dev/shm is not found mounted on node ""

Check for /dev/shm mounted as temporary file system failed

Starting check for /boot mount ...

ERROR:

PRVE-10073 : Required /boot data is not available on node "lxcora01"

PRVE-10073 : Required /boot data is not available on node "lxcora01"

Check for /boot mount failed

Starting check for zeroconf check ...

ERROR:

PRVE-10077 : NOZEROCONF parameter was not specified or was not set to 'yes' in file "/etc/sysconfig/network" on node "lxcora01"

Check for zeroconf check failed

Pre-check for cluster services setup was unsuccessful on all the nodes.

[grid@lxcora01 grid]$

Install Oracle Enterprise Edition ASM 12c

Install Oracle EE ASM 12c as shown below using the GUI installer.

gstanden@vmem1:~$ ssh -Y -C grid@lxcora10

grid@lxcora10's password:

[grid@lxcora10 ~]$[grid@lxcora10 grid]$ ./runInstaller

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 415 MB. Actual 432514 MB Passed

Checking swap space: must be greater than 150 MB. Actual 32839 MB Passed

Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-09-18_02-17-10AM. Please wait ...[grid@lxcora10 grid]$

Login as Linux User "grid" and Test "xclock"

gstanden@vmem1:~$ ssh -Y -C grid@lxcora10

grid@lxcora10's password:

Last login: Thu Sep 18 02:14:52 2014 from 10.0.3.1

[grid@lxcora10 ~]$ xclock

Execute Oracle GUI runInstaller

[grid@lxcora10 ~]$ ls -lrt

total 2337900

drwxr-xr-x 7 grid oinstall 4096 Jul 11 06:36 grid

-rwxr-xr-x 1 grid oinstall 1747021093 Sep 18 01:34 p17694377_121020_Linux-x86-64_3of8.zip

-rwxr-xr-x 1 grid oinstall 646969279 Sep 18 01:34 p17694377_121020_Linux-x86-64_4of8.zip

drwxr-xr-x 2 grid oinstall 4096 Sep 18 02:17 oraInventory

[grid@lxcora10 ~]$ cd grid

[grid@lxcora10 grid]$ ./runInstaller

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 415 MB. Actual 428735 MB Passed

Checking swap space: must be greater than 150 MB. Actual 32843 MB Passed

Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-09-18_09-24-52AM. Please wait ...[grid@lxcora10 grid]$

Use the "Browse..." button to change the location of "Oracle base" as shown below.

Do NOT check box to "Automatically run configuration scripts". They will be run manually in a terminal window as root during install.

Ignore the warnings shown above. The list of warnings should only be those shown above. If there are any additional warnings, they should be corrected and then use "Check Again" to recheck. As long as only the above warnings, are shown, go ahead and check "Ignore All" and then "Next".

Answer "Yes" to the warning dialog box about ignoring some of the prerequisites as shown above.

Optionally save the "response file" which is a file which can be used later for non-GUI installations (e.g. "silent" installations). Review the install and click "Install" to begin the installation. Monitor the progress of the installation windows as shown below.

At the "Setup Oracle Base" step the "Execute Configuration Scripts" window opens.

Click on the "Script Location" of Script Number 1 to highlight it and use <ctrl>+c to copy it to the clipboard (or alternatively just read the script from the window). Open a new terminal window into lxcora10 LXC container and run the scripts as listed, first Script Number 1, and then Script Number 2. They MUST be run sequentially, and not in parallel. The Script Number 1 must complete successfully, and then the Script Number 2 can be run, and it also must complete successfully BEFORE proceeding with the GUI install.

Successful Output of Configuration Script 1

Be sure to run as the "root" user while logged on to the LXC container as shown below.

[root@lxcora10 ~]# /u00/app/oraInventory/orainstRoot.sh

Changing permissions of /u00/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u00/app/oraInventory to oinstall.

The execution of the script is complete.

[root@lxcora10 ~]#

Successful Output of Configuration Script 2

[root@lxcora10 ~]# /u00/app/grid/product/12.1.0/grid/root.sh

Performing root user operation.

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u00/app/grid/product/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Copying dbhome to /usr/local/bin ...

Copying oraenv to /usr/local/bin ...

Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u00/app/grid/product/12.1.0/grid/crs/install/crsconfig_params

LOCAL ADD MODE

Creating OCR keys for user 'grid', privgrp 'oinstall'..

Operation successful.

LOCAL ONLY MODE

Successfully accumulated necessary OCR keys.

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

CRS-4664: Node lxcora10 successfully pinned.

2014/09/18 10:08:23 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

lxcora10 2014/09/18 10:08:56 /u00/app/grid/product/12.1.0/grid/cdata/lxcora10/backup_20140918_100856.olr 0

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'lxcora10'

CRS-2673: Attempting to stop 'ora.evmd' on 'lxcora10'

CRS-2677: Stop of 'ora.evmd' on 'lxcora10' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'lxcora10' has completed

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

2014/09/18 10:09:13 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

[root@lxcora10 ~]#

Click on "Ok" on the "Execute Configuration Scripts" window upon successful completion of both scripts to continue the install as shown below. The Oracle Configuration Assistant programs run as shown below. All Configuration Assistants should complete with a status of "Succeeded" with a green checkmark next to all of them. If any fail, attempt to diagnose and fix issue, and then click "Retry" button.

If all the Configuration Assistants are successful, the "Oracle Cluster Verification Utility" step will flash by almost instantaneously and the "Successful Install" screen will be displayed as shown below.

Check that Expected Grid Infrastructure Processes Are Running

Check the running "grid" linux user processes as shown below. The output should be similar to that shown below.

[root@lxcora10 ~]# ps -ef | grep grid

root 440 295 0 09:21 ? 00:00:00 sshd: grid [priv]

grid 442 440 0 09:21 ? 00:00:24 sshd: grid@pts/0

grid 443 442 0 09:21 pts/0 00:00:00 -bash

grid 4895 1 1 10:08 ? 00:00:05 /u00/app/grid/product/12.1.0/grid/bin/ohasd.bin reboot

grid 4965 1 0 10:09 ? 00:00:04 /u00/app/grid/product/12.1.0/grid/bin/oraagent.bin

grid 4977 1 0 10:09 ? 00:00:02 /u00/app/grid/product/12.1.0/grid/bin/evmd.bin

grid 4994 4977 0 10:09 ? 00:00:03 /u00/app/grid/product/12.1.0/grid/bin/evmlogger.bin -o /u00/app/grid/product/12.1.0/grid/log/[HOSTNAME]/evmd/evmlogger.info -l / u00/app/grid/product/12.1.0/grid/log/[HOSTNAME]/evmd/evmlogger.log

grid 5202 1 0 10:12 ? 00:00:00 /u00/app/grid/product/12.1.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit

grid 5353 1 0 10:12 ? 00:00:01 /u00/app/grid/product/12.1.0/grid/bin/cssdagent

grid 5376 1 0 10:12 ? 00:00:02 /u00/app/grid/product/12.1.0/grid/bin/ocssd.bin

grid 5486 1 0 10:13 ? 00:00:00 asm_pmon_+ASM

grid 5488 1 0 10:13 ? 00:00:00 asm_psp0_+ASM

grid 5490 1 0 10:13 ? 00:00:00 asm_vktm_+ASM

grid 5494 1 0 10:13 ? 00:00:00 asm_gen0_+ASM

grid 5496 1 0 10:13 ? 00:00:00 asm_mman_+ASM

grid 5500 1 0 10:13 ? 00:00:00 asm_diag_+ASM

grid 5502 1 0 10:13 ? 00:00:00 asm_dia0_+ASM

grid 5504 1 0 10:13 ? 00:00:00 asm_dbw0_+ASM

grid 5506 1 0 10:13 ? 00:00:00 asm_lgwr_+ASM

grid 5508 1 0 10:13 ? 00:00:00 asm_ckpt_+ASM

grid 5510 1 0 10:13 ? 00:00:00 asm_smon_+ASM

grid 5512 1 0 10:13 ? 00:00:00 asm_lreg_+ASM

grid 5514 1 0 10:13 ? 00:00:00 asm_pxmn_+ASM

grid 5516 1 0 10:13 ? 00:00:00 asm_rbal_+ASM

grid 5518 1 0 10:13 ? 00:00:00 asm_gmon_+ASM

grid 5520 1 0 10:13 ? 00:00:00 asm_mmon_+ASM

grid 5522 1 0 10:13 ? 00:00:00 asm_mmnl_+ASM

grid 5535 1 0 10:14 ? 00:00:00 oracle+ASM (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

grid 5700 1 0 10:14 ? 00:00:00 asm_ars0_+ASM

root 5865 667 0 10:18 pts/1 00:00:00 grep grid

[root@lxcora10 ~]#

Modify $PATH for root in .bashrc File

Modify the $PATH variable for "root" user so that root can run the "crsctl" executable to check Oracle GI status as shown below.

[root@lxcora10 ~]# pwd

/root

[root@lxcora10 ~]# vi .bashrc

[root@lxcora10 ~]# cat .bashrc

# .bashrc

# User specific aliases and functions

alias rm='rm -i'

alias cp='cp -i'

alias mv='mv -i'

# Source global definitions

if [ -f /etc/bashrc ]; then

. /etc/bashrc

fi

export PATH=$PATH:/u00/app/grid/product/12.1.0/grid/bin

[root@lxcora10 ~]# . .bashrc

[root@lxcora10 ~]#

Add Environment Variables to .bashrc for "grid" Linux User

Add environment variables for "grid" linux user as shown below.

[grid@lxcora01 ~]$ vi .bashcr

[grid@lxcora01 ~]$ . .bashrc

[grid@lxcora01 ~]$ cat .bashrc

# .bashrc

# Source global definitions

if [ -f /etc/bashrc ]; then

. /etc/bashrc

fi

# User specific aliases and functions

umask 022

export ORACLE_HOME=/u00/app/grid/product/12.1.0/grid

export GRID_HOME=/u00/app/grid/product/12.1.0/grid

export ORACLE_SID=+ASM

export PATH=$PATH:$ORACLE_HOME/bin

[grid@lxcora01 ~]$

Check Status of Oracle Grid Infrastructure (GI)

Check status of Oracle GI as shown below. The "ora.ons" and "ora.diskmon" will both by default have status "OFFLINE OFFLINE". All others should show status "ONLINE ONLINE" as shown below.

[root@lxcora10 ~]# crsctl stat res -t

--------------------------------------------------------------------------------

Name Target State Server State details

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.LISTENER.lsnr

ONLINE ONLINE lxcora10 STABLE

ora.SYSTEMDG.dg

ONLINE ONLINE lxcora10 STABLE

ora.asm

ONLINE ONLINE lxcora10 Started,STABLE

ora.ons

OFFLINE OFFLINE lxcora10 STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.cssd

1 ONLINE ONLINE lxcora10 STABLE

ora.diskmon

1 OFFLINE OFFLINE STABLE

ora.evmd

1 ONLINE ONLINE lxcora10 STABLE

--------------------------------------------------------------------------------

[root@lxcora10 ~]#

Shutdown Container and Reboot Host

Shutdown the container and reboot the host to verify that the system restarts successfully after a reboot of the host server OS.

gstanden@vmem1:~$ sudo lxc-stop -n lxcora10

[sudo] password for gstanden:

gstanden@vmem1:~$ sudo lxc-info -n lxcora10

Name: lxcora10

State: STOPPED

gstanden@vmem1:~$

After reboot verify that the Oracle GI software stack including listener and SYSTEMDG ASM diskgroup come up normally indicating device is correctly configured as shown below. The "ora.ons" and "ora.diskmon" will have default status of "OFFLINE OFFLINE". All others should show "ONLINE ONLINE".

[root@lxcora01 ~]# crsctl stat res -t

--------------------------------------------------------------------------------

Name Target State Server State details

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.LISTENER.lsnr

ONLINE ONLINE lxcora01 STABLE

ora.SYSTEMDG.dg

ONLINE ONLINE lxcora01 STABLE

ora.asm

ONLINE ONLINE lxcora01 Started,STABLE

ora.ons

OFFLINE OFFLINE lxcora01 STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.cssd

1 ONLINE ONLINE lxcora01 STABLE

ora.diskmon

1 OFFLINE OFFLINE STABLE

ora.evmd

1 ONLINE ONLINE lxcora01 STABLE

--------------------------------------------------------------------------------

[root@lxcora01 ~]#