SCST KVM UEK 3.8
This guide explains how to build SCST SAN on an Oracle Enterprise Linux 6.5 KVM guest VM based on the excellent instructions here at spiderbird. That guide also has a wealth of reference URLs that can be accessed as well.
This is the revised version that uses the Oracle Enterprise Linux UEK version 3 kernel for the build. This was motivated for consistency (so that OEL 6.5 is running it's own proprietary kernel, and not a CentOS kernel) and so that DTrace tool which Oracle has ported over to it's UEK kernel could be utilized (although issues were encountered on the DTrace part). The first build utilizing the CentOS kernel was done due to time constraints and the kernel source for Oracle UEK kernel version 3 was not handy already downloaded at that time, and since it has to be downloaded as part of the DVD from edelivery.oracle.com a POC was done quickly using CentOS 6 kernel. Now in the present blog that is remedied so that the SCST SAN is a 100% OEL UEK kernel custom-modified for SCST iSCSI Linux SAN.
I intend to use this KVM guest SCST SAN to present 4K logcial/4K physical virtual LUNs to Oracle running in other KVM guests, including Oracle Real Application Clusters running in KVM guests. Because TGT cannot handle LUNs which are 4K logical / 4K physical it was necessary to get up to speed on SCST which can. This sourceforge SAN feature comparison sheet shows the full power of SCST compared to other linux SAN options.
Ensure the virtual ethernet interface, typically "eth0" in the OEL 6.5 KVM guest is set to "connect automatically". If not, configure it as shown in the screenshots below. This can be configured from the server desktop, if using OEL 6.5 with a desktop, or simply edited in the ifcfg-eth0 file as shown below. As shown in bold below, be sure that "ONBOOT=yes" has been set in the ifcfg-ethX file where in this case my interface is ifcfg-eth0.
[root@oracle651 network-scripts]# cat ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
UUID=824a1d97-1dc7-410b-89a5-49b353be826a
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=dhcp
DNS1=127.0.0.1
DOMAIN=vmem.org
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"
HWADDR=52:54:00:66:26:A0
DNS2=192.168.1.1
PEERDNS=no
PEERROUTES=yes
LAST_CONNECT=1407733083
[root@oracle651 network-scripts]#
Download the required Oracle Linux 6.5 Source ISO files. The Oracle Enterprise Linux ISO files are only available as downloads from edelivery.oracle.com so therefore an Oracle Network login will be needed. (Both an Oracle Network Login and the ISO's are free, but registration is required). The ISOs needed are as shown below.
gstanden@vmem1:~/Downloads$ ls -lrt V4*.iso
-rw-rw-r-- 1 libvirt-qemu kvm 3885117440 Aug 7 15:50 V41362-01.iso <- Install ISO
-rw-rw-r-- 1 gstanden gstanden 3221997568 Aug 12 17:55 V41365-01.iso <- Source Code SRPMS ISO #1
-rw-rw-r-- 1 gstanden gstanden 2116456448 Aug 12 18:03 V41366-01.iso <- Source Code SRPMS ISO #2
gstanden@vmem1:~/Downloads$
Do a yum update to update the kernel before beginning this work. Once this work is done the kernel will be locked so that there will be no more kernel updated, because this procedure builds a custom kernel for SCST which will not be updated any further normally (without going through this entire procedure again!).
Once the yum update is complete, reboot the KVM guest so that the just-updated kernel is running in the KVM guest.
Install the following packages using yum as shown below. Do this as root. Note that further on in this blog there was a missing dependency on "xmlto" which I am retroactively addressing here in this blog so there won't be that missing dependency.
[root@oracle651 ~]# yum install newt-devel
[root@oracle651 ~]# yum install asciidoc
[root@oracle651 ~]# yum install xmlto
[root@oracle651 ~]# yum install libdtrace-ctf-devel
As root, add the "mockbuild" user as shown below.
[root@oracle651 ~]# useradd mockbuild
Now su to the "mockbuild" user as shown below. A password is not needed for the mockbuild user unless you explicitly want one. It's just an account that will be used for this build.
[root@oracle651 ~]# su - mockbuild
[mockbuild@oracle651 ~]$ pwd
/home/mockbuild
[mockbuild@oracle651 ~]$ id
uid=501(mockbuild) gid=501(mockbuild) groups=501(mockbuild) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[mockbuild@oracle651 ~]$
Run the following commands as the mockbuild user from the /home/mockbuild directory as shown below. It will be necessary to obtain the OEL UEK version 3 source rpm's from edelivery.oracle.com which are on the source DVD's downloadable from there, and then stage them as shown in /home/mockbuild with correct ownership of mockbuild:mockbuild as shown below.
[mockbuild@oracle651 ~]$ pwd
/home/mockbuild
[mockbuild@oracle651 ~]$ ls -lrt
total 171080
-rw-r--r-- 1 mockbuild mockbuild 85551190 Aug 23 11:58 kernel-uek-2.6.39-400.211.1.el6uek.src.rpm
-rw-r--r-- 1 mockbuild mockbuild 89621592 Aug 23 11:58 kernel-uek-3.8.13-16.2.1.el6uek.src.rpm
drwxrwxr-x 8 mockbuild mockbuild 4096 Aug 23 13:49 rpmbuild
[mockbuild@oracle651 ~]$
Next run the steps as shown below from the /home/mockbuild directory while connected as the linux "mockbuild" user.
[mockbuild@oracle651 ~]$ mkdir -p ~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
[mockbuild@oracle651 ~]$ echo '%_topdir %(echo $HOME)/rpmbuild' > ~/.rpmmacros
[mockbuild@oracle651 ~]$ rpm -i kernel-uek-3.8.13-16.2.1.el6uek.src.rpm 2>&1 | grep -v mock
Before running the below "rpmbuild" command in step after the next one, open another window as root and run the entropy generator as shown here. If the run of rpmbuild in the next step bogs down, run this command again to refill the entropy pool. It will make the rpmbuild complete faster.
[root@oracle651 ~]# rngd -r /dev/urandom
[root@oracle651 ~]#
Now you can run the command. If it bogs down towards the end run the above rngd command again. Note: be sure when running the command below, that if you cut and paste from this blog that the word "--target" has "two dashes" i.e "--" in front of it, and not just one big dash. Sometimes characterset conversion can cause two dashed cut and pasted to become an uninterpretable single large dash.
Also, note the change for kernel-uek.spec as shown in bold below (compared to say CentOS 6 or RHEL 6, for example).
[root@oracle651 ~]# su - mockbuild
[mockbuild@oracle651 ~]$ rpmbuild -bp --target=$(uname -m) ./rpmbuild/SPECS/kernel-uek.spec
Building target platforms: x86_64
Building for target x86_64
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.1l1BDW
+ umask 022
+ cd /home/mockbuild/rpmbuild/BUILD
+ LANG=C
+ export LANG
+ unset DISPLAY
+ patch_command='patch -p1 -F1 -s'
+ '[' -f .shared-srctree ']'
+ '[' '!' -d kernel-3.8.13/vanilla-2.6.13 ']'
+ '[' -d kernel-3.8.13/vanilla-3.8.13 ']'
+ rm -f pax_global_header
+ [[ ! -z '' ]]
+ cd /home/mockbuild/rpmbuild/BUILD
+ rm -rf kernel-3.8.13
+ /bin/mkdir -p kernel-3.8.13
+ cd kernel-3.8.13
+ /usr/bin/bzip2 -dc /home/mockbuild/rpmbuild/SOURCES/linux-3.8.13.tar.bz2
+ /bin/tar -xf -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ /bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ mv linux-3.8.13 vanilla-3.8.13
+ [[ ! -z '' ]]
+ cp -rl vanilla-3.8.13 vanilla-2.6.13
+ cd vanilla-2.6.13
+ cd ..
+ '[' -d linux-3.8.13.x86_64 ']'
+ cp -rl vanilla-2.6.13 linux-3.8.13-16.2.1.el6uek
+ cd linux-3.8.13-16.2.1.el6uek
+ cp /usr/lib/rpm/find-debuginfo.sh /home/mockbuild/rpmbuild/BUILD
+ patch /home/mockbuild/rpmbuild/BUILD/find-debuginfo.sh /home/mockbuild/rpmbuild/SOURCES/debuginfo-g1.diff
patching file /home/mockbuild/rpmbuild/BUILD/find-debuginfo.sh
+ chmod +x /home/mockbuild/rpmbuild/BUILD/find-debuginfo.sh
+ mkdir -p configs
+ cp /home/mockbuild/rpmbuild/SOURCES/config-x86_64-debug configs/config-debug
+ cp /home/mockbuild/rpmbuild/SOURCES/config-x86_64 configs/config
+ find . '(' -name '*.orig' -o -name '*~' ')' -exec rm -f '{}' ';'
+ exit 0
Now execute the following command as shown below. The full correct expected output of the following command is attached to this blog as "scst.log" at the end of this blog page.
[mockbuild@oracle651 ~]$ svn co https://svn.code.sf.net/p/scst/svn/branches/2.2.x scst
The scst code is installed under the /home/mockbuild directory. Now Change directory as shown below and check availability of the required patches for the 3.8 kernel as shown below.
[mockbuild@oracle651 ~]$ uname -a
Linux oracle651.vmem.org 3.8.13-35.3.4.el6uek.x86_64 #2 SMP Wed Jul 30 00:59:13 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[mockbuild@oracle651 ~]$ id
uid=501(mockbuild) gid=501(mockbuild) groups=501(mockbuild)
[mockbuild@oracle651 ~]$ cd ~/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek/
[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ ls -l ~/scst/scst/kernel/scst_exec_req_fifo-3.8.patch
-rw-rw-r-- 1 mockbuild mockbuild 12833 Aug 23 14:21 /home/mockbuild/scst/scst/kernel/scst_exec_req_fifo-3.8.patch
[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$
[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ ls -l ~/scst/iscsi-scst/kernel/patches/put_page_callback-3.8.patch
-rw-rw-r-- 1 mockbuild mockbuild 12499 Aug 23 14:21 /home/mockbuild/scst/iscsi-scst/kernel/patches/put_page_callback-3.8.patch
[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$
Apply the patches as shown below. Note, if you get the following error when you try to run the command it is because your "p1" is typed as a "pl" (p-ell). Be sure to use "p1" (p-one).
[mockbuild@oracle651 ~]$ cd ~/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek/
[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ patch -p1 < ~/scst/scst/kernel/scst_exec_req_fifo-3.8.patch | tee /home/mockbuild/logs/scst_exec_req_fifo-3.8.patch.log
patching file block/blk-map.c
patching file include/linux/blkdev.h
patching file include/linux/scatterlist.h
patching file lib/scatterlist.c
[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$
[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ patch -p1 < ~/scst/iscsi-scst/kernel/patches/put_page_callback-3.8.patch | tee /home/mockbuild/logs/put_page_callback-3.8.patch.log
patching file drivers/net/macvtap.c
patching file drivers/net/tun.c
Hunk #1 succeeded at 1014 (offset 2 lines).
patching file drivers/net/vmxnet3/vmxnet3_drv.c
patching file drivers/net/xen-netback/netback.c
Hunk #1 succeeded at 1329 (offset 239 lines).
Hunk #2 succeeded at 1867 with fuzz 1 (offset 318 lines).
patching file include/linux/mm_types.h
patching file include/linux/net.h
patching file include/linux/skbuff.h
Hunk #1 succeeded at 1949 (offset 5 lines).
Hunk #2 succeeded at 1972 (offset 5 lines).
patching file net/Kconfig
patching file net/core/skbuff.c
patching file net/core/sock.c
patching file net/ipv4/Makefile
patching file net/ipv4/ip_output.c
patching file net/ipv4/tcp.c
patching file net/ipv4/tcp_zero_copy.c
patching file net/ipv6/ip6_output.c
Hunk #1 succeeded at 1461 (offset 10 lines).
[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$
Check to make sure the config files are about the same. On my build, there was no .config at this point. I proceeded on to the "make menuconfig" step next, and made the required changes to configuration, and then saved, and then there was a ".config" file in the directory.
One difference here is that I made the SCSI Disk Support option a built-in instead of a module. It was ticked with <M> and I changed it to <*>.
I ran the "make oldconfig" and then did a diff on the config files as shown below. Most of the differences looked like the changes that were make with menuconfig so I went ahead and proceeded to kernel compile step.
[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ cp -p /boot/config-3.8.13-16.2.1.el6uek.x86_64 .config
[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$
The menuconfig GUI opens. Navigate through the GUI and choose the options as quoted and described at the spiderweb reference here. For completeness, those changes that need to be made and verified in menuconfig are as follows from the reference but restated here for convenience.
Now run the command as shown below:
[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ make menuconfig
and answer change the following as shown below
“Network support”->”Networking options”
“TCP/IP networking”
“TCP/IP zero-copy transfer completion notification”
“Device Drivers”->”SCSI device support”
”SCSI disk support”
“Enable the block layer”->”IO Schedulers”->”CFQ I/O scheduler”->“Default I/O scheduler”
"No-op"
“Processor type an features”->”Preemption Model”
“No Forced Preemption - Server"
Edit the Makefile as shown with the EXTRAVERSION information. In this example, the kernel version will have no special identifier and will just overwrite the existing kernel with a kernel of the same name but with the added SCST features. It is also possible to append a custom tag to mark the kernel as modified such as "-16.2.1.el6uek.scst.x86_64".
[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ vi Makefile
[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ grep EXTRAVERSION Makefile | grep -v KERNELVERSION
EXTRAVERSION =-16.2.1.el6uek.x86_64
[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$
Now run make as shown below.
[mockbuild@oracle651 linux-3.8.13-16.2.1.el6uek]$ time make -j8 all | tee /home/mockbuild/make.log
Now make modules install as shown below, as the "root" user.
[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ su - root
[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# cd /home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64
[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# make modules_install
cp -pR /lib/modules/3.8.13-16.2.1.el6uek.x86_64/ /lib/modules/3.8.13-16.2.1.el6uek.scst.x86_64/
make modules install
cd /home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek/
Next, again as root, do a "make install" as shown below. The errors shown are OK. These modules will be added after reboot.
[root@oracle651 linux-3.8.13-16.2.1.el6uek]# make install | tee /home/mockbuild/install.log
sh /home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek/arch/x86/boot/install.sh 3.8.13-16.2.1.el6uek.x86_64 arch/x86/boot/bzImage \
System.map "/boot"
ERROR: modinfo: could not find module iscsi_scst
ERROR: modinfo: could not find module scst_vdisk
ERROR: modinfo: could not find module scst
[root@oracle651 linux-3.8.13-16.2.1.el6uek]#
Now edit /boot/grub/menu.lst to update the server to use the new kernel at boot as shown below. Change the value of "default=" to use the new kernel. The first kernel listed is always the "0" (zero) kernel so in this example set "detault=0". Also, while editing this file, remove the "rhgb quiet" from that kernel line so that all boot output messages will be buffered to console, and also add at the end of the kernel line "console=ttyS0" so that KVM console is configured.
[root@oracle651 linux-3.8.13-16.2.1.el6uek]# cat /boot/grub/menu.lst
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/mapper/vg_oracle651-lv_root
# initrd /initrd-[generic-]version.img
#boot=/dev/vda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Oracle Linux Server (3.8.13-16.2.1.el6uek.x86_64)
root (hd0,0)
kernel /vmlinuz-3.8.13-16.2.1.el6uek.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM crashkernel=auto console=ttyS0
initrd /initramfs-3.8.13-16.2.1.el6uek.x86_64.img
title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-scst)
root (hd0,0)
kernel /vmlinuz-2.6.32-scst ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM crashkernel=auto console=ttyS0
initrd /initramfs-2.6.32-scst.img
title Oracle Linux Server (3.8.13-35.3.4.el6uek.x86_64)
root (hd0,0)
kernel /vmlinuz-3.8.13-35.3.4.el6uek.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-3.8.13-35.3.4.el6uek.x86_64.img
title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-431.23.3.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-431.23.3.el6.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto
initrd /initramfs-2.6.32-431.23.3.el6.x86_64.img
title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-16.2.1.el6uek.x86_64)
root (hd0,0)
kernel /vmlinuz-3.8.13-16.2.1.el6uek.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-3.8.13-16.2.1.el6uek.x86_64.img
title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-431.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-431.el6.x86_64.img
[root@oracle651 linux-3.8.13-16.2.1.el6uek]#
Now copy the .config file over to the "/boot" directory as shown below. Expected output and commands are shown below.
[root@oracle651 linux-3.8.13-16.2.1.el6uek]# cp .config /boot/config-3.8.13-16.2.1.el6uek.x86_64
cp: overwrite `/boot/config-3.8.13-16.2.1.el6uek.x86_64'? y
[root@oracle651 linux-3.8.13-16.2.1.el6uek]#
Reboot the KVM guest. Be sure the config file has been named exactly as shown above before rebooting, or if your kernel version was slightly different, be sure the kernel version and format of the name of the config file is of the format "config-ver.s.ion-scst.el6.x86_64".
Next change the various Makefiles to release version by issuing “make debug2release” in the main scst dir. This will remove the debugging stuff. Do this as root.
Linux oracle651.vmem.org 3.8.13-16.2.1.el6uek.x86_64 #1 SMP Sun Aug 24 11:05:09 CDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@oracle651 ~]# cd /home/mockbuild/scst
[root@oracle651 scst]# make debug2release | tee /home/mockbuild/debug2release.log
rm -f TAGS
cd scst && make extraclean
make[1]: Entering directory `/home/mockbuild/scst/scst'
cd src && make extraclean
make[2]: Entering directory `/home/mockbuild/scst/scst/src'
rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend Modules.symvers \
Module.symvers Module.markers modules.order
rm -rf .tmp_versions
cd dev_handlers && make clean
make[3]: Entering directory `/home/mockbuild/scst/scst/src/dev_handlers'
rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend Modules.symvers \
Module.symvers Module.markers modules.order
rm -rf .tmp_versions
make[3]: Leaving directory `/home/mockbuild/scst/scst/src/dev_handlers'
cd dev_handlers && make extraclean
make[3]: Entering directory `/home/mockbuild/scst/scst/src/dev_handlers'
rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend Modules.symvers \
Module.symvers Module.markers modules.order
rm -rf .tmp_versions
rm -f *.orig *.rej
make[3]: Leaving directory `/home/mockbuild/scst/scst/src/dev_handlers'
rm -f *.orig *.rej
make[2]: Leaving directory `/home/mockbuild/scst/scst/src'
make[1]: Leaving directory `/home/mockbuild/scst/scst'
make[1]: Entering directory `/home/mockbuild/scst/doc'
rm -f *.txt *.html *.tex *.dvi *.ps *.pdf *.info *.lyx *.rtf
rm -f *.orig *.rej
make[1]: Leaving directory `/home/mockbuild/scst/doc'
make[1]: Entering directory `/home/mockbuild/scst/qla2x00t'
rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend *~ Modules.symvers \
Module.symvers Module.markers modules.order
rm -rf .tmp_versions
rm -f *.orig *.rej
make[1]: Leaving directory `/home/mockbuild/scst/qla2x00t'
make[1]: Entering directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'
rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend *~ Modules.symvers \
Module.symvers Module.markers modules.order
rm -rf .tmp_versions
rm -f *.orig *.rej
make[1]: Leaving directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'
make[1]: Entering directory `/home/mockbuild/scst/srpt'
make -C /lib/modules/3.8.13-16.2.1.el6uek.x86_64/build SUBDIRS=/home/mockbuild/scst/srpt/src clean
make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
rm -f src/Modules.symvers src/Module.symvers src/Module.markers \
src/modules.order
rm -f *.orig *.rej
make[1]: Leaving directory `/home/mockbuild/scst/srpt'
make[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'
make -C usr extraclean
make[2]: Entering directory `/home/mockbuild/scst/iscsi-scst/usr'
rm -f *.o iscsi-scstd iscsi-scst-adm .depend*
rm -f *.orig *.rej
make[2]: Leaving directory `/home/mockbuild/scst/iscsi-scst/usr'
make -C /lib/modules/3.8.13-16.2.1.el6uek.x86_64/build SUBDIRS=/home/mockbuild/scst/iscsi-scst/kernel clean
make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
rm -f kernel/Modules.symvers kernel/Module.symvers \
kernel/Module.markers kernel/modules.order \
include/iscsi_scst_itf_ver.h \
kernel/*.orig kernel/*.rej
make[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'
make[1]: Entering directory `/home/mockbuild/scst/usr/fileio'
rm -f *.o fileio_tgt .depend*
rm -f *.orig *.rej
make[1]: Leaving directory `/home/mockbuild/scst/usr/fileio'
make[1]: Entering directory `/home/mockbuild/scst/scst_local'
make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
rm -f *.orig *.rej
make[1]: Leaving directory `/home/mockbuild/scst/scst_local'
cd scst && make debug2release
make[1]: Entering directory `/home/mockbuild/scst/scst'
cd src && make debug2release
make[2]: Entering directory `/home/mockbuild/scst/scst/src'
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefile
sed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefile
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefile
rm Makefile.aa
cd dev_handlers && make debug2release
make[3]: Entering directory `/home/mockbuild/scst/scst/src/dev_handlers'
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefile
sed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefile
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefile
rm Makefile.aa
make[3]: Leaving directory `/home/mockbuild/scst/scst/src/dev_handlers'
make[2]: Leaving directory `/home/mockbuild/scst/scst/src'
make[1]: Leaving directory `/home/mockbuild/scst/scst'
make[1]: Entering directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefile
sed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefile
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefile
rm Makefile.aa
make[1]: Leaving directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'
make[1]: Entering directory `/home/mockbuild/scst/srpt'
make clean
make[2]: Entering directory `/home/mockbuild/scst/srpt'
make -C /lib/modules/3.8.13-16.2.1.el6uek.x86_64/build SUBDIRS=/home/mockbuild/scst/srpt/src clean
make[3]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
make[3]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
rm -f src/Modules.symvers src/Module.symvers src/Module.markers \
src/modules.order
make[2]: Leaving directory `/home/mockbuild/scst/srpt'
make[1]: Leaving directory `/home/mockbuild/scst/srpt'
make[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ /home/mockbuild/scst/iscsi-scst/kernel/Makefile
sed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ /home/mockbuild/scst/iscsi-scst/kernel/Makefile
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ /home/mockbuild/scst/iscsi-scst/kernel/Makefile
rm /home/mockbuild/scst/iscsi-scst/kernel/Makefile.aa
make[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'
make[1]: Entering directory `/home/mockbuild/scst/usr/fileio'
sed -i.aa s/"^CFLAGS += \-DEXTRACHECKS"/"#CFLAGS += \-DEXTRACHECKS"/ Makefile
sed -i.aa s/"#CFLAGS += \-DTRACING"/"CFLAGS += \-DTRACING"/ Makefile
sed -i.aa s/"^CFLAGS += \-DDEBUG -g -fno-inline -fno-inline-functions"/"#CFLAGS += \-DDEBUG -g -fno-inline -fno-inline-functions"/ Makefile
rm Makefile.aa
make[1]: Leaving directory `/home/mockbuild/scst/usr/fileio'
make[1]: Entering directory `/home/mockbuild/scst/scst_local'
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefile
sed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefile
sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefile
rm Makefile.aa
make[1]: Leaving directory `/home/mockbuild/scst/scst_local'
[root@oracle651 scst]#
Now go ahead and issue “make scst scst_install” as root in the mockbuild directory’s scst folder. Commands and expected output are shown below. Do this as "root" user.
[root@oracle651 scst]# make scst scst_install | tee /home/mockbuild/scst_install.log
cd scst && make all
make[1]: Entering directory `/home/mockbuild/scst/scst'
cd src && make all
make[2]: Entering directory `/home/mockbuild/scst/scst/src'
make -C /lib/modules/3.8.13-16.2.1.el6uek.x86_64/build SUBDIRS=/home/mockbuild/scst/scst/src BUILD_DEV=m
make[3]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
LD /home/mockbuild/scst/scst/src/built-in.o
CC [M] /home/mockbuild/scst/scst/src/scst_main.o
CC [M] /home/mockbuild/scst/scst/src/scst_targ.o
CC [M] /home/mockbuild/scst/scst/src/scst_lib.o
CC [M] /home/mockbuild/scst/scst/src/scst_sysfs.o
CC [M] /home/mockbuild/scst/scst/src/scst_mem.o
CC [M] /home/mockbuild/scst/scst/src/scst_debug.o
CC [M] /home/mockbuild/scst/scst/src/scst_pres.o
CC [M] /home/mockbuild/scst/scst/src/scst_tg.o
LD [M] /home/mockbuild/scst/scst/src/scst.o
LD /home/mockbuild/scst/scst/src/dev_handlers/built-in.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_cdrom.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_changer.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_disk.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_modisk.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_tape.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_vdisk.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_raid.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_processor.o
CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_user.o
Building modules, stage 2.
MODPOST 10 modules
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_cdrom.mod.o
CTF
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_cdrom.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_changer.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_changer.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_disk.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_disk.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_modisk.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_modisk.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_processor.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_processor.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_raid.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_raid.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_tape.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_tape.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_user.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_user.ko
CC /home/mockbuild/scst/scst/src/dev_handlers/scst_vdisk.mod.o
LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_vdisk.ko
CC /home/mockbuild/scst/scst/src/scst.mod.o
LD [M] /home/mockbuild/scst/scst/src/scst.ko
make[3]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
make[2]: Leaving directory `/home/mockbuild/scst/scst/src'
make[1]: Leaving directory `/home/mockbuild/scst/scst'
cd scst && make install
make[1]: Entering directory `/home/mockbuild/scst/scst'
cd src && make install
make[2]: Entering directory `/home/mockbuild/scst/scst/src'
make -C /lib/modules/3.8.13-16.2.1.el6uek.x86_64/build SUBDIRS=/home/mockbuild/scst/scst/src BUILD_DEV=m
make[3]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
Building modules, stage 2.
MODPOST 10 modules
make[3]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
rm -f /lib/modules/3.8.13-16.2.1.el6uek.x86_64/extra/scsi_tgt.ko
install -d /lib/modules/3.8.13-16.2.1.el6uek.x86_64/extra
install -d /lib/modules/3.8.13-16.2.1.el6uek.x86_64/extra/dev_handlers
install -m 644 dev_handlers/*.ko /lib/modules/3.8.13-16.2.1.el6uek.x86_64/extra/dev_handlers
install -m 644 scst.ko /lib/modules/3.8.13-16.2.1.el6uek.x86_64/extra
install -d /usr/local/include/scst
install -m 644 ../include/scst.h /usr/local/include/scst
install -m 644 ../include/scst_sgv.h /usr/local/include/scst
install -m 644 ../include/scst_debug.h /usr/local/include/scst
install -m 644 ../include/scst_user.h /usr/local/include/scst
install -m 644 ../include/scst_const.h /usr/local/include/scst
rm -f /usr/local/include/scst/Modules.symvers
install -m 644 Module.symvers /usr/local/include/scst
/sbin/depmod -a 3.8.13-16.2.1.el6uek.x86_64
mkdir -p /var/lib/scst/pr
****************************************************************
*!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!*
*!! !!*
*!! Now don't forget to rebuild and reinstall all your !!*
*!! target drivers, custom dev handlers and necessary user !!*
*!! space applications. Otherwise, because of the versions !!*
*!! mismatch, you could have many problems and crashes. !!*
*!! See IMPORTANT note in the "Installation" section of !!*
*!! SCST's README file for more info. !!*
*!! !!*
*!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!*
****************************************************************
make[2]: Leaving directory `/home/mockbuild/scst/scst/src'
make[1]: Leaving directory `/home/mockbuild/scst/scst'
[root@oracle651 scst]#
Issue “make iscsi iscsi_install” as root in the /home/mockbuild folder. Commands and expected output are shown below.
[root@oracle651 scst]# make iscsi iscsi_install | tee /home/mockbuild/iscsi_install.log
cd iscsi-scst && make all
make[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'
echo "/* Autogenerated, don't edit */" >include/iscsi_scst_itf_ver.h
echo "" >>include/iscsi_scst_itf_ver.h
echo -n "#define ISCSI_SCST_INTERFACE_VERSION " >>include/iscsi_scst_itf_ver.h
echo -n "ISCSI_VERSION_STRING \"_\" " >>include/iscsi_scst_itf_ver.h
echo "\"`sha1sum include/iscsi_scst.h|awk '{printf $1}'`\"" >>include/iscsi_scst_itf_ver.h
make -C usr SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/include
make[2]: Entering directory `/home/mockbuild/scst/iscsi-scst/usr'
cc -M -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsid.c iscsi_scstd.c conn.c session.c target.c message.c ctldev.c log.c chap.c event.c param.c config.c isns.c md5.c sha1.c >.depend_d
cc -c -o iscsid.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsid.c
cc -c -o iscsi_scstd.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsi_scstd.c
cc -c -o conn.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE conn.c
cc -c -o session.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE session.c
cc -c -o target.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE target.c
cc -c -o message.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE message.c
cc -c -o ctldev.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE ctldev.c
cc -c -o log.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE log.c
cc -c -o chap.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE chap.c
cc -c -o event.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE event.c
cc -c -o param.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE param.c
cc -c -o config.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE config.c
cc -c -o isns.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE isns.c
cc -c -o md5.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE md5.c
cc -c -o sha1.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE sha1.c
cc iscsid.o iscsi_scstd.o conn.o session.o target.o message.o ctldev.o log.o chap.o event.o param.o config.o isns.o md5.o sha1.o -o iscsi-scstd
cc -M -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsi_adm.c param.c >.depend_adm
cc -c -o iscsi_adm.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsi_adm.c
cc iscsi_adm.o param.o -o iscsi-scst-adm
make[2]: Leaving directory `/home/mockbuild/scst/iscsi-scst/usr'
cp /home/mockbuild/scst/iscsi-scst/../scst/src/Module.symvers kernel/
make -C /lib/modules/3.8.13-16.2.1.el6uek.x86_64/build SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/include SUBDIRS=/home/mockbuild/scst/iscsi-scst/kernel modules
make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/iscsi.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/nthread.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/config.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/digest.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/conn.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/session.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/target.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/event.o
CC [M] /home/mockbuild/scst/iscsi-scst/kernel/param.o
LD [M] /home/mockbuild/scst/iscsi-scst/kernel/iscsi-scst.o
Building modules, stage 2.
MODPOST 1 modules
CC /home/mockbuild/scst/iscsi-scst/kernel/iscsi-scst.mod.o
CTF
LD [M] /home/mockbuild/scst/iscsi-scst/kernel/iscsi-scst.ko
make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
make[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'
cd iscsi-scst && make DESTDIR= install
make[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'
make -C usr SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/include
make[2]: Entering directory `/home/mockbuild/scst/iscsi-scst/usr'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/home/mockbuild/scst/iscsi-scst/usr'
cp /home/mockbuild/scst/iscsi-scst/../scst/src/Module.symvers kernel/
make -C /lib/modules/3.8.13-16.2.1.el6uek.x86_64/build SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/include SUBDIRS=/home/mockbuild/scst/iscsi-scst/kernel modules
make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
Building modules, stage 2.
MODPOST 1 modules
make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-3.8.13/linux-3.8.13-16.2.1.el6uek'
removed `/usr/local/sbin/iscsi-scstd'
`usr/iscsi-scstd' -> `/usr/local/sbin/iscsi-scstd'
removed `/usr/local/man/man5/iscsi-scstd.conf.5'
`doc/manpages/iscsi-scstd.conf.5' -> `/usr/local/man/man5/iscsi-scstd.conf.5'
removed `/usr/local/man/man8/iscsi-scstd.8'
`doc/manpages/iscsi-scstd.8' -> `/usr/local/man/man8/iscsi-scstd.8'
removed `/usr/local/sbin/iscsi-scst-adm'
`usr/iscsi-scst-adm' -> `/usr/local/sbin/iscsi-scst-adm'
removed `/usr/local/man/man8/iscsi-scst-adm.8'
`doc/manpages/iscsi-scst-adm.8' -> `/usr/local/man/man8/iscsi-scst-adm.8'
`kernel/iscsi-scst.ko' -> `/lib/modules/3.8.13-16.2.1.el6uek.x86_64/extra/iscsi-scst.ko'
/sbin/depmod -aq 3.8.13-16.2.1.el6uek.x86_64
make[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'
[root@oracle651 scst]#
Finally, issue “make scstadm scstadm_install” as well. This finalizes the compiling part. Do this as root. Commands and expected output are shown below.
[root@oracle651 scst]# make scstadm scstadm_install | tee /home/mockbuild/scstadm.log
cd scstadmin && make all
make[1]: Entering directory `/home/mockbuild/scst/scstadmin'
cd scstadmin && make all
make[2]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'
Checking if your kit is complete...
Looks good
Writing Makefile for SCST-SCST
make -C scst-0.9.10
make[3]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'
cp lib/SCST/SCST.pm blib/lib/SCST/SCST.pm
Manifying blib/man3/SCST::SCST.3pm
make[3]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'
make[2]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'
make[1]: Leaving directory `/home/mockbuild/scst/scstadmin'
cd scstadmin && make install
make[1]: Entering directory `/home/mockbuild/scst/scstadmin'
cd scstadmin && make install
make[2]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'
Writing Makefile for SCST-SCST
make -C scst-0.9.10
make[3]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'
Skip blib/lib/SCST/SCST.pm (unchanged)
Manifying blib/man3/SCST::SCST.3pm
make[3]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'
make -C scst-0.9.10 install DESTDIR=
make[3]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'
Manifying blib/man3/SCST::SCST.3pm
Installing /usr/local/share/man/man3/SCST::SCST.3pm
Appending installation info to /usr/lib64/perl5/perllocal.pod
make[3]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'
install -d /usr/local/sbin
install -m 755 scstadmin /usr/local/sbin
make[2]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'
for m in scstadmin/man*/*.[1-9]; do \
if [ -e $m ]; then \
d=/usr/local/man/$(basename $(dirname $m)); \
if install -d $d; then \
install -m 644 $m $d; \
fi \
fi \
done
install -d /etc/init.d
tmp=/tmp/scst-initd-$$ && init.d/scst >$tmp && \
install -m 755 $tmp /etc/init.d/scst && \
rm -f $tmp
if [ ! -e /etc/default/scst ]; then \
mkdir -p /etc/default; \
install -m 755 default/scst /etc/default; \
fi
for s in iscsi-scst qla2x00t; do \
{ if [ -n "" ]; then chr="chroot "; else chr=""; fi; if type systemctl >/dev/null 2>&1; then $chr systemctl disable "$s.service"; elif type chkconfig >/dev/null 2>&1; then $chr chkconfig --del "$s"; elif type update-rc.d >/dev/null 2>&1; then $chr update-rc.d -f "$s" remove; elif type rc-update >/dev/null 2>&1; then $chr rc-update del "$s"; elif type /usr/lib/lsb/remove_initd >/dev/null 2>&1; then $chr /usr/lib/lsb/remove_initd "$s"; fi; rm -f /etc/init.d/$s; } >/dev/null 2>&1; \
done
echo
echo 'If you want SCST to start automatically at boot time, run' \
'the following command:'
If you want SCST to start automatically at boot time, run the following command:
if [ -n "" ]; then chr="chroot "; else chr=""; fi; if type systemctl >/dev/null 2>&1; then echo $chr systemctl enable "scst.service"; elif type chkconfig >/dev/null 2>&1; then echo $chr chkconfig --add "scst"; elif type update-rc.d >/dev/null 2>&1; then echo $chr update-rc.d "scst" defaults; elif type rc-update >/dev/null 2>&1; then echo $chr rc-update add "scst" default; elif type /usr/lib/lsb/install_initd >/dev/null 2>&1; then echo $chr /usr/lib/lsb/install_initd "scst"; fi
chkconfig --add scst
echo
make[1]: Leaving directory `/home/mockbuild/scst/scstadmin'
[root@oracle651 scst]#
Now set SCST to start on boot. Do this as root. Commands and expected output are shown below.
[root@oracle651 scst]# /usr/lib/lsb/install_initd scst
[root@oracle651 scst]#
Now test start SCST manually. There is no configuration file yet for SCST, so it will give a warning as shown below.
[root@oracle651 scst]# service scst start
Loading and configuring SCSTSCST configuration file /etc/scst.conf missing
[ OK ]
[root@oracle651 scst]#
Now lock down the kernel so that the custom kernel just built will not be updated. This requires updated the /etc/yum.conf file as shown below. The line that needs to be added is shown in bold below "exclude=kernel*". This prevents the kernel from being updated.
[root@oracle651 yum.repos.d]# cat /etc/yum.conf
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=3
exclude=kernel*
# This is the default, if you make this bigger yum won't see if the metadata
# is newer on the remote and so you'll "gain" the bandwidth of not having to
# download the new metadata and "pay" for it by yum not having correct
# information.
# It is esp. important, to have correct metadata, for distributions like
# Fedora which don't keep old packages around. If you don't like this checking
# interupting your command line usage, it's much better to have something
# manually check the metadata once an hour (yum-updatesd will do this).
# metadata_expire=90m
# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d
[root@oracle651 yum.repos.d]#
Reboot and verify that SCST starts up automatically and that all targets, groups, and LUNs are active as shown below.
[root@oracle651 ~]# uname -a
Linux oracle651.vmem.org 3.8.13-16.2.1.el6uek.x86_64 #1 SMP Sun Aug 24 11:05:09 CDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@oracle651 ~]# service scst status
SCST status: OK
[root@oracle651 ~]# ps -ef | grep scst
root 1811 2 1 11:35 ? 00:00:00 [scst_uid]
root 1814 2 0 11:35 ? 00:00:00 [scstd0]
root 1815 2 0 11:35 ? 00:00:00 [scstd1]
root 1816 2 0 11:35 ? 00:00:00 [scstd2]
root 1817 2 0 11:35 ? 00:00:00 [scstd3]
root 1818 2 0 11:35 ? 00:00:00 [scstd4]
root 1819 2 0 11:35 ? 00:00:00 [scstd5]
root 1820 2 0 11:35 ? 00:00:00 [scstd6]
root 1821 2 0 11:35 ? 00:00:00 [scstd7]
root 1822 2 0 11:35 ? 00:00:00 [scst_initd]
root 1824 2 0 11:35 ? 00:00:00 [scst_mgmtd]
root 1873 1 0 11:35 ? 00:00:00 /usr/local/sbin/iscsi-scstd
root 2532 2489 0 11:36 pts/0 00:00:00 grep scst
[root@oracle651 ~]# scstadmin -list_group
Collecting current configuration: done.
Driver: iscsi
Target: iqn.2014-08.org.vmem:oracle651.san.asm.luns
Driver/target 'iscsi/iqn.2014-08.org.vmem:oracle651.san.asm.luns' has no associated LUNs.
Group: oracle631
Assigned LUNs:
LUN Device
-----------------
0 AsmSys631_01
1 AsmDat631_01
10 AsmRec631_02
11 AsmRec631_03
12 AsmRec631_04
13 AsmRec631_05
14 AsmRec631_06
15 AsmRec631_07
16 AsmRec631_08
2 AsmDat631_02
20 AsmSlb631_01
21 AsmSlb631_02
22 AsmSlb631_03
23 AsmSlb631_04
24 AsmSlb631_05
25 AsmSlb631_06
26 AsmSlb631_07
27 AsmSlb631_08
3 AsmDat631_03
4 AsmDat631_04
5 AsmDat631_05
6 AsmDat631_06
7 AsmDat631_07
8 AsmDat631_08
9 AsmRec631_01
Assigned Initiators:
Initiator
----------------------------------
iqn.2014-08.org.vmem:oracle631.asm
Group: oracle632
Assigned LUNs:
LUN Device
-----------------
0 AsmSys632_01
1 AsmDat632_01
10 AsmRec632_02
11 AsmRec632_03
12 AsmRec632_04
13 AsmRec632_05
14 AsmRec632_06
15 AsmRec632_07
16 AsmRec632_08
17 AsmTmp632_01
18 AsmTmp632_02
19 AsmTmp632_03
2 AsmDat632_02
20 AsmTmp632_04
21 AsmTmp632_05
22 AsmTmp632_06
23 AsmTmp632_07
24 AsmTmp632_08
25 AsmSlb632_01
26 AsmSlb632_02
27 AsmSlb632_03
28 AsmSlb632_04
29 AsmSlb632_05
3 AsmDat632_03
30 AsmSlb632_06
31 AsmSlb632_07
32 AsmSlb632_08
4 AsmDat632_04
5 AsmDat632_05
6 AsmDat632_06
7 AsmDat632_07
8 AsmDat632_08
9 AsmRec632_01
Assigned Initiators:
Initiator
----------------------------------
iqn.2014-08.org.vmem:oracle632.asm
All done.
[root@oracle651 ~]#
Configure DTrace
In ULN, enable the required repos as shown in the screenshot below. Save the ULN subscriptions.
Perform a yum update as shown below.
gstanden@vmem1:~$ ssh root@10.207.39.74
root@10.207.39.74's password:
Last login: Sat Aug 23 17:47:51 2014
[root@oracle651 ~]# clear
[root@oracle651 ~]# yum update
Loaded plugins: refresh-packagekit, rhnplugin, security
This system is receiving updates from ULN.
ol6_x86_64_Dtrace_BETA | 1.2 kB 00:00
ol6_x86_64_Dtrace_BETA/primary | 1.1 MB 00:01
ol6_x86_64_Dtrace_BETA 45/45
ol6_x86_64_Dtrace_latest | 1.2 kB 00:00
ol6_x86_64_Dtrace_userspace_latest | 1.2 kB 00:00
ol6_x86_64_Dtrace_userspace_latest/primary | 2.0 kB 00:00
ol6_x86_64_Dtrace_userspace_latest 8/8
ol6_x86_64_UEKR3_latest | 1.2 kB 00:00
ol6_x86_64_ksplice | 1.2 kB 00:00
ol6_x86_64_latest | 1.4 kB 00:00
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package dtrace-utils.x86_64 0:0.3.2-1.el6 will be updated
---> Package dtrace-utils.x86_64 0:0.4.4-2.el6 will be an update
--> Processing Dependency: dtrace-modules-headers for package: dtrace-utils-0.4.4-2.el6.x86_64
---> Package glibc.i686 0:2.12-1.132.el6_5.2 will be updated
---> Package glibc.x86_64 0:2.12-1.132.el6_5.2 will be updated
---> Package glibc.i686 0:2.12-1.132.el6_5.3 will be an update
---> Package glibc.x86_64 0:2.12-1.132.el6_5.3 will be an update
---> Package glibc-common.x86_64 0:2.12-1.132.el6_5.2 will be updated
---> Package glibc-common.x86_64 0:2.12-1.132.el6_5.3 will be an update
---> Package glibc-devel.x86_64 0:2.12-1.132.el6_5.2 will be updated
---> Package glibc-devel.x86_64 0:2.12-1.132.el6_5.3 will be an update
---> Package glibc-headers.x86_64 0:2.12-1.132.el6_5.2 will be updated
---> Package glibc-headers.x86_64 0:2.12-1.132.el6_5.3 will be an update
---> Package grubby.x86_64 0:7.0.15-5.0.2.el6 will be updated
---> Package grubby.x86_64 0:7.0.15-5.0.4.el6 will be an update
---> Package hwloc.x86_64 0:1.5-1.el6 will be updated
---> Package hwloc.x86_64 0:1.5-2.el6_5 will be an update
---> Package openssl.x86_64 0:1.0.1e-16.el6_5.14 will be updated
---> Package openssl.x86_64 0:1.0.1e-16.el6_5.15 will be an update
---> Package openssl-devel.x86_64 0:1.0.1e-16.el6_5.14 will be updated
---> Package openssl-devel.x86_64 0:1.0.1e-16.el6_5.15 will be an update
--> Running transaction check
---> Package dtrace-modules-3.8.13-16.2.1.el6uek-headers.x86_64 0:0.4.1-3.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==========================================================================================================================================================================
Package Arch Version Repository Size
==========================================================================================================================================================================
Updating:
dtrace-utils x86_64 0.4.4-2.el6 ol6_x86_64_Dtrace_userspace_latest 262 k
glibc i686 2.12-1.132.el6_5.3 ol6_x86_64_latest 4.3 M
glibc x86_64 2.12-1.132.el6_5.3 ol6_x86_64_latest 3.8 M
glibc-common x86_64 2.12-1.132.el6_5.3 ol6_x86_64_latest 14 M
glibc-devel x86_64 2.12-1.132.el6_5.3 ol6_x86_64_latest 977 k
glibc-headers x86_64 2.12-1.132.el6_5.3 ol6_x86_64_latest 608 k
grubby x86_64 7.0.15-5.0.4.el6 ol6_x86_64_latest 43 k
hwloc x86_64 1.5-2.el6_5 ol6_x86_64_latest 1.4 M
openssl x86_64 1.0.1e-16.el6_5.15 ol6_x86_64_latest 1.5 M
openssl-devel x86_64 1.0.1e-16.el6_5.15 ol6_x86_64_latest 1.2 M
Installing for dependencies:
dtrace-modules-3.8.13-16.2.1.el6uek-headers x86_64 0.4.1-3.el6 ol6_x86_64_UEKR3_latest 27 k
Transaction Summary
==========================================================================================================================================================================
Install 1 Package(s)
Upgrade 10 Package(s)
Total download size: 28 M
Is this ok [y/N]:
Confirm yes and install the updates. The successful completion message is shown below.
Dependency Installed:
dtrace-modules-3.8.13-16.2.1.el6uek-headers.x86_64 0:0.4.1-3.el6
Updated:
dtrace-utils.x86_64 0:0.4.4-2.el6 glibc.i686 0:2.12-1.132.el6_5.3 glibc.x86_64 0:2.12-1.132.el6_5.3 glibc-common.x86_64 0:2.12-1.132.el6_5.3
glibc-devel.x86_64 0:2.12-1.132.el6_5.3 glibc-headers.x86_64 0:2.12-1.132.el6_5.3 grubby.x86_64 0:7.0.15-5.0.4.el6 hwloc.x86_64 0:1.5-2.el6_5
openssl.x86_64 0:1.0.1e-16.el6_5.15 openssl-devel.x86_64 0:1.0.1e-16.el6_5.15
Complete!
[root@oracle651 ~]#
All that remains now is to reboot and test that it starts. Also, a configuration file must be created, which will be the next step. There was not a lot of documentation on the web about how to create a iSCSI SAN with SCST but I found a couple of very helpful posts here and here.
Configuration of a simple 4K flash-aware SCST iSCSI SAN as shown below on the oracle651 KVM guest.
[root@oracle651 ~]# modprobe scst
[root@oracle651 ~]# modprobe scst_vdisk
[root@oracle651 ~]# modprobe iscsi-scst
[root@oracle651 ~]# scstadmin -add_target iqn.2014-08.oracle.asm.storage:luns -driver iscsi
[root@oracle651 ~]# scstadmin -open_dev disk0 -handler vdisk_blockio -attributes filename=/dev/vdc,blocksize=4096
[root@oracle651 ~]# scstadmin -open_dev disk1 -handler vdisk_blockio -attributes filename=/dev/vdb
[root@oracle651 ~]# scstadmin -add_lun 0 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk0
[root@oracle651 ~]# scstadmin -add_lun 1 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk1
[root@oracle651 ~]# scstadmin -enable_target iqn.2014-08.oracle.asm.storage:luns -driver iscs
[root@oracle651 ~]# scstadmin -set_drv_attr iscsi -attributes enabled=1
[root@oracle651 ~]# scstadmin -write_config /etc/scst.conf
The /etc/scst.conf file generated by the above command is shown below. Notice the "blocksize 4096" parameter which can be used.
[root@oracle651 ~]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v2.2.1.
HANDLER vdisk_blockio {
DEVICE disk0 {
blocksize 4096
filename /dev/vdc
}
DEVICE disk1 {
filename /dev/vdb
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2014-08.oracle.asm.storage:luns {
cpu_mask 3
enabled 1
rel_tgt_id 1
LUN 0 disk0
LUN 1 disk1
}
}
[root@oracle651 ~]# cat /etc/sysconfig/selinux
In order to get the initiator over on the iSCSI client on oracle631 KVM guest to see the portal, stop iptables and disable it on reboot as shown below and also disable selinux as shown below and turn off iptables on the iSCSI SAN host.
[root@oracle651 ~]# cat /etc/sysconfig/selinux
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
[root@oracle651 ~]# service iptables stop
[root@oracle651 ~]# chkconfig iptables off
[root@oracle651 ~]# reboot
Finally, use iscsiadm on the oracle631 KVM guest to discover and configure the access to the SCST SAN on oracle651 KVM guest. The iscsi-initiator utils are installed as shown below, and then the connection to the SAN is configured as shown.
[root@oracle631 ~]# yum install iscsi-initiator-utils
[root@oracle631 ~]# iscsiadm --mode discovery --type sendtargets --portal 10.207.39.74
10.207.39.74:3260,1 iqn.2014-08.oracle.asm.storage:luns
[root@oracle631 ~]# iscsiadm -m node -T iqn.2014-08.oracle.asm.storage:luns -p 10.207.39.74 --op update -n node.startup -v automatic
[root@oracle631 ~]# iscsiadm --mode node --targetname iqn.2014-08.oracle.asm.storage:luns --portal 10.207.39.74 --login
Logging in to [iface: default, target: iqn.2014-08.oracle.asm.storage:luns, portal: 10.207.39.74,3260] (multiple)
Login to [iface: default, target: iqn.2014-08.oracle.asm.storage:luns, portal: 10.207.39.74,3260] successful.
[root@oracle631 ~]#
Here are the LUNs over on the client side and show of their characteristics on oracle631 host as shown below. One very interesting and intriguing feature is that SCST sees the 512 emulated LUN on oracle631 KVM guest (the client) for what it really is i.e (512 logical / 4096 physical) even though the virtual disk on the oracle651 KVM guest (the SAN) sees that LUN as (512 logical / 512 physical), as shown below.
[root@oracle631 ~]# ls -lrt /dev/sd*
brw-rw---- 1 root disk 8, 0 Aug 11 20:11 /dev/sda
brw-rw---- 1 root disk 8, 16 Aug 11 20:11 /dev/sdb
[root@oracle631 ~]# scsi_id -g -u -d /dev/sda
26162356632353435
[root@oracle631 ~]# scsi_id -g -u -d /dev/sdb
26238666462643332
[root@oracle631 ~]# fdisk -l /dev/sda
Note: sector size is 4096 (not 512)
Disk /dev/sda: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 1280 cylinders
Units = cylinders of 2048 * 4096 = 8388608 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 524288 bytes
Disk identifier: 0x00000000
[root@oracle631 ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes <-- Notice
I/O size (minimum/optimal): 4096 bytes / 524288 bytes
Disk identifier: 0x00000000
[root@oracle631 ~]#
Here are the properties of the same LUNs over on the SAN oracle651 as shown below.
[root@oracle651 ~]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v2.2.1.
HANDLER vdisk_blockio {
DEVICE disk0 {
blocksize 4096
filename /dev/vdc
}
DEVICE disk1 {
filename /dev/vdb
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2014-08.oracle.asm.storage:luns {
cpu_mask 3
enabled 1
rel_tgt_id 1
LUN 0 disk0
LUN 1 disk1
}
}
[root@oracle651 ~]# fdisk -l /dev/vdc
Note: sector size is 4096 (not 512)
Disk /dev/vdc: 10.7 GB, 10737418240 bytes
16 heads, 56 sectors/track, 2925 cylinders
Units = cylinders of 896 * 4096 = 3670016 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
[root@oracle651 ~]# fdisk -l /dev/vdb
Disk /dev/vdb: 10.7 GB, 10737418240 bytes
16 heads, 63 sectors/track, 20805 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes <-- Notice
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
[root@oracle651 ~]#
Recall that the blocksize defaults in KVM to (logical/physical) 512 bytes / 512 bytes unless it is set otherwise in the XML file of the KVM guest, as shown below i the XML file that defines oracle651 the KVM guest which is the SAN provider. Looking at the end of the XML file at bottom it can be seen that only disk2 has been set to specified blocksizes. Therefore, disk1 is presented to the KVM guest oracle651 as a 512 bytes / 512 bytes LUN by default. Thus it is surprising that SCST sees it for what it is 'physically' which is a LUN sitting on a consumer-grade SSD (Crucial M550) and flash always uses 4096 byte physical sectors, and 512 byte logical emulation. The LUN as seen by oracle631 the iscsi client sees it as what it 'really' is, 512 bytes / 4096 bytes (logical/physical). More investigation is warranted to understand this better, but it would seem that SCST is ignoring the KVM setting and reading the storage itself, and that the blocksize setting in the SCST configuration file refers to the logical block size, and thus set the logical block size of the other LUN to 4096 bytes (the LUN that was presented by KVM on oracle631 as (logical/physical) 4096 bytes / 4096 bytes.
gstanden@vmem1:~$ virsh dumpxml oracle651
<domain type='kvm' id='6' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>oracle651</name>
<uuid>9a017de6-d362-4269-90fe-09331460aca0</uuid>
<memory unit='KiB'>1048576</memory>
<currentMemory unit='KiB'>1048576</currentMemory>
<vcpu placement='static'>2</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
<boot dev='hd'/>
<boot dev='cdrom'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/var/lib/libvirt/images/oracle651-0.img'/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/var/lib/libvirt/images/oracle651-1.img'/>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/var/lib/libvirt/images/oracle651-2.img'/>
<target dev='vdc' bus='virtio'/>
<alias name='virtio-disk2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</disk>
<disk type='block' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='hdc' bus='ide' tray='open'/>
<readonly/>
<alias name='ide0-1-0'/>
<address type='drive' controller='0' bus='1' target='0' unit='0'/>
</disk>
<controller type='usb' index='0'>
<alias name='usb0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:66:26:a0'/>
<source bridge='sw1'/>
<virtualport type='openvswitch'>
<parameters interfaceid='0efa5f2a-cbf2-46df-9ff2-66bddb09218d'/>
</virtualport>
<target dev='s4'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/12'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/12'>
<source path='/dev/pts/12'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
<listen type='address' address='127.0.0.1'/>
</graphics>
<sound model='ich6'>
<alias name='sound0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</sound>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</memballoon>
</devices>
<seclabel type='dynamic' model='apparmor' relabel='yes'>
<label>libvirt-9a017de6-d362-4269-90fe-09331460aca0</label>
<imagelabel>libvirt-9a017de6-d362-4269-90fe-09331460aca0</imagelabel>
</seclabel>
<qemu:commandline>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk0.scsi=off'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk0.config-wce=off'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk0.x-data-plane=on'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk1.scsi=off'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk1.config-wce=off'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk1.x-data-plane=on'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk2.scsi=off'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk2.config-wce=off'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk2.x-data-plane=on'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk2.logical_block_size=4096'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.virtio-disk2.physical_block_size=4096'/>
</qemu:commandline>
</domain>
gstanden@vmem1:~$
Some useful additonal info about the LUNs over on the oracle651 client.
[root@oracle631 disk]# ls -lrt by-path | grep asm
lrwxrwxrwx 1 root root 9 Aug 11 20:11 ip-10.207.39.74:3260-iscsi-iqn.2014-08.oracle.asm.storage:luns-lun-0 -> ../../sda
lrwxrwxrwx 1 root root 9 Aug 11 20:11 ip-10.207.39.74:3260-iscsi-iqn.2014-08.oracle.asm.storage:luns-lun-1 -> ../../sdb
[root@oracle631 disk]# ls -lrt by-id | grep scsi
lrwxrwxrwx 1 root root 9 Aug 11 20:11 scsi-26162356632353435 -> ../../sda
lrwxrwxrwx 1 root root 9 Aug 11 20:11 scsi-26238666462643332 -> ../../sdb
[root@oracle631 disk]#
More useful information, including the vendor name (for use in the multipath.conf file) as shown below.
[root@oracle631 disk]# cat /proc/scsi/scsi
Attached devices:
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: QEMU Model: QEMU DVD-ROM Rev: 2.0.
Type: CD-ROM ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
Vendor: SCST_BIO Model: disk0 Rev: 221
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 01
Vendor: SCST_BIO Model: disk1 Rev: 221
Type: Direct-Access ANSI SCSI revision: 05
[root@oracle631 disk]#
And yet more information about the SCST iSCSI LUN /dev/sda shown below using udevadm command.
[root@oracle631 disk]# udevadm info -a -p $(udevadm info -q path -n /dev/sda)
Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.
looking at device '/devices/platform/host2/session1/target2:0:0/2:0:0:0/block/sda':
KERNEL=="sda"
SUBSYSTEM=="block"
DRIVER==""
ATTR{ro}=="0"
ATTR{size}=="20971520"
ATTR{stat}==" 455 0 3648 329 0 0 0 0 0 328 328"
ATTR{range}=="16"
ATTR{discard_alignment}=="0"
ATTR{events}==""
ATTR{ext_range}=="256"
ATTR{events_poll_msecs}=="-1"
ATTR{alignment_offset}=="0"
ATTR{inflight}==" 0 0"
ATTR{removable}=="0"
ATTR{capability}=="50"
ATTR{events_async}==""
looking at parent device '/devices/platform/host2/session1/target2:0:0/2:0:0:0':
KERNELS=="2:0:0:0"
SUBSYSTEMS=="scsi"
DRIVERS=="sd"
ATTRS{rev}==" 221"
ATTRS{type}=="0"
ATTRS{scsi_level}=="6"
ATTRS{model}=="disk0 "
ATTRS{state}=="running"
ATTRS{queue_type}=="none"
ATTRS{modalias}=="scsi:t-0x00"
ATTRS{iodone_cnt}=="0x212"
ATTRS{iorequest_cnt}=="0x212"
ATTRS{queue_ramp_up_period}=="120000"
ATTRS{timeout}=="30"
ATTRS{evt_media_change}=="0"
ATTRS{ioerr_cnt}=="0x2"
ATTRS{queue_depth}=="32"
ATTRS{vendor}=="SCST_BIO"
ATTRS{device_blocked}=="0"
ATTRS{dh_state}=="detached"
ATTRS{iocounterbits}=="32"
ATTRS{eh_timeout}=="10"
looking at parent device '/devices/platform/host2/session1/target2:0:0':
KERNELS=="target2:0:0"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/platform/host2/session1':
KERNELS=="session1"
SUBSYSTEMS==""
DRIVERS==""
looking at parent device '/devices/platform/host2':
KERNELS=="host2"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/platform':
KERNELS=="platform"
SUBSYSTEMS==""
DRIVERS==""
And now for /dev/sdb shown below using udevadm command.
[root@oracle631 disk]# udevadm info -a -p $(udevadm info -q path -n /dev/sdb)
Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.
looking at device '/devices/platform/host2/session1/target2:0:0/2:0:0:1/block/sdb':
KERNEL=="sdb"
SUBSYSTEM=="block"
DRIVER==""
ATTR{ro}=="0"
ATTR{size}=="20971520"
ATTR{stat}==" 455 0 3664 337 0 0 0 0 0 337 337"
ATTR{range}=="16"
ATTR{discard_alignment}=="0"
ATTR{events}==""
ATTR{ext_range}=="256"
ATTR{events_poll_msecs}=="-1"
ATTR{alignment_offset}=="0"
ATTR{inflight}==" 0 0"
ATTR{removable}=="0"
ATTR{capability}=="50"
ATTR{events_async}==""
looking at parent device '/devices/platform/host2/session1/target2:0:0/2:0:0:1':
KERNELS=="2:0:0:1"
SUBSYSTEMS=="scsi"
DRIVERS=="sd"
ATTRS{rev}==" 221"
ATTRS{type}=="0"
ATTRS{scsi_level}=="6"
ATTRS{model}=="disk1 "
ATTRS{state}=="running"
ATTRS{queue_type}=="none"
ATTRS{modalias}=="scsi:t-0x00"
ATTRS{iodone_cnt}=="0x211"
ATTRS{iorequest_cnt}=="0x211"
ATTRS{queue_ramp_up_period}=="120000"
ATTRS{timeout}=="30"
ATTRS{evt_media_change}=="0"
ATTRS{ioerr_cnt}=="0x2"
ATTRS{queue_depth}=="32"
ATTRS{vendor}=="SCST_BIO"
ATTRS{device_blocked}=="0"
ATTRS{dh_state}=="detached"
ATTRS{iocounterbits}=="32"
ATTRS{eh_timeout}=="10"
looking at parent device '/devices/platform/host2/session1/target2:0:0':
KERNELS=="target2:0:0"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/platform/host2/session1':
KERNELS=="session1"
SUBSYSTEMS==""
DRIVERS==""
looking at parent device '/devices/platform/host2':
KERNELS=="host2"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/platform':
KERNELS=="platform"
SUBSYSTEMS==""
DRIVERS==""
Additional information after the configuration of the SCST LUNs on oracle651 using device-mapper-multipath and /etc/udev/rules.d/12-dm-permissions.rules as shown below. Friendly names set from the /etc/multipath.conf file correctly.
[root@oracle631 ~]# ls -lrt /dev/mapper
total 0
crw-rw---- 1 root root 10, 236 Aug 11 21:30 control
lrwxrwxrwx 1 root root 7 Aug 11 21:30 vg_oracle631-lv_swap -> ../dm-1
lrwxrwxrwx 1 root root 7 Aug 11 21:30 vg_oracle631-lv_root -> ../dm-0
lrwxrwxrwx 1 root root 7 Aug 11 21:30 vg_u00-lv_oracle -> ../dm-2
lrwxrwxrwx 1 root root 7 Aug 11 21:30 ASM_4096B_1 -> ../dm-3
lrwxrwxrwx 1 root root 7 Aug 11 21:30 ASM_0512B_1 -> ../dm-4
Friendly names accurately indicate logical block sizes as shown below.
[root@oracle631 ~]# fdisk -l /dev/dm-3 <--ASM_4096B_1
Note: sector size is 4096 (not 512)
Disk /dev/dm-3: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 163 cylinders
Units = cylinders of 16065 * 4096 = 65802240 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes <-- 4096 logical sector size
I/O size (minimum/optimal): 4096 bytes / 524288 bytes
Disk identifier: 0x00000000
[root@oracle631 ~]# fdisk -l /dev/dm-4
Disk /dev/dm-4: 10.7 GB, 10737418240 bytes <--ASM_0512B_1
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes <-- 512 logical sector size
I/O size (minimum/optimal): 4096 bytes / 524288 bytes
Disk identifier: 0x00000000
[root@oracle631 ~]#
Permissions set correctly from the /etc/udev/rules.d/12-dm-permissions.rules file.
[root@oracle631 ~]# ls -lrt /dev/dm-3
brw-rw---- 1 grid asmadmin 252, 3 Aug 11 21:30 /dev/dm-3
[root@oracle631 ~]# ls -lrt /dev/dm-4
brw-rw---- 1 grid asmadmin 252, 4 Aug 11 21:30 /dev/dm-4
The /etc/multipath.conf file for the SCST LUNs on oracle631 as shown below.
[root@oracle631 ~]# cat /etc/multipath.conf
# Configuration file for device mapper multipath.
# For a complete list of the default configuration values:
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults
# For a list of configuration options with descriptions:
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotated
defaults {
user_friendly_names yes
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
}
devices {
device {
vendor "SCST_BIO"
product "Direct-Access"
path_grouping_policy multibus
path_grouping_policy group_by_prio
path_selector "round-robin 0"
failback immediate
rr_min_io 1
rr_weight priorities
}
}
multipaths {
multipath {
wwid 26162356632353435
alias ASM_4096B_1
}
multipath {
wwid 26238666462643332
alias ASM_0512B_1
}
}
[root@oracle631 ~]#
Listing of the /etc/udev/rules.d/12-dm-permissions.rules file on oracle631 the scsi client as shown below with section that handles the above LUNs bolded (near bottom of listing).
[root@oracle631 ~]# cat /etc/udev/rules.d/12-dm-permissions.rules
# Copyright (C) 2009 Red Hat, Inc. All rights reserved.
#
# This file is part of LVM2.
# Udev rules for device-mapper devices.
#
# These rules set permissions for DM devices.
#
# This file is considered to be a template where users can put their
# own entries and then put a copy of it manually to a usual place with
# user-edited udev rules (usually /etc/udev/rules.d).
#
# There are some environment variables set that can be used:
# DM_UDEV_RULES_VSN - DM udev rules version
# DM_NAME - actual DM device's name
# DM_UUID - UUID set for DM device (blank if not specified)
# DM_SUSPENDED - suspended state of DM device (0 or 1)
# DM_LV_NAME - logical volume name (not set if LVM device not present)
# DM_VG_NAME - volume group name (not set if LVM device not present)
# DM_LV_LAYER - logical volume layer (not set if LVM device not present)
# "add" event is processed on coldplug only!
ACTION!="add|change", GOTO="dm_end"
ENV{DM_UDEV_RULES_VSN}!="?*", GOTO="dm_end"
# A few demonstrational examples...
# PLAIN DM DEVICES
#
# Set permissions for a DM device named 'my_device' exactly
# ENV{DM_NAME}=="my_device", OWNER:="root", GROUP:="root", MODE:="660"
# Set permissions for all DM devices having 'MY_UUID-' UUID prefix
# ENV{DM_UUID}=="MY_UUID-?*", OWNER:="root", GROUP:="root", MODE:="660"
# LVM DEVICES
#
# Set permissions for all LVM devices
# ENV{DM_UUID}=="LVM-?*", OWNER:="root", GROUP:="root", MODE:="660"
# Set permissions for all devices that belong to one LVM VG
# ENV{DM_VG_NAME}=="VolGroup00", OWNER:="root", GROUP:="root", MODE:="660"
# Set permissions for an LVM device with VG named VolGroup00 and LV named LogVol00 exactly
# ENV{DM_VG_NAME}=="VolGroup00", ENV{DM_LV_NAME}=="LogVol00", OWNER:="root", GROUP:="root", MODE:="660"
# Set permissions for all LVM devices that does not belong to a VG named VolGroup00
# ENV{DM_VG_NAME}!="VolGroup00", OWNER:="root", GROUP:="root", MODE:="660"
# ENCRYPTED DEVICES (using cryptsetup >= 1.1)
#
# Set permissions for all encrypted devices created by cryptsetup (plain devices)
# ENV{DM_UUID}=="CRYPT-PLAIN-?*", OWNER:="root", GROUP:="root", MODE:="660"
# Set permissions for all encrypted devices created by cryptsetup (LUKS extension)
# ENV{DM_UUID}=="CRYPT-LUKS1-?*", OWNER:="root", GROUP:="root", MODE:="660"
# Set permissions for an encrypted device created by cryptsetup and having an exact luks UUID
# ENV{DM_UUID}=="CRYPT-LUKS1-22fce5c8313c43c68d84b50a3b0fee78-?*", OWNER:="root", GROUP:="root", MODE:="660"
# MULTIPATH DEVICES
#
# Set permissions for all multipath devices
# ENV{DM_UUID}=="mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"
ENV{DM_UUID}=="mpath-26162356632353435", OWNER:="grid", GROUP:="asmadmin", MODE:="660" # <-- LUN modes are set here
ENV{DM_UUID}=="mpath-26238666462643332", OWNER:="grid", GROUP:="asmadmin", MODE:="660" # <-- LUN modes are set here
# Set permissions for first two partitions created on a multipath device (and detected by kpartx)
# ENV{DM_UUID}=="part[1-2]-mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"
# ...you can use any combination of the comparisons with the environment variables
# listed at the beginning of this file (udev provides simple pattern matching by
# using *, ? and [] that you can use, see 'man udev' for more information).
# Set default permissions for all DM devices if not set before.
# OWNER:="root", GROUP:="root", MODE:="660"
LABEL="dm_end"
[root@oracle631 ~]#
If the console terminal is running on the KVM guest SCST SAN host oracle651, then when a SAN client oracle631 is shutdown, the following message is buffered to console and written to dmesg as shown below.
[root@oracle651 ~]# iscsi-scst: Logout received from initiator iqn.1988-12.com.oracle:49c39d4d389d
iscsi-scst: Closing connection at initiator's iqn.1988-12.com.oracle:49c39d4d389d request
scst: TM fn 6 (ffff88003dfd50b8)
scst: TM fn 6 (ffff88003dfd50b8) finished, status 0
scst: Processing thread disk00_0 (PID 1783) finished
scst: Processing thread disk10_0 (PID 1789) finished
[root@oracle651 ~]# exit
logout
Oracle Linux Server release 6.5
Kernel 2.6.32-scst on an x86_64
oracle651.vmem.org login:
Here is an example of the commands for adding a file-backed LUN rather than a block device as shown below.
[root@oracle651 ~] dd if=/dev/zero of=/root/scst-scripts/asm_disk1.img bs=512 count=0 seek=1M
[root@oracle651 ~] scstadmin -open_dev disk2 -handler vdisk_fileio -attributes filename=/root/scst-scripts/asm_disk1.img,blocksize=4096
[root@oracle651 ~] scstadmin -add_lun 2 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk2
[root@oracle651 ~] scstadmin -write_config /etc/scst.conf
The last command saves it to the /etc/scst.conf configuration file as shown below. The "vdisk_fileio" handler has been added, as well as the LUN.
[root@oracle651 scst-scripts]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v2.2.1.
HANDLER vdisk_blockio {
DEVICE disk0 {
blocksize 4096
filename /dev/vdc
}
DEVICE disk1 {
filename /dev/vdb
}
}
HANDLER vdisk_fileio {
DEVICE disk2 {
blocksize 4096
filename /root/scst-scripts/asm_disk1.img
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2014-08.oracle.asm.storage:luns {
cpu_mask 3
enabled 1
rel_tgt_id 1
LUN 0 disk0
LUN 1 disk1
LUN 2 disk2
}
}
[root@oracle651 scst-scripts]#
Here is an example of removing the LUN that was just added. Notice that the HANDLER information for the deleted LUN remains, but the TARGET DRIVER LUN list is updated and the removed LUN has been removed from the list in /etc/scst.conf as shown below.
[root@oracle651 scst-scripts]# scstadmin -rem_lun 2 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk2
Collecting current configuration: done.
Performing this action may result in lost or corrupt data, are you sure you wish to continue (y/[n]) ? y
-> Making requested changes.
-> Removing LUN 2 from driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
Writing current configuration to file '/etc/scst.conf'
All done.
[root@oracle651 scst-scripts]# scstadmin -write_config /etc/scst.conf
Collecting current configuration: done.
Writing current configuration to file '/etc/scst.conf'..
All done.
[root@oracle651 scst-scripts]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v2.2.1.
HANDLER vdisk_blockio {
DEVICE disk0 {
blocksize 4096
filename /dev/vdc
}
DEVICE disk1 {
filename /dev/vdb
}
}
HANDLER vdisk_fileio { <-- disk2 handler is still defined after LUN 2 was dropped from target.
DEVICE disk2 {
blocksize 4096
filename /root/scst-scripts/asm_disk1.img
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2014-08.oracle.asm.storage:luns {
cpu_mask 3
enabled 1
rel_tgt_id 1
LUN 0 disk0
LUN 1 disk1
} <-- LUN 2 disk2 is removed here.
}
[root@oracle651 scst-scripts]# history
Thus, the LUN can be re-added back with just the one command below if it is needed back in the target as shown below.
[root@oracle651 ~]# scstadmin -add_lun 2 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk2
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'disk2' at LUN 2 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -write_config /etc/scst.conf
Collecting current configuration: done.
Writing current configuration to file '/etc/scst.conf'..
All done.
[root@oracle651 ~]#
View /etc/scst.conf again and see that the LUN has been added back to the TARGET as shown below.
[root@oracle651 ~]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v2.2.1.
HANDLER vdisk_blockio {
DEVICE disk0 {
blocksize 4096
filename /dev/vdc
}
DEVICE disk1 {
filename /dev/vdb
}
}
HANDLER vdisk_fileio {
DEVICE disk2 {
blocksize 4096
filename /root/scst-scripts/asm_disk1.img
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2014-08.oracle.asm.storage:luns {
cpu_mask 3
enabled 1
rel_tgt_id 1
LUN 0 disk0
LUN 1 disk1
LUN 2 disk2
}
}
[root@oracle651 ~]#
Examples of adding file-backed SCST LUNs to the OS of the SCST OEL host are shown below. The example below shows adding the 8 LUNs per the Violin Memory Oracle Best Practice Document dated April 2014 to the SCST SAN configuration.
[root@oracle651 ~]# scstadmin -add_lun 11 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco1
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco1' at LUN 11 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -add_lun 12 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco2
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco2' at LUN 12 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -add_lun 13 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco3
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco3' at LUN 13 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -add_lun 14 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco4
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco4' at LUN 14 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -add_lun 15 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco5
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco5' at LUN 15 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -add_lun 16 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco6
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco6' at LUN 16 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -add_lun 17 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco7
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco7' at LUN 17 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]# scstadmin -add_lun 18 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco8
Collecting current configuration: done.
-> Making requested changes.
-> Adding device 'asm_reco8' at LUN 18 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.
-> Driver/target is not a fibre channel target, ignoring.
-> Done.
All done.
[root@oracle651 ~]#
Now write the configuration changes to the /etc/scst.conf configuration file as shown below.
[root@oracle651 ~]# scstadmin -write_config /etc/scst.conf
Collecting current configuration: done.
Writing current configuration to file '/etc/scst.conf'..
All done.
[root@oracle651 ~]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v2.2.1.
HANDLER vdisk_blockio {
DEVICE disk0 {
blocksize 4096
filename /dev/vdc
}
DEVICE disk1 {
filename /dev/vdb
}
}
HANDLER vdisk_fileio {
DEVICE asm_data1 {
filename /scst/asm_data1.img
}
DEVICE asm_data2 {
filename /scst/asm_data2.img
}
DEVICE asm_data3 {
filename /scst/asm_data3.img
}
DEVICE asm_data4 {
filename /scst/asm_data4.img
}
DEVICE asm_data5 {
filename /scst/asm_data5.img
}
DEVICE asm_data6 {
filename /scst/asm_data6.img
}
DEVICE asm_data7 {
filename /scst/asm_data7.img
}
DEVICE asm_data8 {
filename /scst/asm_data8.img
}
DEVICE asm_reco1 {
filename /scst/asm_reco1.img
}
DEVICE asm_reco2 {
filename /scst/asm_reco2.img
}
DEVICE asm_reco3 {
filename /scst/asm_reco3.img
}
DEVICE asm_reco4 {
filename /scst/asm_reco4.img
}
DEVICE asm_reco5 {
filename /scst/asm_reco5.img
}
DEVICE asm_reco6 {
filename /scst/asm_reco6.img
}
DEVICE asm_reco7 {
filename /scst/asm_reco7.img
}
DEVICE asm_reco8 {
filename /scst/asm_reco8.img
}
DEVICE systemdg1 {
filename /scst/systemdg1.img
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2014-08.oracle.asm.storage:luns {
cpu_mask 3
enabled 1
rel_tgt_id 1
LUN 0 disk0
LUN 1 disk1
LUN 2 systemdg1
LUN 3 asm_data1
LUN 4 asm_data2
LUN 5 asm_data3
LUN 6 asm_data4
LUN 7 asm_data5
LUN 8 asm_data6
LUN 9 asm_data7
LUN 10 asm_data8
LUN 11 asm_reco1
LUN 12 asm_reco2
LUN 13 asm_reco3
LUN 14 asm_reco4
LUN 15 asm_reco5
LUN 16 asm_reco6
LUN 17 asm_reco7
LUN 18 asm_reco8
}
}
[root@oracle651 ~]#
Adding the new LUNs to the Oracle database server is shown below. As soon as the above command to add the LUNs to the target is completed, the LUNs will immediately appear on the the SCSI client machine - the database server - (as long as it is logged in to the SAN) and will be visible as shown below using the multipath command.
[root@oracle631 ~]# multipath -ll -v2 | grep reco
mpathr (23734383163616238) dm-17 SCST_FIO,asm_reco1
mpathq (23239613933333133) dm-16 SCST_FIO,asm_reco4
mpathp (23533633466613536) dm-14 SCST_FIO,asm_reco3
mpatho (23430363636323231) dm-13 SCST_FIO,asm_reco2
mpathv (26661333739313737) dm-21 SCST_FIO,asm_reco8
mpathu (23164346539623861) dm-20 SCST_FIO,asm_reco7
mpatht (2656563303366642d) dm-19 SCST_FIO,asm_reco6
mpaths (23361306261623634) dm-18 SCST_FIO,asm_reco5
[root@oracle631 ~]#
Use the output above the /etc/multipath.conf can be edited.
[root@oracle631 ~]# vi /etc/multipath.conf
[root@oracle631 ~]# cat /etc/multipath.conf
# Configuration file for device mapper multipath.
# For a complete list of the default configuration values:
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults
# For a list of configuration options with descriptions:
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotated
defaults {
user_friendly_names yes
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
}
devices {
device {
vendor "SCST_BIO"
product "Direct-Access"
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
rr_min_io 1
rr_weight priorities
}
}
multipaths {
multipath {
wwid 26162356632353435
alias ASM_4096B_1
}
multipath {
wwid 26238666462643332
alias ASM_0512B_1
}
multipath {
wwid 23863316131356162
alias ASM_4096F_1
}
multipath {
wwid 23233626538613166
alias ASM_0512F_1
}
multipath {
wwid 26161383334393638
alias ASM_DATA8
}
multipath {
wwid 23464666134333935
alias ASM_DATA7
}
multipath {
wwid 23565353864626532
alias ASM_DATA6
}
multipath {
wwid 23661626637333762
alias ASM_DATA5
}
multipath {
wwid 23739316465623063
alias ASM_DATA4
}
multipath {
wwid 2333730323234392d
alias ASM_DATA3
}
multipath {
wwid 23130643262613365
alias ASM_DATA2
}
multipath {
wwid 23234333531326137
alias ASM_DATA1
}
multipath {
wwid 23734383163616238
alias ASM_RECO1
}
multipath {
wwid 23430363636323231
alias ASM_RECO2
}
multipath {
wwid 23533633466613536
alias ASM_RECO3
}
multipath {
wwid 23239613933333133
alias ASM_RECO4
}
multipath {
wwid 23361306261623634
alias ASM_RECO5
}
multipath {
wwid 2656563303366642d
alias ASM_RECO6
}
multipath {
wwid 23164346539623861
alias ASM_RECO7
}
multipath {
wwid 26661333739313737
alias ASM_RECO8
}
}
[root@oracle631 ~]#
Check the permissions, ownership and group of the LUNs to be sure they are correct. With these iSCSI LUNs from SCST normally the add instantly to the client machine with correct permissions and modes and no reboot is needed. The only step required is to edit the /etc/multipath.conf file and restart multipath as shown below.
[root@oracle631 ~]# ls -lrt /dev/mapper
total 0
crw-rw---- 1 root root 10, 236 Aug 13 10:17 control
lrwxrwxrwx 1 root root 7 Aug 13 10:17 vg_oracle631-lv_swap -> ../dm-1
lrwxrwxrwx 1 root root 7 Aug 13 10:17 vg_oracle631-lv_root -> ../dm-0
lrwxrwxrwx 1 root root 7 Aug 13 10:17 vg_u00-lv_oracle -> ../dm-2
lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_0512B_1 -> ../dm-7
lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA1 -> ../dm-5
lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_0512F_1 -> ../dm-4
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA6 -> ../dm-10
lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA3 -> ../dm-8
lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA4 -> ../dm-6
lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA5 -> ../dm-9
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA8 -> ../dm-11
lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_4096B_1 -> ../dm-3
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO1 -> ../dm-17
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA7 -> ../dm-12
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA2 -> ../dm-15
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO4 -> ../dm-16
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO2 -> ../dm-13
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO3 -> ../dm-14
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO6 -> ../dm-19
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO8 -> ../dm-21
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO5 -> ../dm-18
lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO7 -> ../dm-20
[root@oracle631 ~]# ls -lrt /dev/dm-*
brw-rw---- 1 root disk 252, 1 Aug 13 10:17 /dev/dm-1
brw-rw---- 1 root disk 252, 0 Aug 13 10:17 /dev/dm-0
brw-rw---- 1 root disk 252, 2 Aug 13 10:17 /dev/dm-2
brw-rw---- 1 grid asmadmin 252, 7 Aug 13 10:38 /dev/dm-7
brw-rw---- 1 grid asmadmin 252, 5 Aug 13 10:38 /dev/dm-5
brw-rw---- 1 grid asmadmin 252, 4 Aug 13 10:38 /dev/dm-4
brw-rw---- 1 grid asmadmin 252, 10 Aug 13 10:38 /dev/dm-10
brw-rw---- 1 grid asmadmin 252, 8 Aug 13 10:38 /dev/dm-8
brw-rw---- 1 grid asmadmin 252, 6 Aug 13 10:38 /dev/dm-6
brw-rw---- 1 grid asmadmin 252, 9 Aug 13 10:38 /dev/dm-9
brw-rw---- 1 grid asmadmin 252, 3 Aug 13 10:38 /dev/dm-3
brw-rw---- 1 grid asmadmin 252, 11 Aug 13 10:38 /dev/dm-11
brw-rw---- 1 grid asmadmin 252, 17 Aug 13 10:38 /dev/dm-17
brw-rw---- 1 grid asmadmin 252, 12 Aug 13 10:38 /dev/dm-12
brw-rw---- 1 grid asmadmin 252, 15 Aug 13 10:38 /dev/dm-15
brw-rw---- 1 grid asmadmin 252, 16 Aug 13 10:38 /dev/dm-16
brw-rw---- 1 grid asmadmin 252, 13 Aug 13 10:38 /dev/dm-13
brw-rw---- 1 grid asmadmin 252, 14 Aug 13 10:38 /dev/dm-14
brw-rw---- 1 grid asmadmin 252, 19 Aug 13 10:38 /dev/dm-19
brw-rw---- 1 grid asmadmin 252, 21 Aug 13 10:38 /dev/dm-21
brw-rw---- 1 grid asmadmin 252, 18 Aug 13 10:38 /dev/dm-18
brw-rw---- 1 grid asmadmin 252, 20 Aug 13 10:38 /dev/dm-20
[root@oracle631 ~]#
Configure the LUNs with ASMLib as shown below. Alternatively, for a large number of LUNs scripted methods can be used as shown in the Violin Memory Oracle Database Best Practice Document April 2014.
[root@oracle631 ~]# oracleasm createdisk ASM_RECO1 /dev/mapper/ASM_RECO1
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]# oracleasm createdisk ASM_RECO2 /dev/mapper/ASM_RECO2
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]# oracleasm createdisk ASM_RECO3 /dev/mapper/ASM_RECO3
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]# oracleasm createdisk ASM_RECO4 /dev/mapper/ASM_RECO4
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]# oracleasm createdisk ASM_RECO5 /dev/mapper/ASM_RECO5
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]# oracleasm createdisk ASM_RECO6 /dev/mapper/ASM_RECO6
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]# oracleasm createdisk ASM_RECO7 /dev/mapper/ASM_RECO7
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]# oracleasm createdisk ASM_RECO8 /dev/mapper/ASM_RECO8
Writing disk header: done
Instantiating disk: done
[root@oracle631 ~]#
Connect to the server as grid user as shown below and run "asmca" to build the RECO asm diskgroup. Note that asmca can be used here because these are 512 emulated LUNs so no special attributes are needed. If these were 4096 native LUNs command line would have to be used to build the diskgroup.
gstanden@vmem1:~$ ssh -Y -C grid@10.207.39.72
grid@10.207.39.72's password:
Last login: Wed Aug 13 10:45:42 2014 from 10.207.39.1
[grid@oracle631 ~]$ asmca
The screenshots of asmca are shown below.
The final configuration of disks for the Oracle database install.
SQL> l
1* select name, label, path, total_mb, header_status, mount_status, mode_status, sector_size from v$asm_disk
SQL> /
NAME LABEL PATH TOTAL_MB HEADER_STATU MOUNT_S MODE_ST SECTOR_SIZE
------------------------------ ------------------------------- -------------------- ---------- ------------ ------- ------- -----------
ASM_DATA1 ASM_DATA1 ORCL:ASM_DATA1 1024 MEMBER CACHED ONLINE 512
ASM_DATA2 ASM_DATA2 ORCL:ASM_DATA2 1024 MEMBER CACHED ONLINE 512
ASM_DATA3 ASM_DATA3 ORCL:ASM_DATA3 1024 MEMBER CACHED ONLINE 512
ASM_DATA4 ASM_DATA4 ORCL:ASM_DATA4 1024 MEMBER CACHED ONLINE 512
ASM_DATA5 ASM_DATA5 ORCL:ASM_DATA5 1024 MEMBER CACHED ONLINE 512
ASM_DATA6 ASM_DATA6 ORCL:ASM_DATA6 1024 MEMBER CACHED ONLINE 512
ASM_DATA7 ASM_DATA7 ORCL:ASM_DATA7 1024 MEMBER CACHED ONLINE 512
ASM_DATA8 ASM_DATA8 ORCL:ASM_DATA8 1024 MEMBER CACHED ONLINE 512
SYSTEMDG SYSTEMDG ORCL:SYSTEMDG 1024 MEMBER CACHED ONLINE 512
ASM_RECO1 ASM_RECO1 ORCL:ASM_RECO1 1024 MEMBER CACHED ONLINE 512
ASM_RECO2 ASM_RECO2 ORCL:ASM_RECO2 1024 MEMBER CACHED ONLINE 512
ASM_RECO3 ASM_RECO3 ORCL:ASM_RECO3 1024 MEMBER CACHED ONLINE 512
ASM_RECO4 ASM_RECO4 ORCL:ASM_RECO4 1024 MEMBER CACHED ONLINE 512
ASM_RECO5 ASM_RECO5 ORCL:ASM_RECO5 1024 MEMBER CACHED ONLINE 512
ASM_RECO6 ASM_RECO6 ORCL:ASM_RECO6 1024 MEMBER CACHED ONLINE 512
ASM_RECO7 ASM_RECO7 ORCL:ASM_RECO7 1024 MEMBER CACHED ONLINE 512
ASM_RECO8 ASM_RECO8 ORCL:ASM_RECO8 1024 MEMBER CACHED ONLINE 512
17 rows selected.
SQL>
Obtain the Oracle binaries and install for your chosen version of database. Begin the install. Here the oracle install media is unpacked under the /home/oracle directory as shown below.
[oracle@oracle631 database]$ pwd
/home/oracle/database
[oracle@oracle631 database]$ ./runInstaller
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 2527 MB Passed
Checking swap space: must be greater than 150 MB. Actual 5087 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-08-13_11-36-50AM. Please wait ...
[oracle@oracle631 database]$