This guide explains how to build SCST SAN on an Oracle Enterprise Linux 6.5 KVM guest VM based on the instructions here at spiderbird. This blog is based on that excellent guide. That guide also has a wealth of reference URLs that can be accessed as well. I intend to use this KVM guest SCST SAN to present 4K logcial/4K physical virtual LUNs to Oracle running in other KVM guests, including Oracle Real Application Clusters running in KVM guests. Because TGT cannot handle LUNs which are 4K logical / 4K physical it was necessary to get up to speed on SCST which can. This sourceforge SAN feature comparison sheet shows the full power of SCST compared to other linux SAN options. Ensure the virtual ethernet interface, typically "eth0" in the OEL 6.5 KVM guest is set to "connect automatically". If not, configure it as shown in the screenshots below. This can be configured from the server desktop, if using OEL 6.5 with a desktop, or simply edited in the ifcfg-eth0 file as shown below. As shown in bold below, be sure that "ONBOOT=yes" has been set in the ifcfg-ethX file where in this case my interface is ifcfg-eth0. [root@oracle651 network-scripts]# cat ifcfg-eth0DEVICE=eth0TYPE=EthernetUUID=824a1d97-1dc7-410b-89a5-49b353be826aONBOOT=yesNM_CONTROLLED=yesBOOTPROTO=dhcpDNS1=127.0.0.1DOMAIN=vmem.orgDEFROUTE=yesIPV4_FAILURE_FATAL=yesIPV6INIT=noNAME="System eth0"HWADDR=52:54:00:66:26:A0DNS2=192.168.1.1PEERDNS=noPEERROUTES=yesLAST_CONNECT=1407733083[root@oracle651 network-scripts]# If using any virtualiztion options, remove them now (does not apply for this blog because this is being installed into a KVM guest). But for completeness, this is included from the spiderbird blog. These are options below which could be selected duiring OEL 6.5 OS install. Select the
“Software Development Workstation” option then unselect groups
virtualization, virtualization-client and virtualization-platform or let
the installer finish then use “yum groupremove virtualization
virtualization-client virtualization-platform” once you’ve booted into
the OS the first time. By selecting the “Software Development
Workstation” as your install type, it will pretty much install every
package you’ll need to do the kernel compile.Once the yum update is complete, reboot the KVM guest so that the just-updated kernel is running in the KVM guest. Install the following packages using yum as shown below. Do this as root. Note that further on in this blog there was a missing dependency on "xmlto" which I am retroactively addressing here in this blog so there won't be that missing dependency. [root@oracle651 ~]# yum install newt-devel[root@oracle651 ~]# yum install asciidoc[root@oracle651 ~]# yum install xmltoAs root, add the "mockbuild" user as shown below. [root@oracle651 ~]# useradd mockbuildNow su to the "mockbuild" user as shown below. A password is not needed for the mockbuild user unless you explicitly want one. It's just an account that will be used for this build. [root@oracle651 ~]# su - mockbuild[mockbuild@oracle651 ~]$ pwd/home/mockbuild[mockbuild@oracle651 ~]$ iduid=501(mockbuild) gid=501(mockbuild) groups=501(mockbuild) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023[mockbuild@oracle651 ~]$ Run the following commands as the mockbuild user from the /home/mockbuild directory as shown below. [mockbuild@oracle651 ~]$ mkdir -p ~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}[mockbuild@oracle651 ~]$ echo '%_topdir %(echo $HOME)/rpmbuild' > ~/.rpmmacros[mockbuild@oracle651 ~]$ rpm -i http://vault.centos.org/6.5/updates/Source/SPackages/kernel-2.6.32-431.23.3.el6.src.rpm 2>&1 | grep -v mockNote that even though this is the Oracle Red Hat Compatible kernel, I downloaded the CentOS kernel source from vault.centos.org, which should be binary compatible with RedHat kernel source. However, it should be possible to obtain the actual Oracle Red Hat Compatible kernel source which *should* also work. Now you can run the command. If it bogs down towards the end run the above rngd command again. Note: be sure when running the command below, that if you cut and paste from this blog that the word "--target" has "two dashes" i.e "--" in front of it, and not just one big dash. Sometimes characterset conversion can cause two dashed cut and pasted to become an uninterpretable single large dash. Before running the below "rpmbuild" command in step after the next one, open another window as root and run the entropy generator as shown here. If the run of rpmbuild in the next step bogs down, run this command again to refill the entropy pool. It will make the rpmbuild complete faster. [root@oracle651 ~]# rngd -r /dev/urandom[root@oracle651 ~]# [root@oracle651 ~]# su - mockbuild[mockbuild@oracle651 ~]$ rpmbuild -bp --target=$(uname -m) ./rpmbuild/SPECS/kernel.specBuilding target platforms: x86_64Building for target x86_64Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.rlmm5g+ umask 022+ cd /home/mockbuild/rpmbuild/BUILD+ LANG=C+ export LANG+ unset DISPLAY+ patch_command='patch -p1 -F1 -s'+ '[' '!' -d kernel-2.6.32-431.23.3.el6/vanilla-2.6.32-431.23.3.el6/ ']'+ rm -f pax_global_header+ cd /home/mockbuild/rpmbuild/BUILD+ rm -rf kernel-2.6.32-431.23.3.el6+ /bin/mkdir -p kernel-2.6.32-431.23.3.el6+ cd kernel-2.6.32-431.23.3.el6+ /usr/bin/bzip2 -dc /home/mockbuild/rpmbuild/SOURCES/linux-2.6.32-431.23.3.el6.tar.bz2+ /bin/tar -xf -+ STATUS=0+ '[' 0 -ne 0 ']'+ /bin/chmod -Rf a+rX,u+w,g-w,o-w .+ mv linux-2.6.32-431.23.3.el6 vanilla-2.6.32-431.23.3.el6+ '[' -d linux-2.6.32-431.23.3.el6.x86_64 ']'+ cp -rl vanilla-2.6.32-431.23.3.el6 linux-2.6.32-431.23.3.el6.x86_64+ cd linux-2.6.32-431.23.3.el6.x86_64+ cp /home/mockbuild/rpmbuild/SOURCES/config-debug /home/mockbuild/rpmbuild/SOURCES/config-debug-rhel /home/mockbuild/rpmbuild/SOURCES/config-framepointer /home/mockbuild/rpmbuild/SOURCES/config-generic /home/mockbuild/rpmbuild/SOURCES/config-generic-rhel /home/mockbuild/rpmbuild/SOURCES/config-i686 /home/mockbuild/rpmbuild/SOURCES/config-i686-debug /home/mockbuild/rpmbuild/SOURCES/config-i686-debug-rhel /home/mockbuild/rpmbuild/SOURCES/config-i686-nodebug /home/mockbuild/rpmbuild/SOURCES/config-i686-nodebug-rhel /home/mockbuild/rpmbuild/SOURCES/config-i686-rhel /home/mockbuild/rpmbuild/SOURCES/config-ia64-generic-rhel /home/mockbuild/rpmbuild/SOURCES/config-nodebug /home/mockbuild/rpmbuild/SOURCES/config-nodebug-rhel /home/mockbuild/rpmbuild/SOURCES/config-powerpc-generic /home/mockbuild/rpmbuild/SOURCES/config-powerpc-generic-rhel /home/mockbuild/rpmbuild/SOURCES/config-powerpc64 /home/mockbuild/rpmbuild/SOURCES/config-powerpc64-debug /home/mockbuild/rpmbuild/SOURCES/config-powerpc64-debug-rhel /home/mockbuild/rpmbuild/SOURCES/config-powerpc64-kdump /home/mockbuild/rpmbuild/SOURCES/config-powerpc64-kdump-rhel /home/mockbuild/rpmbuild/SOURCES/config-powerpc64-rhel /home/mockbuild/rpmbuild/SOURCES/config-s390x /home/mockbuild/rpmbuild/SOURCES/config-s390x-debug /home/mockbuild/rpmbuild/SOURCES/config-s390x-debug-rhel /home/mockbuild/rpmbuild/SOURCES/config-s390x-generic-rhel /home/mockbuild/rpmbuild/SOURCES/config-s390x-kdump /home/mockbuild/rpmbuild/SOURCES/config-s390x-kdump-rhel /home/mockbuild/rpmbuild/SOURCES/config-s390x-rhel /home/mockbuild/rpmbuild/SOURCES/config-x86-generic /home/mockbuild/rpmbuild/SOURCES/config-x86-generic-rhel /home/mockbuild/rpmbuild/SOURCES/config-x86_64-debug /home/mockbuild/rpmbuild/SOURCES/config-x86_64-debug-rhel /home/mockbuild/rpmbuild/SOURCES/config-x86_64-generic /home/mockbuild/rpmbuild/SOURCES/config-x86_64-generic-rhel /home/mockbuild/rpmbuild/SOURCES/config-x86_64-nodebug /home/mockbuild/rpmbuild/SOURCES/config-x86_64-nodebug-rhel .+ cp /home/mockbuild/rpmbuild/SOURCES/merge.pl /home/mockbuild/rpmbuild/SOURCES/Makefile.common /home/mockbuild/rpmbuild/SOURCES/perf /home/mockbuild/rpmbuild/SOURCES/kabitool /home/mockbuild/rpmbuild/SOURCES/check-kabi .+ make -f /home/mockbuild/rpmbuild/SOURCES/Makefile.config VERSION=2.6.32 configsperl merge.pl config-nodebug-merged config-generic-merged > temp-genericperl merge.pl config-x86_64-generic-merged temp-generic > temp-x86_64-genericperl merge.pl config-x86_64-nodebug-merged temp-x86_64-generic x86_64 > kernel-2.6.32-x86_64.configperl merge.pl config-debug-merged config-generic-merged > temp-debug-genericperl merge.pl config-x86_64-generic-merged temp-debug-generic > temp-x86_64-debug-genericperl merge.pl config-x86_64-debug-merged temp-x86_64-debug-generic x86_64 > kernel-2.6.32-x86_64-debug.configperl merge.pl config-x86-generic-merged temp-generic > temp-x86-genericperl merge.pl config-i686-merged temp-x86-generic > temp-i686-finalperl merge.pl config-i686-nodebug-merged temp-i686-final i386 > kernel-2.6.32-i686.configperl merge.pl config-x86-generic-merged temp-debug-generic > temp-x86-debug-genericperl merge.pl config-i686-merged temp-x86-debug-generic > temp-i686-debug-finalperl merge.pl config-i686-debug-merged temp-i686-debug-final i386 > kernel-2.6.32-i686-debug.configperl merge.pl config-s390x-merged temp-generic > temp-s390x-genericperl merge.pl config-s390x-merged temp-s390x-generic s390 > kernel-2.6.32-s390x.configperl merge.pl config-s390x-merged temp-debug-generic > temp-s390x-debug-genericperl merge.pl config-s390x-debug-merged temp-s390x-debug-generic s390 > kernel-2.6.32-s390x-debug.configperl merge.pl config-s390x-kdump-merged kernel-2.6.32-s390x.config s390 > kernel-2.6.32-s390x-kdump.configperl merge.pl config-powerpc-generic-merged temp-generic > temp-powerpc-genericperl merge.pl config-powerpc64-merged temp-powerpc-generic powerpc > kernel-2.6.32-ppc64.configperl merge.pl config-powerpc64-kdump-merged kernel-2.6.32-ppc64.config powerpc > kernel-2.6.32-ppc64-kdump.configperl merge.pl config-powerpc-generic-merged temp-debug-generic > temp-powerpc-debug-genericperl merge.pl config-powerpc64-merged temp-powerpc-debug-generic powerpc > temp-powerpc64-debug-genericperl merge.pl config-powerpc64-debug-merged temp-powerpc64-debug-generic powerpc > kernel-2.6.32-ppc64-debug.config+ ApplyOptionalPatch linux-kernel-test.patch+ local patch=linux-kernel-test.patch+ shift+ '[' '!' -f /home/mockbuild/rpmbuild/SOURCES/linux-kernel-test.patch ']'++ wc -l /home/mockbuild/rpmbuild/SOURCES/linux-kernel-test.patch++ awk '{print $1}'+ local C=1+ '[' 1 -gt 9 ']'+ chmod +x scripts/checkpatch.pl+ mkdir configs+ for cfg in 'kernel-2.6.32-*.config'++ grep -c kernel-2.6.32-i686-debug.config++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config+ '[' 0 -eq 0 ']'+ rm -f kernel-2.6.32-i686-debug.config+ for cfg in 'kernel-2.6.32-*.config'++ grep -c kernel-2.6.32-i686.config++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config+ '[' 0 -eq 0 ']'+ rm -f kernel-2.6.32-i686.config+ for cfg in 'kernel-2.6.32-*.config'++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config++ grep -c kernel-2.6.32-ppc64-debug.config+ '[' 0 -eq 0 ']'+ rm -f kernel-2.6.32-ppc64-debug.config+ for cfg in 'kernel-2.6.32-*.config'++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config++ grep -c kernel-2.6.32-ppc64-kdump.config+ '[' 0 -eq 0 ']'+ rm -f kernel-2.6.32-ppc64-kdump.config+ for cfg in 'kernel-2.6.32-*.config'++ grep -c kernel-2.6.32-ppc64.config++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config+ '[' 0 -eq 0 ']'+ rm -f kernel-2.6.32-ppc64.config+ for cfg in 'kernel-2.6.32-*.config'++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config++ grep -c kernel-2.6.32-s390x-debug.config+ '[' 0 -eq 0 ']'+ rm -f kernel-2.6.32-s390x-debug.config+ for cfg in 'kernel-2.6.32-*.config'++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config++ grep -c kernel-2.6.32-s390x-kdump.config+ '[' 0 -eq 0 ']'+ rm -f kernel-2.6.32-s390x-kdump.config+ for cfg in 'kernel-2.6.32-*.config'++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config++ grep -c kernel-2.6.32-s390x.config+ '[' 0 -eq 0 ']'+ rm -f kernel-2.6.32-s390x.config+ for cfg in 'kernel-2.6.32-*.config'++ grep -c kernel-2.6.32-x86_64-debug.config++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config+ '[' 1 -eq 0 ']'+ for cfg in 'kernel-2.6.32-*.config'++ echo kernel-2.6.32-x86_64-debug.config kernel-2.6.32-x86_64.config++ grep -c kernel-2.6.32-x86_64.config+ '[' 1 -eq 0 ']'+ for i in '*.config'+ mv kernel-2.6.32-x86_64-debug.config .config++ head -1 .config++ cut -b 3-+ Arch=x86_64+ make ARCH=x86_64 nonint_oldconfig+ echo '# x86_64'+ cat .config+ rm -f include/generated/kernel.arch+ rm -f include/generated/kernel.cross+ for i in '*.config'+ mv kernel-2.6.32-x86_64.config .config++ head -1 .config++ cut -b 3-+ Arch=x86_64+ make ARCH=x86_64 nonint_oldconfig+ echo '# x86_64'+ cat .config+ rm -f include/generated/kernel.arch+ rm -f include/generated/kernel.cross+ find . '(' -name '*.orig' -o -name '*~' ')' -exec rm -f '{}' ';'+ cp /home/mockbuild/rpmbuild/SOURCES/extrakeys.pub .+ cat###### Now generating a PGP key pair to be used for signing modules.###### If this takes a long time, you might wish to run rngd in the background to### keep the supply of entropy topped up. It needs to be run as root, and### should use a hardware random number generator if one is available, eg:###### rngd -r /dev/hwrandom###### If one isn't available, the pseudo-random number generator can be used:###### rngd -r /dev/urandom###+ gpg --homedir . --batch --gen-key /home/mockbuild/rpmbuild/SOURCES/genkeygpg: WARNING: unsafe permissions on homedir `.'gpg: keyring `./secring.gpg' createdgpg: keyring `./pubring.gpg' created+ cat###### Key pair generated.###+ '[' -s /home/mockbuild/rpmbuild/SOURCES/extrakeys.pub ']'+ gpg --homedir . --no-default-keyring --keyring kernel.pub --import /home/mockbuild/rpmbuild/SOURCES/extrakeys.pubgpg: WARNING: unsafe permissions on homedir `.'gpg: ./trustdb.gpg: trustdb createdgpg: key CD09BEDA: public key "Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>" importedgpg: Total number processed: 1gpg: imported: 1+ gpg --homedir . --export --keyring ./kernel.pub CentOSgpg: WARNING: unsafe permissions on homedir `.'+ gcc -o scripts/bin2c scripts/bin2c.c+ scripts/bin2c ksign_def_public_key __initdata+ cd ..+ exit 0[mockbuild@oracle651 ~]Now execute the following command as shown below. [mockbuild@oracle651 ~]$ svn co https://svn.code.sf.net/p/scst/svn/branches/2.2.x scstThe scst code is installed under the /home/mockbuild directory. Now Change directory as shown below. [mockbuild@oracle651 ~]$ cd ~/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64/Now apply the two required patches for SCST as shown below. [mockbuild@oracle651 ~]$ cd ~/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64/[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ patch -p1 < ~/scst/scst/kernel/rhel/scst_exec_req_fifo-2.6.32.patchpatching file block/blk-map.cHunk #2 succeeded at 275 (offset 3 lines).patching file include/linux/blkdev.hHunk #1 succeeded at 697 (offset -11 lines).Hunk #2 succeeded at 811 (offset -3 lines).patching file include/linux/scatterlist.hpatching file lib/scatterlist.c[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ patch -p1 < ~/scst/iscsi-scst/kernel/patches/rhel/put_page_callback-2.6.32-220.patchpatching file include/linux/Kbuildpatching file include/linux/mm_types.hpatching file include/linux/net.hHunk #2 succeeded at 390 (offset 23 lines).patching file net/core/dev.cHunk #1 succeeded at 3622 (offset 625 lines).patching file net/core/skbuff.cHunk #2 succeeded at 394 (offset 50 lines).Hunk #3 succeeded at 699 (offset 51 lines).Hunk #4 succeeded at 714 (offset 51 lines).Hunk #5 succeeded at 880 (offset 40 lines).Hunk #6 succeeded at 961 with fuzz 2 (offset 35 lines).Hunk #7 succeeded at 1240 with fuzz 2 (offset 38 lines).Hunk #8 succeeded at 1409 (offset 38 lines).Hunk #9 succeeded at 1511 (offset 38 lines).Hunk #10 succeeded at 1535 (offset 38 lines).Hunk #11 succeeded at 1545 (offset 38 lines).Hunk #12 succeeded at 1566 (offset 38 lines).Hunk #13 succeeded at 2196 (offset 38 lines).Hunk #14 succeeded at 2319 (offset 39 lines).Hunk #15 succeeded at 2341 (offset 39 lines).Hunk #16 succeeded at 2753 (offset 56 lines).patching file net/ipv4/ip_output.cHunk #1 succeeded at 998 (offset 17 lines).Hunk #2 succeeded at 1235 (offset 22 lines).patching file net/ipv4/MakefileHunk #1 succeeded at 51 (offset 2 lines).patching file net/ipv4/tcp.cHunk #1 succeeded at 820 with fuzz 1 (offset 21 lines).Hunk #2 succeeded at 1029 (offset 22 lines).Hunk #3 succeeded at 1070 (offset 22 lines).patching file net/ipv4/tcp_output.cHunk #1 succeeded at 1061 (offset 150 lines).patching file net/ipv4/tcp_zero_copy.cpatching file net/ipv6/ip6_output.cHunk #1 succeeded at 1361 (offset -39 lines).patching file net/Kconfig[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ Check to make sure the config files are about the same. Here they only differ by comment lines so that is acceptable. [mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ diff .config /boot/config-2.6.32-431.23.3.el6.x86_643,4c3,4< # Linux kernel version: 2.6.32< # Mon Aug 11 14:27:09 2014---> # Linux kernel version: 2.6.32-431.23.3.el6.x86_64> # Tue Jul 29 09:15:22 2014[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ Now do the menuconfig step as shown below. The menuconfig GUI open. Navigate through the GUI and choose the options as quoted and described at the spiderweb reference here. For completeness, those changes that need to be made and verified in menuconfig are as follows from the reference but restated here for convenience. Select “Network support”->”Networking options” make sure “TCP/IP networking” is checked (should be) and check “TCP/IP zero-copy transfer completion notification” underneath so SCST will be notified when a transfer of data across the nic has finished. This way, it can hurry up & send the next block. Should speed things up. Next, verify that “Device Drivers”->”SCSI device support”->”SCSI disk support” is already checked. Again, for RHEL 6.4 it should be. Verify that “Enable the block layer”->”IO Schedulers”->”CFQ I/O scheduler” and “Default I/O scheduler” set to CFQ are both set.Lastly, set “Processor type an features”->”Preemption Model” is set to “No Forced Preemption” versus Desktop mode.[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ make menuconfig HOSTCC scripts/kconfig/lxdialog/checklist.o HOSTCC scripts/kconfig/lxdialog/inputbox.o HOSTCC scripts/kconfig/lxdialog/menubox.o HOSTCC scripts/kconfig/lxdialog/textbox.o HOSTCC scripts/kconfig/lxdialog/util.o HOSTCC scripts/kconfig/lxdialog/yesno.o HOSTCC scripts/kconfig/mconf.o HOSTLD scripts/kconfig/mconfscripts/kconfig/mconf arch/x86/Kconfig## configuration written to .config#*** End of Linux kernel configuration.*** Execute 'make' to build the kernel or try 'make help'.[mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ Now run make as shown below. [mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ time make -j8 all | tee makescst.logNow make modules install as shown below, as the "root" user. [mockbuild@oracle651 linux-2.6.32-431.23.3.el6.x86_64]$ su - root[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# cd /home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# make modules_installNext, again as root, do a "make install" as shown below. [root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# make install | tee make-install.logsh /home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64/arch/x86/boot/install.sh 2.6.32-scst arch/x86/boot/bzImage \ System.map "/boot"[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# Now edit /boot/grub/menu.lst to update the server to use the new kernel at boot as shown below. Change the value of "default=" to use the new kernel. The first kernel listed is always the "0" (zero) kernel so in this example set "detault=0". Also, while editing this file, remove the "rhgb quiet" from that kernel line so that all boot output messages will be buffered to console, and also add at the end of the kernel line "console=ttyS0" so that KVM console is configured. # grub.conf generated by anaconda## Note that you do not have to rerun grub after making changes to this file# NOTICE: You have a /boot partition. This means that# all kernel and initrd paths are relative to /boot/, eg.# root (hd0,0)# kernel /vmlinuz-version ro root=/dev/mapper/vg_oracle651-lv_root# initrd /initrd-[generic-]version.img#boot=/dev/vdadefault=0timeout=5splashimage=(hd0,0)/grub/splash.xpm.gzhiddenmenutitle Oracle Linux Server Red Hat Compatible Kernel (2.6.32-scst) root (hd0,0) kernel /vmlinuz-2.6.32-scst ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM crashkernel=auto console=ttyS0 initrd /initramfs-2.6.32-scst.imgtitle Oracle Linux Server (3.8.13-35.3.4.el6uek.x86_64) root (hd0,0) kernel /vmlinuz-3.8.13-35.3.4.el6uek.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet initrd /initramfs-3.8.13-35.3.4.el6uek.x86_64.imgtitle Oracle Linux Server Red Hat Compatible Kernel (2.6.32-431.23.3.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-431.23.3.el6.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto initrd /initramfs-2.6.32-431.23.3.el6.x86_64.imgtitle Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-16.2.1.el6uek.x86_64) root (hd0,0) kernel /vmlinuz-3.8.13-16.2.1.el6uek.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet initrd /initramfs-3.8.13-16.2.1.el6uek.x86_64.imgtitle Oracle Linux Server Red Hat Compatible Kernel (2.6.32-431.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/vg_oracle651-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_oracle651/lv_swap rd_NO_MD rd_LVM_LV=vg_oracle651/lv_root SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet initrd /initramfs-2.6.32-431.el6.x86_64.imgNow copy the .config file over to the "/boot" directory as shown below. Expected output and commands are shown below. [root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# pwd/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# cp -p .config /boot/config-2.6.32-431.23.3-scst.el6.x86_64[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# ls -lrt /boot/config-2.6* | sort-rw-r--r--. 1 root root 105195 Nov 21 2013 /boot/config-2.6.32-431.el6.x86_64-rw-r--r--. 1 root root 105200 Jul 29 11:42 /boot/config-2.6.32-431.23.3.el6.x86_64-rw-rw-r--. 1 mockbuild mockbuild 105242 Aug 11 15:13 /boot/config-2.6.32-431.23.3-scst.el6.x86_64[root@oracle651 linux-2.6.32-431.23.3.el6.x86_64]# Reboot the KVM guest. Be sure the config file has been named exactly as shown above before rebooting, or if your kernel version was slightly different, be sure the kernel version and format of the name of the config file is of the format "config-ver.s.ion-scst.el6.x86_64". Next change the various Makefiles to release version by issuing “make debug2release” in the main scst dir. This will remove the debugging stuff. Do this as root. [root@oracle651 scst]# pwd/home/mockbuild/scst[root@oracle651 scst]# iduid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023[root@oracle651 scst]# make debug2release | tee make_debug2release.logrm -f TAGScd scst && make extracleanmake[1]: Entering directory `/home/mockbuild/scst/scst'cd src && make extracleanmake[2]: Entering directory `/home/mockbuild/scst/scst/src'rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend Modules.symvers \ Module.symvers Module.markers modules.orderrm -rf .tmp_versionscd dev_handlers && make cleanmake[3]: Entering directory `/home/mockbuild/scst/scst/src/dev_handlers'rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend Modules.symvers \ Module.symvers Module.markers modules.orderrm -rf .tmp_versionsmake[3]: Leaving directory `/home/mockbuild/scst/scst/src/dev_handlers'cd dev_handlers && make extracleanmake[3]: Entering directory `/home/mockbuild/scst/scst/src/dev_handlers'rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend Modules.symvers \ Module.symvers Module.markers modules.orderrm -rf .tmp_versionsrm -f *.orig *.rejmake[3]: Leaving directory `/home/mockbuild/scst/scst/src/dev_handlers'rm -f *.orig *.rejmake[2]: Leaving directory `/home/mockbuild/scst/scst/src'make[1]: Leaving directory `/home/mockbuild/scst/scst'make[1]: Entering directory `/home/mockbuild/scst/doc'rm -f *.txt *.html *.tex *.dvi *.ps *.pdf *.info *.lyx *.rtfrm -f *.orig *.rejmake[1]: Leaving directory `/home/mockbuild/scst/doc'make[1]: Entering directory `/home/mockbuild/scst/qla2x00t'rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend *~ Modules.symvers \ Module.symvers Module.markers modules.orderrm -rf .tmp_versionsrm -f *.orig *.rejmake[1]: Leaving directory `/home/mockbuild/scst/qla2x00t'make[1]: Entering directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'rm -f *.o *.ko .*.cmd *.mod.c .*.d .depend *~ Modules.symvers \ Module.symvers Module.markers modules.orderrm -rf .tmp_versionsrm -f *.orig *.rejmake[1]: Leaving directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'make[1]: Entering directory `/home/mockbuild/scst/srpt'make -C /lib/modules/2.6.32-scst/build SUBDIRS=/home/mockbuild/scst/srpt/src cleanmake[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'rm -f src/Modules.symvers src/Module.symvers src/Module.markers \ src/modules.orderrm -f *.orig *.rejmake[1]: Leaving directory `/home/mockbuild/scst/srpt'make[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'make -C usr extracleanmake[2]: Entering directory `/home/mockbuild/scst/iscsi-scst/usr'rm -f *.o iscsi-scstd iscsi-scst-adm .depend*rm -f *.orig *.rejmake[2]: Leaving directory `/home/mockbuild/scst/iscsi-scst/usr'make -C /lib/modules/2.6.32-scst/build SUBDIRS=/home/mockbuild/scst/iscsi-scst/kernel cleanmake[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'rm -f kernel/Modules.symvers kernel/Module.symvers \ kernel/Module.markers kernel/modules.order \ include/iscsi_scst_itf_ver.h \ kernel/*.orig kernel/*.rejmake[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'make[1]: Entering directory `/home/mockbuild/scst/usr/fileio'rm -f *.o fileio_tgt .depend*rm -f *.orig *.rejmake[1]: Leaving directory `/home/mockbuild/scst/usr/fileio'make[1]: Entering directory `/home/mockbuild/scst/scst_local'make[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'make[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'rm -f *.orig *.rejmake[1]: Leaving directory `/home/mockbuild/scst/scst_local'cd scst && make debug2releasemake[1]: Entering directory `/home/mockbuild/scst/scst'cd src && make debug2releasemake[2]: Entering directory `/home/mockbuild/scst/scst/src'sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefilesed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefilesed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefilerm Makefile.aacd dev_handlers && make debug2releasemake[3]: Entering directory `/home/mockbuild/scst/scst/src/dev_handlers'sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefilesed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefilesed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefilerm Makefile.aamake[3]: Leaving directory `/home/mockbuild/scst/scst/src/dev_handlers'make[2]: Leaving directory `/home/mockbuild/scst/scst/src'make[1]: Leaving directory `/home/mockbuild/scst/scst'make[1]: Entering directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefilesed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefilesed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefilerm Makefile.aamake[1]: Leaving directory `/home/mockbuild/scst/qla2x00t/qla2x00-target'make[1]: Entering directory `/home/mockbuild/scst/srpt'make cleanmake[2]: Entering directory `/home/mockbuild/scst/srpt'make -C /lib/modules/2.6.32-scst/build SUBDIRS=/home/mockbuild/scst/srpt/src cleanmake[3]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'make[3]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'rm -f src/Modules.symvers src/Module.symvers src/Module.markers \ src/modules.ordermake[2]: Leaving directory `/home/mockbuild/scst/srpt'make[1]: Leaving directory `/home/mockbuild/scst/srpt'make[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ /home/mockbuild/scst/iscsi-scst/kernel/Makefilesed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ /home/mockbuild/scst/iscsi-scst/kernel/Makefilesed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ /home/mockbuild/scst/iscsi-scst/kernel/Makefilerm /home/mockbuild/scst/iscsi-scst/kernel/Makefile.aamake[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'make[1]: Entering directory `/home/mockbuild/scst/usr/fileio'sed -i.aa s/"^CFLAGS += \-DEXTRACHECKS"/"#CFLAGS += \-DEXTRACHECKS"/ Makefilesed -i.aa s/"#CFLAGS += \-DTRACING"/"CFLAGS += \-DTRACING"/ Makefilesed -i.aa s/"^CFLAGS += \-DDEBUG -g -fno-inline -fno-inline-functions"/"#CFLAGS += \-DDEBUG -g -fno-inline -fno-inline-functions"/ Makefilerm Makefile.aamake[1]: Leaving directory `/home/mockbuild/scst/usr/fileio'make[1]: Entering directory `/home/mockbuild/scst/scst_local'sed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_EXTRACHECKS"/ Makefilesed -i.aa s/"#EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/"EXTRA_CFLAGS += \-DCONFIG_SCST_TRACING"/ Makefilesed -i.aa s/"^EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/"#EXTRA_CFLAGS += \-DCONFIG_SCST_DEBUG -g -fno-inline -fno-inline-functions"/ Makefilerm Makefile.aamake[1]: Leaving directory `/home/mockbuild/scst/scst_local'[root@oracle651 scst]# Now go ahead and issue “make scst scst_install” as root in the mockbuild directory’s scst folder. Commands and expected output are shown below. Do this as "root" user. [root@oracle651 scst]# pwd/home/mockbuild/scst[root@oracle651 scst]# iduid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023[root@oracle651 scst]# make scst scst_install | tee make_scst_scst_install.logcd scst && make allmake[1]: Entering directory `/home/mockbuild/scst/scst'cd src && make allmake[2]: Entering directory `/home/mockbuild/scst/scst/src'make -C /lib/modules/2.6.32-scst/build SUBDIRS=/home/mockbuild/scst/scst/src BUILD_DEV=mmake[3]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64' LD /home/mockbuild/scst/scst/src/built-in.o CC [M] /home/mockbuild/scst/scst/src/scst_main.o CC [M] /home/mockbuild/scst/scst/src/scst_targ.o CC [M] /home/mockbuild/scst/scst/src/scst_lib.o CC [M] /home/mockbuild/scst/scst/src/scst_sysfs.o CC [M] /home/mockbuild/scst/scst/src/scst_mem.o CC [M] /home/mockbuild/scst/scst/src/scst_debug.o CC [M] /home/mockbuild/scst/scst/src/scst_pres.o CC [M] /home/mockbuild/scst/scst/src/scst_tg.o LD [M] /home/mockbuild/scst/scst/src/scst.o LD /home/mockbuild/scst/scst/src/dev_handlers/built-in.o CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_cdrom.o CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_changer.o CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_disk.o CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_modisk.o CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_tape.o CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_vdisk.o CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_raid.o CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_processor.o CC [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_user.o Building modules, stage 2. MODPOST 10 modules CC /home/mockbuild/scst/scst/src/dev_handlers/scst_cdrom.mod.o LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_cdrom.ko.unsigned NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_cdrom.ko CC /home/mockbuild/scst/scst/src/dev_handlers/scst_changer.mod.o LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_changer.ko.unsigned NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_changer.ko CC /home/mockbuild/scst/scst/src/dev_handlers/scst_disk.mod.o LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_disk.ko.unsigned NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_disk.ko CC /home/mockbuild/scst/scst/src/dev_handlers/scst_modisk.mod.o LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_modisk.ko.unsigned NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_modisk.ko CC /home/mockbuild/scst/scst/src/dev_handlers/scst_processor.mod.o LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_processor.ko.unsigned NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_processor.ko CC /home/mockbuild/scst/scst/src/dev_handlers/scst_raid.mod.o LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_raid.ko.unsigned NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_raid.ko CC /home/mockbuild/scst/scst/src/dev_handlers/scst_tape.mod.o LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_tape.ko.unsigned NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_tape.ko CC /home/mockbuild/scst/scst/src/dev_handlers/scst_user.mod.o LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_user.ko.unsigned NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_user.ko CC /home/mockbuild/scst/scst/src/dev_handlers/scst_vdisk.mod.o LD [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_vdisk.ko.unsigned NO SIGN [M] /home/mockbuild/scst/scst/src/dev_handlers/scst_vdisk.ko CC /home/mockbuild/scst/scst/src/scst.mod.o LD [M] /home/mockbuild/scst/scst/src/scst.ko.unsigned NO SIGN [M] /home/mockbuild/scst/scst/src/scst.komake[3]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'make[2]: Leaving directory `/home/mockbuild/scst/scst/src'make[1]: Leaving directory `/home/mockbuild/scst/scst'cd scst && make installmake[1]: Entering directory `/home/mockbuild/scst/scst'cd src && make installmake[2]: Entering directory `/home/mockbuild/scst/scst/src'make -C /lib/modules/2.6.32-scst/build SUBDIRS=/home/mockbuild/scst/scst/src BUILD_DEV=mmake[3]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64' Building modules, stage 2. MODPOST 10 modulesmake[3]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'rm -f /lib/modules/2.6.32-scst/extra/scsi_tgt.koinstall -d /lib/modules/2.6.32-scst/extrainstall -d /lib/modules/2.6.32-scst/extra/dev_handlersinstall -m 644 dev_handlers/*.ko /lib/modules/2.6.32-scst/extra/dev_handlersinstall -m 644 scst.ko /lib/modules/2.6.32-scst/extrainstall -d /usr/local/include/scstinstall -m 644 ../include/scst.h /usr/local/include/scstinstall -m 644 ../include/scst_sgv.h /usr/local/include/scstinstall -m 644 ../include/scst_debug.h /usr/local/include/scstinstall -m 644 ../include/scst_user.h /usr/local/include/scstinstall -m 644 ../include/scst_const.h /usr/local/include/scstrm -f /usr/local/include/scst/Modules.symversinstall -m 644 Module.symvers /usr/local/include/scst/sbin/depmod -a 2.6.32-scstmkdir -p /var/lib/scst/pr*****************************************************************!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!**!! !!**!! Now don't forget to rebuild and reinstall all your !!**!! target drivers, custom dev handlers and necessary user !!**!! space applications. Otherwise, because of the versions !!**!! mismatch, you could have many problems and crashes. !!**!! See IMPORTANT note in the "Installation" section of !!**!! SCST's README file for more info. !!**!! !!**!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!*****************************************************************make[2]: Leaving directory `/home/mockbuild/scst/scst/src'make[1]: Leaving directory `/home/mockbuild/scst/scst'[root@oracle651 scst]# Issue “make iscsi iscsi_install” as root in the /home/mockbuild folder. Commands and expected output are shown below. [root@oracle651 scst]# pwd/home/mockbuild/scst[root@oracle651 scst]# iduid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023[root@oracle651 scst]# make iscsi iscsi_install | tee make_iscsi_installcd iscsi-scst && make allmake[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'echo "/* Autogenerated, don't edit */" >include/iscsi_scst_itf_ver.hecho "" >>include/iscsi_scst_itf_ver.hecho -n "#define ISCSI_SCST_INTERFACE_VERSION " >>include/iscsi_scst_itf_ver.hecho -n "ISCSI_VERSION_STRING \"_\" " >>include/iscsi_scst_itf_ver.hecho "\"`sha1sum include/iscsi_scst.h|awk '{printf $1}'`\"" >>include/iscsi_scst_itf_ver.hmake -C usr SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/includemake[2]: Entering directory `/home/mockbuild/scst/iscsi-scst/usr'cc -M -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsid.c iscsi_scstd.c conn.c session.c target.c message.c ctldev.c log.c chap.c event.c param.c config.c isns.c md5.c sha1.c >.depend_dcc -c -o iscsid.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsid.ccc -c -o iscsi_scstd.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsi_scstd.ccc -c -o conn.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE conn.ccc -c -o session.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE session.ccc -c -o target.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE target.ccc -c -o message.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE message.ccc -c -o ctldev.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE ctldev.ccc -c -o log.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE log.ccc -c -o chap.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE chap.ccc -c -o event.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE event.ccc -c -o param.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE param.ccc -c -o config.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE config.ccc -c -o isns.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE isns.ccc -c -o md5.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE md5.ccc -c -o sha1.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE sha1.ccc iscsid.o iscsi_scstd.o conn.o session.o target.o message.o ctldev.o log.o chap.o event.o param.o config.o isns.o md5.o sha1.o -o iscsi-scstdcc -M -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsi_adm.c param.c >.depend_admcc -c -o iscsi_adm.o -O2 -fno-inline -Wall -Wextra -Wstrict-prototypes -Wno-sign-compare -Wimplicit-function-declaration -Wno-unused-parameter -Wno-missing-field-initializers -g -I../include -I/home/mockbuild/scst/iscsi-scst/../scst/include -D_GNU_SOURCE iscsi_adm.ccc iscsi_adm.o param.o -o iscsi-scst-admmake[2]: Leaving directory `/home/mockbuild/scst/iscsi-scst/usr'cp /home/mockbuild/scst/iscsi-scst/../scst/src/Module.symvers kernel/make -C /lib/modules/2.6.32-scst/build SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/include SUBDIRS=/home/mockbuild/scst/iscsi-scst/kernel modulesmake[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64' CC [M] /home/mockbuild/scst/iscsi-scst/kernel/iscsi.o CC [M] /home/mockbuild/scst/iscsi-scst/kernel/nthread.o CC [M] /home/mockbuild/scst/iscsi-scst/kernel/config.o CC [M] /home/mockbuild/scst/iscsi-scst/kernel/digest.o CC [M] /home/mockbuild/scst/iscsi-scst/kernel/conn.o CC [M] /home/mockbuild/scst/iscsi-scst/kernel/session.o CC [M] /home/mockbuild/scst/iscsi-scst/kernel/target.o CC [M] /home/mockbuild/scst/iscsi-scst/kernel/event.o CC [M] /home/mockbuild/scst/iscsi-scst/kernel/param.o LD [M] /home/mockbuild/scst/iscsi-scst/kernel/iscsi-scst.o Building modules, stage 2. MODPOST 1 modules CC /home/mockbuild/scst/iscsi-scst/kernel/iscsi-scst.mod.o LD [M] /home/mockbuild/scst/iscsi-scst/kernel/iscsi-scst.ko.unsigned NO SIGN [M] /home/mockbuild/scst/iscsi-scst/kernel/iscsi-scst.komake[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'make[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'cd iscsi-scst && make DESTDIR= installmake[1]: Entering directory `/home/mockbuild/scst/iscsi-scst'make -C usr SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/includemake[2]: Entering directory `/home/mockbuild/scst/iscsi-scst/usr'make[2]: Nothing to be done for `all'.make[2]: Leaving directory `/home/mockbuild/scst/iscsi-scst/usr'cp /home/mockbuild/scst/iscsi-scst/../scst/src/Module.symvers kernel/make -C /lib/modules/2.6.32-scst/build SCST_INC_DIR=/home/mockbuild/scst/iscsi-scst/../scst/include SUBDIRS=/home/mockbuild/scst/iscsi-scst/kernel modulesmake[2]: Entering directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64' Building modules, stage 2. MODPOST 1 modulesmake[2]: Leaving directory `/home/mockbuild/rpmbuild/BUILD/kernel-2.6.32-431.23.3.el6/linux-2.6.32-431.23.3.el6.x86_64'`usr/iscsi-scstd' -> `/usr/local/sbin/iscsi-scstd'install: creating directory `/usr/local/man'install: creating directory `/usr/local/man/man5'`doc/manpages/iscsi-scstd.conf.5' -> `/usr/local/man/man5/iscsi-scstd.conf.5'install: creating directory `/usr/local/man/man8'`doc/manpages/iscsi-scstd.8' -> `/usr/local/man/man8/iscsi-scstd.8'`usr/iscsi-scst-adm' -> `/usr/local/sbin/iscsi-scst-adm'`doc/manpages/iscsi-scst-adm.8' -> `/usr/local/man/man8/iscsi-scst-adm.8'`kernel/iscsi-scst.ko' -> `/lib/modules/2.6.32-scst/extra/iscsi-scst.ko'/sbin/depmod -aq 2.6.32-scstmake[1]: Leaving directory `/home/mockbuild/scst/iscsi-scst'Finally, issue “make scstadm scstadm_install” as well. This finalizes the compiling part. Do this as root. Commands and expected output are shown below. [root@oracle651 scst]# make scstadm scstadm_install | tee make_scst_install.logcd scstadmin && make allmake[1]: Entering directory `/home/mockbuild/scst/scstadmin'cd scstadmin && make allmake[2]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'Checking if your kit is complete...Looks goodWriting Makefile for SCST-SCSTmake -C scst-0.9.10make[3]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'cp lib/SCST/SCST.pm blib/lib/SCST/SCST.pmManifying blib/man3/SCST::SCST.3pmmake[3]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'make[2]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'make[1]: Leaving directory `/home/mockbuild/scst/scstadmin'cd scstadmin && make installmake[1]: Entering directory `/home/mockbuild/scst/scstadmin'cd scstadmin && make installmake[2]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'Writing Makefile for SCST-SCSTmake -C scst-0.9.10make[3]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'Skip blib/lib/SCST/SCST.pm (unchanged)Manifying blib/man3/SCST::SCST.3pmmake[3]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'make -C scst-0.9.10 install DESTDIR=make[3]: Entering directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'Manifying blib/man3/SCST::SCST.3pmInstalling /usr/local/share/perl5/SCST/SCST.pmInstalling /usr/local/share/man/man3/SCST::SCST.3pmAppending installation info to /usr/lib64/perl5/perllocal.podmake[3]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs/scst-0.9.10'install -d /usr/local/sbininstall -m 755 scstadmin /usr/local/sbinmake[2]: Leaving directory `/home/mockbuild/scst/scstadmin/scstadmin.sysfs'for m in scstadmin/man*/*.[1-9]; do \ if [ -e $m ]; then \ d=/usr/local/man/$(basename $(dirname $m)); \ if install -d $d; then \ install -m 644 $m $d; \ fi \ fi \ doneinstall -d /etc/init.dtmp=/tmp/scst-initd-$$ && init.d/scst >$tmp && \ install -m 755 $tmp /etc/init.d/scst && \ rm -f $tmpif [ ! -e /etc/default/scst ]; then \ mkdir -p /etc/default; \ install -m 755 default/scst /etc/default; \ fifor s in iscsi-scst qla2x00t; do \ { if [ -n "" ]; then chr="chroot "; else chr=""; fi; if type systemctl >/dev/null 2>&1; then $chr systemctl disable "$s.service"; elif type chkconfig >/dev/null 2>&1; then $chr chkconfig --del "$s"; elif type update-rc.d >/dev/null 2>&1; then $chr update-rc.d -f "$s" remove; elif type rc-update >/dev/null 2>&1; then $chr rc-update del "$s"; elif type /usr/lib/lsb/remove_initd >/dev/null 2>&1; then $chr /usr/lib/lsb/remove_initd "$s"; fi; rm -f /etc/init.d/$s; } >/dev/null 2>&1; \ doneechoecho 'If you want SCST to start automatically at boot time, run' \ 'the following command:'If you want SCST to start automatically at boot time, run the following command:if [ -n "" ]; then chr="chroot "; else chr=""; fi; if type systemctl >/dev/null 2>&1; then echo $chr systemctl enable "scst.service"; elif type chkconfig >/dev/null 2>&1; then echo $chr chkconfig --add "scst"; elif type update-rc.d >/dev/null 2>&1; then echo $chr update-rc.d "scst" defaults; elif type rc-update >/dev/null 2>&1; then echo $chr rc-update add "scst" default; elif type /usr/lib/lsb/install_initd >/dev/null 2>&1; then echo $chr /usr/lib/lsb/install_initd "scst"; fichkconfig --add scstechomake[1]: Leaving directory `/home/mockbuild/scst/scstadmin'[root@oracle651 scst]#Now set SCST to start on boot. Do this as root. Commands and expected output are shown below. [root@oracle651 scst]# /usr/lib/lsb/install_initd scst[root@oracle651 scst]# Now test start SCST manually. There is no configuration file yet for SCST, so it will give a warning as shown below. [root@oracle651 scst]# service scst startLoading and configuring SCSTSCST configuration file /etc/scst.conf missing [ OK ][root@oracle651 scst]# Now lock down the kernel so that the custom kernel just built will not be updated. This requires updated the /etc/yum.conf file as shown below. The line that needs to be added is shown in bold below "exclude=kernel*". This prevents the kernel from being updated. [root@oracle651 yum.repos.d]# cat /etc/yum.conf[main]cachedir=/var/cache/yum/$basearch/$releaseverkeepcache=0debuglevel=2logfile=/var/log/yum.logexactarch=1obsoletes=1gpgcheck=1plugins=1installonly_limit=3exclude=kernel*# This is the default, if you make this bigger yum won't see if the metadata# is newer on the remote and so you'll "gain" the bandwidth of not having to# download the new metadata and "pay" for it by yum not having correct# information.# It is esp. important, to have correct metadata, for distributions like# Fedora which don't keep old packages around. If you don't like this checking# interupting your command line usage, it's much better to have something# manually check the metadata once an hour (yum-updatesd will do this).# metadata_expire=90m# PUT YOUR REPOS HERE OR IN separate files named file.repo# in /etc/yum.repos.d[root@oracle651 yum.repos.d]# All that remains now is to reboot and test that it starts. Also, a configuration file must be created, which will be the next step. There was not a lot of documentation on the web about how to create a iSCSI SAN with SCST but I found a couple of very helpful posts here and here. Configuration of a simple 4K flash-aware SCST iSCSI SAN as shown below on the oracle651 KVM guest. [root@oracle651 ~]# modprobe scst [root@oracle651 ~]# modprobe scst_vdisk[root@oracle651 ~]# modprobe iscsi-scst[root@oracle651 ~]# scstadmin -add_target iqn.2014-08.oracle.asm.storage:luns -driver iscsi[root@oracle651 ~]# scstadmin -open_dev disk0 -handler vdisk_blockio -attributes filename=/dev/vdc,blocksize=4096[root@oracle651 ~]# scstadmin -open_dev disk1 -handler vdisk_blockio -attributes filename=/dev/vdb[root@oracle651 ~]# scstadmin -add_lun 0 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk0[root@oracle651 ~]# scstadmin -add_lun 1 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk1[root@oracle651 ~]# scstadmin -enable_target iqn.2014-08.oracle.asm.storage:luns -driver iscs[root@oracle651 ~]# scstadmin -set_drv_attr iscsi -attributes enabled=1[root@oracle651 ~]# scstadmin -write_config /etc/scst.confThe /etc/scst.conf file generated by the above command is shown below. Notice the "blocksize 4096" parameter which can be used. [root@oracle651 ~]# cat /etc/scst.conf# Automatically generated by SCST Configurator v2.2.1.HANDLER vdisk_blockio { DEVICE disk0 { blocksize 4096 filename /dev/vdc } DEVICE disk1 { filename /dev/vdb }}TARGET_DRIVER iscsi { enabled 1 TARGET iqn.2014-08.oracle.asm.storage:luns { cpu_mask 3 enabled 1 rel_tgt_id 1 LUN 0 disk0 LUN 1 disk1 }}[root@oracle651 ~]# cat /etc/sysconfig/selinuxIn order to get the initiator over on the iSCSI client on oracle631 KVM guest to see the portal, stop iptables and disable it on reboot as shown below and also disable selinux as shown below and turn off iptables on the iSCSI SAN host. [root@oracle651 ~]# cat /etc/sysconfig/selinux# This file controls the state of SELinux on the system.# SELINUX= can take one of these three values:# enforcing - SELinux security policy is enforced.# permissive - SELinux prints warnings instead of enforcing.# disabled - No SELinux policy is loaded.SELINUX=disabled# SELINUXTYPE= can take one of these two values:# targeted - Targeted processes are protected,# mls - Multi Level Security protection.SELINUXTYPE=targeted [root@oracle651 ~]# service iptables stop[root@oracle651 ~]# chkconfig iptables off[root@oracle651 ~]# rebootFinally, use iscsiadm on the oracle631 KVM guest to discover and configure the access to the SCST SAN on oracle651 KVM guest. The iscsi-initiator utils are installed as shown below, and then the connection to the SAN is configured as shown. 10.207.39.74:3260,1 iqn.2014-08.oracle.asm.storage:luns[root@oracle631 ~]# iscsiadm -m node -T iqn.2014-08.oracle.asm.storage:luns -p 10.207.39.74 --op update -n node.startup -v automatic[root@oracle631 ~]# iscsiadm --mode node --targetname iqn.2014-08.oracle.asm.storage:luns --portal 10.207.39.74 --loginLogging in to [iface: default, target: iqn.2014-08.oracle.asm.storage:luns, portal: 10.207.39.74,3260] (multiple)Login to [iface: default, target: iqn.2014-08.oracle.asm.storage:luns, portal: 10.207.39.74,3260] successful.Here are the LUNs over on the client side and show of their characteristics on oracle631 host as shown below. One very interesting and intriguing feature is that SCST sees the 512 emulated LUN on oracle631 KVM guest (the client) for what it really is i.e (512 logical / 4096 physical) even though the virtual disk on the oracle651 KVM guest (the SAN) sees that LUN as (512 logical / 512 physical), as shown below. [root@oracle631 ~]# ls -lrt /dev/sd*brw-rw---- 1 root disk 8, 0 Aug 11 20:11 /dev/sdabrw-rw---- 1 root disk 8, 16 Aug 11 20:11 /dev/sdb[root@oracle631 ~]# scsi_id -g -u -d /dev/sda26162356632353435[root@oracle631 ~]# scsi_id -g -u -d /dev/sdb26238666462643332[root@oracle631 ~]# fdisk -l /dev/sdaNote: sector size is 4096 (not 512)Disk /dev/sda: 10.7 GB, 10737418240 bytes64 heads, 32 sectors/track, 1280 cylindersUnits = cylinders of 2048 * 4096 = 8388608 bytesSector size (logical/physical): 4096 bytes / 4096 bytesI/O size (minimum/optimal): 4096 bytes / 524288 bytesDisk identifier: 0x00000000[root@oracle631 ~]# fdisk -l /dev/sdbDisk /dev/sdb: 10.7 GB, 10737418240 bytes64 heads, 32 sectors/track, 10240 cylindersUnits = cylinders of 2048 * 512 = 1048576 bytesSector size (logical/physical): 512 bytes / 4096 bytes <-- Notice I/O size (minimum/optimal): 4096 bytes / 524288 bytesDisk identifier: 0x00000000[root@oracle631 ~]# Here are the properties of the same LUNs over on the SAN oracle651 as shown below. [root@oracle651 ~]# cat /etc/scst.conf# Automatically generated by SCST Configurator v2.2.1.HANDLER vdisk_blockio { DEVICE disk0 { blocksize 4096 filename /dev/vdc } DEVICE disk1 { filename /dev/vdb }}TARGET_DRIVER iscsi { enabled 1 TARGET iqn.2014-08.oracle.asm.storage:luns { cpu_mask 3 enabled 1 rel_tgt_id 1 LUN 0 disk0 LUN 1 disk1 }}[root@oracle651 ~]# fdisk -l /dev/vdcNote: sector size is 4096 (not 512)Disk /dev/vdc: 10.7 GB, 10737418240 bytes16 heads, 56 sectors/track, 2925 cylindersUnits = cylinders of 896 * 4096 = 3670016 bytesSector size (logical/physical): 4096 bytes / 4096 bytesI/O size (minimum/optimal): 4096 bytes / 4096 bytesDisk identifier: 0x00000000[root@oracle651 ~]# fdisk -l /dev/vdbDisk /dev/vdb: 10.7 GB, 10737418240 bytes16 heads, 63 sectors/track, 20805 cylindersUnits = cylinders of 1008 * 512 = 516096 bytesSector size (logical/physical): 512 bytes / 512 bytes <-- NoticeI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x00000000[root@oracle651 ~]# Recall that the blocksize defaults in KVM to (logical/physical) 512 bytes / 512 bytes unless it is set otherwise in the XML file of the KVM guest, as shown below i the XML file that defines oracle651 the KVM guest which is the SAN provider. Looking at the end of the XML file at bottom it can be seen that only disk2 has been set to specified blocksizes. Therefore, disk1 is presented to the KVM guest oracle651 as a 512 bytes / 512 bytes LUN by default. Thus it is surprising that SCST sees it for what it is 'physically' which is a LUN sitting on a consumer-grade SSD (Crucial M550) and flash always uses 4096 byte physical sectors, and 512 byte logical emulation. The LUN as seen by oracle631 the iscsi client sees it as what it 'really' is, 512 bytes / 4096 bytes (logical/physical). More investigation is warranted to understand this better, but it would seem that SCST is ignoring the KVM setting and reading the storage itself, and that the blocksize setting in the SCST configuration file refers to the logical block size, and thus set the logical block size of the other LUN to 4096 bytes (the LUN that was presented by KVM on oracle631 as (logical/physical) 4096 bytes / 4096 bytes. gstanden@vmem1:~$ virsh dumpxml oracle651<domain type='kvm' id='6' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>oracle651</name> <uuid>9a017de6-d362-4269-90fe-09331460aca0</uuid> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type> <boot dev='hd'/> <boot dev='cdrom'/> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/kvm-spice</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source file='/var/lib/libvirt/images/oracle651-0.img'/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source file='/var/lib/libvirt/images/oracle651-1.img'/> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source file='/var/lib/libvirt/images/oracle651-2.img'/> <target dev='vdc' bus='virtio'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <target dev='hdc' bus='ide' tray='open'/> <readonly/> <alias name='ide0-1-0'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <controller type='usb' index='0'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <interface type='bridge'> <mac address='52:54:00:66:26:a0'/> <source bridge='sw1'/> <virtualport type='openvswitch'> <parameters interfaceid='0efa5f2a-cbf2-46df-9ff2-66bddb09218d'/> </virtualport> <target dev='s4'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/12'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/12'> <source path='/dev/pts/12'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> </graphics> <sound model='ich6'> <alias name='sound0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='cirrus' vram='9216' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='apparmor' relabel='yes'> <label>libvirt-9a017de6-d362-4269-90fe-09331460aca0</label> <imagelabel>libvirt-9a017de6-d362-4269-90fe-09331460aca0</imagelabel> </seclabel> <qemu:commandline> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk0.scsi=off'/> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk0.config-wce=off'/> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk0.x-data-plane=on'/> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk1.scsi=off'/> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk1.config-wce=off'/> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk1.x-data-plane=on'/> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk2.scsi=off'/> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk2.config-wce=off'/> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk2.x-data-plane=on'/> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk2.logical_block_size=4096'/> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk2.physical_block_size=4096'/> </qemu:commandline></domain>gstanden@vmem1:~$Some useful additonal info about the LUNs over on the oracle651 client. [root@oracle631 disk]# ls -lrt by-path | grep asmlrwxrwxrwx 1 root root 9 Aug 11 20:11 ip-10.207.39.74:3260-iscsi-iqn.2014-08.oracle.asm.storage:luns-lun-0 -> ../../sdalrwxrwxrwx 1 root root 9 Aug 11 20:11 ip-10.207.39.74:3260-iscsi-iqn.2014-08.oracle.asm.storage:luns-lun-1 -> ../../sdb[root@oracle631 disk]# ls -lrt by-id | grep scsilrwxrwxrwx 1 root root 9 Aug 11 20:11 scsi-26162356632353435 -> ../../sdalrwxrwxrwx 1 root root 9 Aug 11 20:11 scsi-26238666462643332 -> ../../sdb[root@oracle631 disk]# More useful information, including the vendor name (for use in the multipath.conf file) as shown below. [root@oracle631 disk]# cat /proc/scsi/scsiAttached devices:Host: scsi1 Channel: 00 Id: 00 Lun: 00 Vendor: QEMU Model: QEMU DVD-ROM Rev: 2.0. Type: CD-ROM ANSI SCSI revision: 05Host: scsi2 Channel: 00 Id: 00 Lun: 00 Vendor: SCST_BIO Model: disk0 Rev: 221 Type: Direct-Access ANSI SCSI revision: 05Host: scsi2 Channel: 00 Id: 00 Lun: 01 Vendor: SCST_BIO Model: disk1 Rev: 221 Type: Direct-Access ANSI SCSI revision: 05[root@oracle631 disk]# And yet more information about the SCST iSCSI LUN /dev/sda shown below using udevadm command. [root@oracle631 disk]# udevadm info -a -p $(udevadm info -q path -n /dev/sda)Udevadm info starts with the device specified by the devpath and thenwalks up the chain of parent devices. It prints for every devicefound, all possible attributes in the udev rules key format.A rule to match, can be composed by the attributes of the deviceand the attributes from one single parent device. looking at device '/devices/platform/host2/session1/target2:0:0/2:0:0:0/block/sda': KERNEL=="sda" SUBSYSTEM=="block" DRIVER=="" ATTR{ro}=="0" ATTR{size}=="20971520" ATTR{stat}==" 455 0 3648 329 0 0 0 0 0 328 328" ATTR{range}=="16" ATTR{discard_alignment}=="0" ATTR{events}=="" ATTR{ext_range}=="256" ATTR{events_poll_msecs}=="-1" ATTR{alignment_offset}=="0" ATTR{inflight}==" 0 0" ATTR{removable}=="0" ATTR{capability}=="50" ATTR{events_async}=="" looking at parent device '/devices/platform/host2/session1/target2:0:0/2:0:0:0': KERNELS=="2:0:0:0" SUBSYSTEMS=="scsi" DRIVERS=="sd" ATTRS{rev}==" 221" ATTRS{type}=="0" ATTRS{scsi_level}=="6" ATTRS{model}=="disk0 " ATTRS{state}=="running" ATTRS{queue_type}=="none" ATTRS{modalias}=="scsi:t-0x00" ATTRS{iodone_cnt}=="0x212" ATTRS{iorequest_cnt}=="0x212" ATTRS{queue_ramp_up_period}=="120000" ATTRS{timeout}=="30" ATTRS{evt_media_change}=="0" ATTRS{ioerr_cnt}=="0x2" ATTRS{queue_depth}=="32" ATTRS{vendor}=="SCST_BIO" ATTRS{device_blocked}=="0" ATTRS{dh_state}=="detached" ATTRS{iocounterbits}=="32" ATTRS{eh_timeout}=="10" looking at parent device '/devices/platform/host2/session1/target2:0:0': KERNELS=="target2:0:0" SUBSYSTEMS=="scsi" DRIVERS=="" looking at parent device '/devices/platform/host2/session1': KERNELS=="session1" SUBSYSTEMS=="" DRIVERS=="" looking at parent device '/devices/platform/host2': KERNELS=="host2" SUBSYSTEMS=="scsi" DRIVERS=="" looking at parent device '/devices/platform': KERNELS=="platform" SUBSYSTEMS=="" DRIVERS==""And now for /dev/sdb shown below using udevadm command. [root@oracle631 disk]# udevadm info -a -p $(udevadm info -q path -n /dev/sdb)Udevadm info starts with the device specified by the devpath and thenwalks up the chain of parent devices. It prints for every devicefound, all possible attributes in the udev rules key format.A rule to match, can be composed by the attributes of the deviceand the attributes from one single parent device. looking at device '/devices/platform/host2/session1/target2:0:0/2:0:0:1/block/sdb': KERNEL=="sdb" SUBSYSTEM=="block" DRIVER=="" ATTR{ro}=="0" ATTR{size}=="20971520" ATTR{stat}==" 455 0 3664 337 0 0 0 0 0 337 337" ATTR{range}=="16" ATTR{discard_alignment}=="0" ATTR{events}=="" ATTR{ext_range}=="256" ATTR{events_poll_msecs}=="-1" ATTR{alignment_offset}=="0" ATTR{inflight}==" 0 0" ATTR{removable}=="0" ATTR{capability}=="50" ATTR{events_async}=="" looking at parent device '/devices/platform/host2/session1/target2:0:0/2:0:0:1': KERNELS=="2:0:0:1" SUBSYSTEMS=="scsi" DRIVERS=="sd" ATTRS{rev}==" 221" ATTRS{type}=="0" ATTRS{scsi_level}=="6" ATTRS{model}=="disk1 " ATTRS{state}=="running" ATTRS{queue_type}=="none" ATTRS{modalias}=="scsi:t-0x00" ATTRS{iodone_cnt}=="0x211" ATTRS{iorequest_cnt}=="0x211" ATTRS{queue_ramp_up_period}=="120000" ATTRS{timeout}=="30" ATTRS{evt_media_change}=="0" ATTRS{ioerr_cnt}=="0x2" ATTRS{queue_depth}=="32" ATTRS{vendor}=="SCST_BIO" ATTRS{device_blocked}=="0" ATTRS{dh_state}=="detached" ATTRS{iocounterbits}=="32" ATTRS{eh_timeout}=="10" looking at parent device '/devices/platform/host2/session1/target2:0:0': KERNELS=="target2:0:0" SUBSYSTEMS=="scsi" DRIVERS=="" looking at parent device '/devices/platform/host2/session1': KERNELS=="session1" SUBSYSTEMS=="" DRIVERS=="" looking at parent device '/devices/platform/host2': KERNELS=="host2" SUBSYSTEMS=="scsi" DRIVERS=="" looking at parent device '/devices/platform': KERNELS=="platform" SUBSYSTEMS=="" DRIVERS==""Additional information after the configuration of the SCST LUNs on oracle651 using device-mapper-multipath and /etc/udev/rules.d/12-dm-permissions.rules as shown below. Friendly names set from the /etc/multipath.conf file correctly. [root@oracle631 ~]# ls -lrt /dev/mappertotal 0crw-rw---- 1 root root 10, 236 Aug 11 21:30 controllrwxrwxrwx 1 root root 7 Aug 11 21:30 vg_oracle631-lv_swap -> ../dm-1lrwxrwxrwx 1 root root 7 Aug 11 21:30 vg_oracle631-lv_root -> ../dm-0lrwxrwxrwx 1 root root 7 Aug 11 21:30 vg_u00-lv_oracle -> ../dm-2lrwxrwxrwx 1 root root 7 Aug 11 21:30 ASM_4096B_1 -> ../dm-3lrwxrwxrwx 1 root root 7 Aug 11 21:30 ASM_0512B_1 -> ../dm-4Friendly names accurately indicate logical block sizes as shown below. [root@oracle631 ~]# fdisk -l /dev/dm-3 <--ASM_4096B_1Note: sector size is 4096 (not 512)Disk /dev/dm-3: 10.7 GB, 10737418240 bytes255 heads, 63 sectors/track, 163 cylindersUnits = cylinders of 16065 * 4096 = 65802240 bytesSector size (logical/physical): 4096 bytes / 4096 bytes <-- 4096 logical sector size I/O size (minimum/optimal): 4096 bytes / 524288 bytesDisk identifier: 0x00000000[root@oracle631 ~]# fdisk -l /dev/dm-4Disk /dev/dm-4: 10.7 GB, 10737418240 bytes <--ASM_0512B_1255 heads, 63 sectors/track, 1305 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesSector size (logical/physical): 512 bytes / 4096 bytes <-- 512 logical sector sizeI/O size (minimum/optimal): 4096 bytes / 524288 bytesDisk identifier: 0x00000000[root@oracle631 ~]#Permissions set correctly from the /etc/udev/rules.d/12-dm-permissions.rules file. [root@oracle631 ~]# ls -lrt /dev/dm-3brw-rw---- 1 grid asmadmin 252, 3 Aug 11 21:30 /dev/dm-3[root@oracle631 ~]# ls -lrt /dev/dm-4brw-rw---- 1 grid asmadmin 252, 4 Aug 11 21:30 /dev/dm-4The /etc/multipath.conf file for the SCST LUNs on oracle631 as shown below. [root@oracle631 ~]# cat /etc/multipath.conf# Configuration file for device mapper multipath.# For a complete list of the default configuration values:# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults# For a list of configuration options with descriptions:# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotateddefaults { user_friendly_names yes}blacklist { devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^hd[a-z]"}devices { device { vendor "SCST_BIO" product "Direct-Access" path_grouping_policy multibus path_grouping_policy group_by_prio path_selector "round-robin 0" failback immediate rr_min_io 1 rr_weight priorities }}multipaths { multipath { wwid 26162356632353435 alias ASM_4096B_1 } multipath { wwid 26238666462643332 alias ASM_0512B_1 }}[root@oracle631 ~]# Listing of the /etc/udev/rules.d/12-dm-permissions.rules file on oracle631 the scsi client as shown below with section that handles the above LUNs bolded (near bottom of listing). [root@oracle631 ~]# cat /etc/udev/rules.d/12-dm-permissions.rules # Copyright (C) 2009 Red Hat, Inc. All rights reserved.## This file is part of LVM2.# Udev rules for device-mapper devices.## These rules set permissions for DM devices.## This file is considered to be a template where users can put their# own entries and then put a copy of it manually to a usual place with# user-edited udev rules (usually /etc/udev/rules.d).## There are some environment variables set that can be used:# DM_UDEV_RULES_VSN - DM udev rules version# DM_NAME - actual DM device's name# DM_UUID - UUID set for DM device (blank if not specified)# DM_SUSPENDED - suspended state of DM device (0 or 1)# DM_LV_NAME - logical volume name (not set if LVM device not present)# DM_VG_NAME - volume group name (not set if LVM device not present)# DM_LV_LAYER - logical volume layer (not set if LVM device not present)# "add" event is processed on coldplug only!ACTION!="add|change", GOTO="dm_end"ENV{DM_UDEV_RULES_VSN}!="?*", GOTO="dm_end"# A few demonstrational examples...# PLAIN DM DEVICES## Set permissions for a DM device named 'my_device' exactly# ENV{DM_NAME}=="my_device", OWNER:="root", GROUP:="root", MODE:="660"# Set permissions for all DM devices having 'MY_UUID-' UUID prefix# ENV{DM_UUID}=="MY_UUID-?*", OWNER:="root", GROUP:="root", MODE:="660"# LVM DEVICES## Set permissions for all LVM devices# ENV{DM_UUID}=="LVM-?*", OWNER:="root", GROUP:="root", MODE:="660"# Set permissions for all devices that belong to one LVM VG# ENV{DM_VG_NAME}=="VolGroup00", OWNER:="root", GROUP:="root", MODE:="660"# Set permissions for an LVM device with VG named VolGroup00 and LV named LogVol00 exactly# ENV{DM_VG_NAME}=="VolGroup00", ENV{DM_LV_NAME}=="LogVol00", OWNER:="root", GROUP:="root", MODE:="660"# Set permissions for all LVM devices that does not belong to a VG named VolGroup00# ENV{DM_VG_NAME}!="VolGroup00", OWNER:="root", GROUP:="root", MODE:="660"# ENCRYPTED DEVICES (using cryptsetup >= 1.1)## Set permissions for all encrypted devices created by cryptsetup (plain devices)# ENV{DM_UUID}=="CRYPT-PLAIN-?*", OWNER:="root", GROUP:="root", MODE:="660"# Set permissions for all encrypted devices created by cryptsetup (LUKS extension)# ENV{DM_UUID}=="CRYPT-LUKS1-?*", OWNER:="root", GROUP:="root", MODE:="660"# Set permissions for an encrypted device created by cryptsetup and having an exact luks UUID# ENV{DM_UUID}=="CRYPT-LUKS1-22fce5c8313c43c68d84b50a3b0fee78-?*", OWNER:="root", GROUP:="root", MODE:="660"# MULTIPATH DEVICES## Set permissions for all multipath devices# ENV{DM_UUID}=="mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"ENV{DM_UUID}=="mpath-26162356632353435", OWNER:="grid", GROUP:="asmadmin", MODE:="660" # <-- LUN modes are set hereENV{DM_UUID}=="mpath-26238666462643332", OWNER:="grid", GROUP:="asmadmin", MODE:="660" # <-- LUN modes are set here# Set permissions for first two partitions created on a multipath device (and detected by kpartx)# ENV{DM_UUID}=="part[1-2]-mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"# ...you can use any combination of the comparisons with the environment variables# listed at the beginning of this file (udev provides simple pattern matching by# using *, ? and [] that you can use, see 'man udev' for more information).# Set default permissions for all DM devices if not set before.# OWNER:="root", GROUP:="root", MODE:="660"LABEL="dm_end"[root@oracle631 ~]# If the console terminal is running on the KVM guest SCST SAN host oracle651, then when a SAN client oracle631 is shutdown, the following message is buffered to console and written to dmesg as shown below. [root@oracle651 ~]# iscsi-scst: Logout received from initiator iqn.1988-12.com.oracle:49c39d4d389discsi-scst: Closing connection at initiator's iqn.1988-12.com.oracle:49c39d4d389d requestscst: TM fn 6 (ffff88003dfd50b8)scst: TM fn 6 (ffff88003dfd50b8) finished, status 0scst: Processing thread disk00_0 (PID 1783) finishedscst: Processing thread disk10_0 (PID 1789) finished[root@oracle651 ~]# exitlogoutOracle Linux Server release 6.5Kernel 2.6.32-scst on an x86_64oracle651.vmem.org login: Here is an example of the commands for adding a file-backed LUN rather than a block device as shown below. [root@oracle651 ~] dd if=/dev/zero of=/root/scst-scripts/asm_disk1.img bs=512 count=0 seek=1M[root@oracle651 ~] scstadmin -open_dev disk2 -handler vdisk_fileio -attributes filename=/root/scst-scripts/asm_disk1.img,blocksize=4096[root@oracle651 ~] scstadmin -add_lun 2 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk2[root@oracle651 ~] scstadmin -write_config /etc/scst.confThe last command saves it to the /etc/scst.conf configuration file as shown below. The "vdisk_fileio" handler has been added, as well as the LUN. [root@oracle651 scst-scripts]# cat /etc/scst.conf# Automatically generated by SCST Configurator v2.2.1.HANDLER vdisk_blockio { DEVICE disk0 { blocksize 4096 filename /dev/vdc } DEVICE disk1 { filename /dev/vdb }}HANDLER vdisk_fileio { DEVICE disk2 { blocksize 4096 filename /root/scst-scripts/asm_disk1.img }}TARGET_DRIVER iscsi { enabled 1 TARGET iqn.2014-08.oracle.asm.storage:luns { cpu_mask 3 enabled 1 rel_tgt_id 1 LUN 0 disk0 LUN 1 disk1 LUN 2 disk2 }}[root@oracle651 scst-scripts]# Here is an example of removing the LUN that was just added. Notice that the HANDLER information for the deleted LUN remains, but the TARGET DRIVER LUN list is updated and the removed LUN has been removed from the list in /etc/scst.conf as shown below. [root@oracle651 scst-scripts]# scstadmin -rem_lun 2 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk2Collecting current configuration: done.Performing this action may result in lost or corrupt data, are you sure you wish to continue (y/[n]) ? y-> Making requested changes. -> Removing LUN 2 from driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done.Writing current configuration to file '/etc/scst.conf'All done.[root@oracle651 scst-scripts]# scstadmin -write_config /etc/scst.confCollecting current configuration: done.Writing current configuration to file '/etc/scst.conf'.. All done.[root@oracle651 scst-scripts]# cat /etc/scst.conf# Automatically generated by SCST Configurator v2.2.1.HANDLER vdisk_blockio { DEVICE disk0 { blocksize 4096 filename /dev/vdc } DEVICE disk1 { filename /dev/vdb }}HANDLER vdisk_fileio { <-- disk2 handler is still defined after LUN 2 was dropped from target. DEVICE disk2 { blocksize 4096 filename /root/scst-scripts/asm_disk1.img }}TARGET_DRIVER iscsi { enabled 1 TARGET iqn.2014-08.oracle.asm.storage:luns { cpu_mask 3 enabled 1 rel_tgt_id 1 LUN 0 disk0 LUN 1 disk1 } <-- LUN 2 disk2 is removed here. }[root@oracle651 scst-scripts]# historyThus, the LUN can be re-added back with just the one command below if it is needed back in the target as shown below. [root@oracle651 ~]# scstadmin -add_lun 2 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device disk2Collecting current configuration: done.-> Making requested changes. -> Adding device 'disk2' at LUN 2 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done. -> Driver/target is not a fibre channel target, ignoring. -> Done.All done.[root@oracle651 ~]# scstadmin -write_config /etc/scst.confCollecting current configuration: done.Writing current configuration to file '/etc/scst.conf'.. All done.[root@oracle651 ~]# View /etc/scst.conf again and see that the LUN has been added back to the TARGET as shown below. [root@oracle651 ~]# cat /etc/scst.conf# Automatically generated by SCST Configurator v2.2.1.HANDLER vdisk_blockio { DEVICE disk0 { blocksize 4096 filename /dev/vdc } DEVICE disk1 { filename /dev/vdb }}HANDLER vdisk_fileio { DEVICE disk2 { blocksize 4096 filename /root/scst-scripts/asm_disk1.img }}TARGET_DRIVER iscsi { enabled 1 TARGET iqn.2014-08.oracle.asm.storage:luns { cpu_mask 3 enabled 1 rel_tgt_id 1 LUN 0 disk0 LUN 1 disk1 LUN 2 disk2 }}[root@oracle651 ~]# Examples of adding file-backed SCST LUNs to the OS of the SCST OEL host are shown below. The example below shows adding the 8 LUNs per the Violin Memory Oracle Best Practice Document dated April 2014 to the SCST SAN configuration. [root@oracle651 ~]# scstadmin -add_lun 11 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco1Collecting current configuration: done.-> Making requested changes. -> Adding device 'asm_reco1' at LUN 11 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done. -> Driver/target is not a fibre channel target, ignoring. -> Done.All done.[root@oracle651 ~]# scstadmin -add_lun 12 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco2Collecting current configuration: done.-> Making requested changes. -> Adding device 'asm_reco2' at LUN 12 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done. -> Driver/target is not a fibre channel target, ignoring. -> Done.All done.[root@oracle651 ~]# scstadmin -add_lun 13 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco3Collecting current configuration: done.-> Making requested changes. -> Adding device 'asm_reco3' at LUN 13 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done. -> Driver/target is not a fibre channel target, ignoring. -> Done.All done.[root@oracle651 ~]# scstadmin -add_lun 14 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco4Collecting current configuration: done.-> Making requested changes. -> Adding device 'asm_reco4' at LUN 14 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done. -> Driver/target is not a fibre channel target, ignoring. -> Done.All done.[root@oracle651 ~]# scstadmin -add_lun 15 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco5Collecting current configuration: done.-> Making requested changes. -> Adding device 'asm_reco5' at LUN 15 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done. -> Driver/target is not a fibre channel target, ignoring. -> Done.All done.[root@oracle651 ~]# scstadmin -add_lun 16 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco6Collecting current configuration: done.-> Making requested changes. -> Adding device 'asm_reco6' at LUN 16 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done. -> Driver/target is not a fibre channel target, ignoring. -> Done.All done.[root@oracle651 ~]# scstadmin -add_lun 17 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco7Collecting current configuration: done.-> Making requested changes. -> Adding device 'asm_reco7' at LUN 17 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done. -> Driver/target is not a fibre channel target, ignoring. -> Done.All done.[root@oracle651 ~]# scstadmin -add_lun 18 -driver iscsi -target iqn.2014-08.oracle.asm.storage:luns -device asm_reco8Collecting current configuration: done.-> Making requested changes. -> Adding device 'asm_reco8' at LUN 18 to driver/target 'iscsi/iqn.2014-08.oracle.asm.storage:luns': done. -> Driver/target is not a fibre channel target, ignoring. -> Done.All done.[root@oracle651 ~]# Now write the configuration changes to the /etc/scst.conf configuration file as shown below. [root@oracle651 ~]# scstadmin -write_config /etc/scst.confCollecting current configuration: done.Writing current configuration to file '/etc/scst.conf'.. All done.[root@oracle651 ~]# cat /etc/scst.conf# Automatically generated by SCST Configurator v2.2.1.HANDLER vdisk_blockio { DEVICE disk0 { blocksize 4096 filename /dev/vdc } DEVICE disk1 { filename /dev/vdb }}HANDLER vdisk_fileio { DEVICE asm_data1 { filename /scst/asm_data1.img } DEVICE asm_data2 { filename /scst/asm_data2.img } DEVICE asm_data3 { filename /scst/asm_data3.img } DEVICE asm_data4 { filename /scst/asm_data4.img } DEVICE asm_data5 { filename /scst/asm_data5.img } DEVICE asm_data6 { filename /scst/asm_data6.img } DEVICE asm_data7 { filename /scst/asm_data7.img } DEVICE asm_data8 { filename /scst/asm_data8.img } DEVICE asm_reco1 { filename /scst/asm_reco1.img } DEVICE asm_reco2 { filename /scst/asm_reco2.img } DEVICE asm_reco3 { filename /scst/asm_reco3.img } DEVICE asm_reco4 { filename /scst/asm_reco4.img } DEVICE asm_reco5 { filename /scst/asm_reco5.img } DEVICE asm_reco6 { filename /scst/asm_reco6.img } DEVICE asm_reco7 { filename /scst/asm_reco7.img } DEVICE asm_reco8 { filename /scst/asm_reco8.img } DEVICE systemdg1 { filename /scst/systemdg1.img }}TARGET_DRIVER iscsi { enabled 1 TARGET iqn.2014-08.oracle.asm.storage:luns { cpu_mask 3 enabled 1 rel_tgt_id 1 LUN 0 disk0 LUN 1 disk1 LUN 2 systemdg1 LUN 3 asm_data1 LUN 4 asm_data2 LUN 5 asm_data3 LUN 6 asm_data4 LUN 7 asm_data5 LUN 8 asm_data6 LUN 9 asm_data7 LUN 10 asm_data8 LUN 11 asm_reco1 LUN 12 asm_reco2 LUN 13 asm_reco3 LUN 14 asm_reco4 LUN 15 asm_reco5 LUN 16 asm_reco6 LUN 17 asm_reco7 LUN 18 asm_reco8 }}[root@oracle651 ~]# Adding the new LUNs to the Oracle database server is shown below. As soon as the above command to add the LUNs to the target is completed, the LUNs will immediately appear on the the SCSI client machine - the database server - (as long as it is logged in to the SAN) and will be visible as shown below using the multipath command. [root@oracle631 ~]# multipath -ll -v2 | grep recompathr (23734383163616238) dm-17 SCST_FIO,asm_reco1mpathq (23239613933333133) dm-16 SCST_FIO,asm_reco4mpathp (23533633466613536) dm-14 SCST_FIO,asm_reco3mpatho (23430363636323231) dm-13 SCST_FIO,asm_reco2mpathv (26661333739313737) dm-21 SCST_FIO,asm_reco8mpathu (23164346539623861) dm-20 SCST_FIO,asm_reco7mpatht (2656563303366642d) dm-19 SCST_FIO,asm_reco6mpaths (23361306261623634) dm-18 SCST_FIO,asm_reco5[root@oracle631 ~]# Use the output above the /etc/multipath.conf can be edited. [root@oracle631 ~]# vi /etc/multipath.conf[root@oracle631 ~]# cat /etc/multipath.conf# Configuration file for device mapper multipath.# For a complete list of the default configuration values:# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults# For a list of configuration options with descriptions:# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotateddefaults { user_friendly_names yes}blacklist { devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^hd[a-z]"}devices { device { vendor "SCST_BIO" product "Direct-Access" path_grouping_policy multibus path_selector "round-robin 0" failback immediate rr_min_io 1 rr_weight priorities }}multipaths { multipath { wwid 26162356632353435 alias ASM_4096B_1 } multipath { wwid 26238666462643332 alias ASM_0512B_1 } multipath { wwid 23863316131356162 alias ASM_4096F_1 } multipath { wwid 23233626538613166 alias ASM_0512F_1 } multipath { wwid 26161383334393638 alias ASM_DATA8 } multipath { wwid 23464666134333935 alias ASM_DATA7 } multipath { wwid 23565353864626532 alias ASM_DATA6 } multipath { wwid 23661626637333762 alias ASM_DATA5 } multipath { wwid 23739316465623063 alias ASM_DATA4 } multipath { wwid 2333730323234392d alias ASM_DATA3 } multipath { wwid 23130643262613365 alias ASM_DATA2 } multipath { wwid 23234333531326137 alias ASM_DATA1 } multipath { wwid 23734383163616238 alias ASM_RECO1 } multipath { wwid 23430363636323231 alias ASM_RECO2 } multipath { wwid 23533633466613536 alias ASM_RECO3 } multipath { wwid 23239613933333133 alias ASM_RECO4 } multipath { wwid 23361306261623634 alias ASM_RECO5 } multipath { wwid 2656563303366642d alias ASM_RECO6 } multipath { wwid 23164346539623861 alias ASM_RECO7 } multipath { wwid 26661333739313737 alias ASM_RECO8 }}[root@oracle631 ~]# Check the permissions, ownership and group of the LUNs to be sure they are correct. With these iSCSI LUNs from SCST normally the add instantly to the client machine with correct permissions and modes and no reboot is needed. The only step required is to edit the /etc/multipath.conf file and restart multipath as shown below. [root@oracle631 ~]# ls -lrt /dev/mappertotal 0crw-rw---- 1 root root 10, 236 Aug 13 10:17 controllrwxrwxrwx 1 root root 7 Aug 13 10:17 vg_oracle631-lv_swap -> ../dm-1lrwxrwxrwx 1 root root 7 Aug 13 10:17 vg_oracle631-lv_root -> ../dm-0lrwxrwxrwx 1 root root 7 Aug 13 10:17 vg_u00-lv_oracle -> ../dm-2lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_0512B_1 -> ../dm-7lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA1 -> ../dm-5lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_0512F_1 -> ../dm-4lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA6 -> ../dm-10lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA3 -> ../dm-8lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA4 -> ../dm-6lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_DATA5 -> ../dm-9lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA8 -> ../dm-11lrwxrwxrwx 1 root root 7 Aug 13 10:38 ASM_4096B_1 -> ../dm-3lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO1 -> ../dm-17lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA7 -> ../dm-12lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_DATA2 -> ../dm-15lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO4 -> ../dm-16lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO2 -> ../dm-13lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO3 -> ../dm-14lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO6 -> ../dm-19lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO8 -> ../dm-21lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO5 -> ../dm-18lrwxrwxrwx 1 root root 8 Aug 13 10:38 ASM_RECO7 -> ../dm-20[root@oracle631 ~]# ls -lrt /dev/dm-*brw-rw---- 1 root disk 252, 1 Aug 13 10:17 /dev/dm-1brw-rw---- 1 root disk 252, 0 Aug 13 10:17 /dev/dm-0brw-rw---- 1 root disk 252, 2 Aug 13 10:17 /dev/dm-2brw-rw---- 1 grid asmadmin 252, 7 Aug 13 10:38 /dev/dm-7brw-rw---- 1 grid asmadmin 252, 5 Aug 13 10:38 /dev/dm-5brw-rw---- 1 grid asmadmin 252, 4 Aug 13 10:38 /dev/dm-4brw-rw---- 1 grid asmadmin 252, 10 Aug 13 10:38 /dev/dm-10brw-rw---- 1 grid asmadmin 252, 8 Aug 13 10:38 /dev/dm-8brw-rw---- 1 grid asmadmin 252, 6 Aug 13 10:38 /dev/dm-6brw-rw---- 1 grid asmadmin 252, 9 Aug 13 10:38 /dev/dm-9brw-rw---- 1 grid asmadmin 252, 3 Aug 13 10:38 /dev/dm-3brw-rw---- 1 grid asmadmin 252, 11 Aug 13 10:38 /dev/dm-11brw-rw---- 1 grid asmadmin 252, 17 Aug 13 10:38 /dev/dm-17brw-rw---- 1 grid asmadmin 252, 12 Aug 13 10:38 /dev/dm-12brw-rw---- 1 grid asmadmin 252, 15 Aug 13 10:38 /dev/dm-15brw-rw---- 1 grid asmadmin 252, 16 Aug 13 10:38 /dev/dm-16brw-rw---- 1 grid asmadmin 252, 13 Aug 13 10:38 /dev/dm-13brw-rw---- 1 grid asmadmin 252, 14 Aug 13 10:38 /dev/dm-14brw-rw---- 1 grid asmadmin 252, 19 Aug 13 10:38 /dev/dm-19brw-rw---- 1 grid asmadmin 252, 21 Aug 13 10:38 /dev/dm-21brw-rw---- 1 grid asmadmin 252, 18 Aug 13 10:38 /dev/dm-18brw-rw---- 1 grid asmadmin 252, 20 Aug 13 10:38 /dev/dm-20[root@oracle631 ~]# Configure the LUNs with ASMLib as shown below. Alternatively, for a large number of LUNs scripted methods can be used as shown in the Violin Memory Oracle Database Best Practice Document April 2014. [root@oracle631 ~]# oracleasm createdisk ASM_RECO1 /dev/mapper/ASM_RECO1Writing disk header: doneInstantiating disk: done[root@oracle631 ~]# oracleasm createdisk ASM_RECO2 /dev/mapper/ASM_RECO2Writing disk header: doneInstantiating disk: done[root@oracle631 ~]# oracleasm createdisk ASM_RECO3 /dev/mapper/ASM_RECO3Writing disk header: doneInstantiating disk: done[root@oracle631 ~]# oracleasm createdisk ASM_RECO4 /dev/mapper/ASM_RECO4Writing disk header: doneInstantiating disk: done[root@oracle631 ~]# oracleasm createdisk ASM_RECO5 /dev/mapper/ASM_RECO5Writing disk header: doneInstantiating disk: done[root@oracle631 ~]# oracleasm createdisk ASM_RECO6 /dev/mapper/ASM_RECO6Writing disk header: doneInstantiating disk: done[root@oracle631 ~]# oracleasm createdisk ASM_RECO7 /dev/mapper/ASM_RECO7Writing disk header: doneInstantiating disk: done[root@oracle631 ~]# oracleasm createdisk ASM_RECO8 /dev/mapper/ASM_RECO8Writing disk header: doneInstantiating disk: done[root@oracle631 ~]# Connect to the server as grid user as shown below and run "asmca" to build the RECO asm diskgroup. Note that asmca can be used here because these are 512 emulated LUNs so no special attributes are needed. If these were 4096 native LUNs command line would have to be used to build the diskgroup. gstanden@vmem1:~$ ssh -Y -C grid@10.207.39.72grid@10.207.39.72's password: Last login: Wed Aug 13 10:45:42 2014 from 10.207.39.1[grid@oracle631 ~]$ asmcaThe screenshots of asmca are shown below. The final configuration of disks for the Oracle database install.SQL> l 1* select name, label, path, total_mb, header_status, mount_status, mode_status, sector_size from v$asm_diskSQL> /NAME LABEL PATH TOTAL_MB HEADER_STATU MOUNT_S MODE_ST SECTOR_SIZE------------------------------ ------------------------------- -------------------- ---------- ------------ ------- ------- -----------ASM_DATA1 ASM_DATA1 ORCL:ASM_DATA1 1024 MEMBER CACHED ONLINE 512ASM_DATA2 ASM_DATA2 ORCL:ASM_DATA2 1024 MEMBER CACHED ONLINE 512ASM_DATA3 ASM_DATA3 ORCL:ASM_DATA3 1024 MEMBER CACHED ONLINE 512ASM_DATA4 ASM_DATA4 ORCL:ASM_DATA4 1024 MEMBER CACHED ONLINE 512ASM_DATA5 ASM_DATA5 ORCL:ASM_DATA5 1024 MEMBER CACHED ONLINE 512ASM_DATA6 ASM_DATA6 ORCL:ASM_DATA6 1024 MEMBER CACHED ONLINE 512ASM_DATA7 ASM_DATA7 ORCL:ASM_DATA7 1024 MEMBER CACHED ONLINE 512ASM_DATA8 ASM_DATA8 ORCL:ASM_DATA8 1024 MEMBER CACHED ONLINE 512SYSTEMDG SYSTEMDG ORCL:SYSTEMDG 1024 MEMBER CACHED ONLINE 512ASM_RECO1 ASM_RECO1 ORCL:ASM_RECO1 1024 MEMBER CACHED ONLINE 512ASM_RECO2 ASM_RECO2 ORCL:ASM_RECO2 1024 MEMBER CACHED ONLINE 512ASM_RECO3 ASM_RECO3 ORCL:ASM_RECO3 1024 MEMBER CACHED ONLINE 512ASM_RECO4 ASM_RECO4 ORCL:ASM_RECO4 1024 MEMBER CACHED ONLINE 512ASM_RECO5 ASM_RECO5 ORCL:ASM_RECO5 1024 MEMBER CACHED ONLINE 512ASM_RECO6 ASM_RECO6 ORCL:ASM_RECO6 1024 MEMBER CACHED ONLINE 512ASM_RECO7 ASM_RECO7 ORCL:ASM_RECO7 1024 MEMBER CACHED ONLINE 512ASM_RECO8 ASM_RECO8 ORCL:ASM_RECO8 1024 MEMBER CACHED ONLINE 51217 rows selected.SQL> Obtain the Oracle binaries and install for your chosen version of database. Begin the install. Here the oracle install media is unpacked under the /home/oracle directory as shown below. [oracle@oracle631 database]$ pwd/home/oracle/database[oracle@oracle631 database]$ ./runInstaller Starting Oracle Universal Installer...Checking Temp space: must be greater than 120 MB. Actual 2527 MB PassedChecking swap space: must be greater than 150 MB. Actual 5087 MB PassedChecking monitor: must be configured to display at least 256 colors. Actual 16777216 PassedPreparing to launch Oracle Universal Installer from /tmp/OraInstall2014-08-13_11-36-50AM. Please wait ...[oracle@oracle631 database]$ |
SCST >