KVM Add PV Guest VG

Create and Add New File-Backed LUN to KVM Guest

This page details how to add a PV to a VG inside a KVM guest when the KVM guest is using virtio-blk-data-plane.

The virt-install GUI can add a new storage LUN online without shutting down the machine as shown below. Note that LUNs can also be added from the command line. (Not shown). If prefer to go that route, use the reference here at Raymii website or similar guides to use qemu-img from the command line instead of GUI tool.

Proceeding with GUI method as shown below.

Hit finish and the new LUN is added. You can also opt to use thin-provisioned file LUNs. In that case uncheck "Allocated entire disk now" as shown above in the screenshot.

However, the new LUN cannot be added to a VG yet if the VG is using PV which are (512 logical / 4096 physical) until the physical sector size has been set in the XML file. The KVM guest will need to be shutdown, and the changes made to the XML to enable virtio-blk-data-plane. Edit the XML of the KVM guest as shown below. The parts in bold are the sections required to

(1) enable virtio-blk-data-plane, and;

(2) set the logical and physical blocksize of the virtual LUN.

gstanden@vmem1:~$ virsh dumpxml oracle652

<domain type='kvm' id='3' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <-- Be sure xmlns:qemu is set for virtio-blk-data-plane

<name>oracle652</name>

<uuid>cf142c15-9452-aa69-0d52-aa55a0bcacbf</uuid>

<memory unit='KiB'>1048576</memory>

<currentMemory unit='KiB'>1048576</currentMemory>

<vcpu placement='static'>8</vcpu>

<resource>

<partition>/machine</partition>

</resource>

<os>

<type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>

<boot dev='hd'/>

<boot dev='cdrom'/>

<bootmenu enable='yes'/>

</os>

<features>

<acpi/>

<apic/>

<pae/>

</features>

<clock offset='utc'/>

<on_poweroff>destroy</on_poweroff>

<on_reboot>restart</on_reboot>

<on_crash>restart</on_crash>

<devices>

<emulator>/usr/bin/kvm-spice</emulator>

<disk type='file' device='disk'>

<driver name='qemu' type='raw' cache='none' io='native'/>

<source file='/var/lib/libvirt/images/oracle652-0.img'/>

<target dev='vda' bus='virtio'/>

<alias name='virtio-disk0'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>

</disk>

<disk type='file' device='disk'>

<driver name='qemu' type='raw' cache='none' io='native'/>

<source file='/var/lib/libvirt/images/oracle652.img'/>

<target dev='vdb' bus='virtio'/>

<alias name='virtio-disk1'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>

</disk>

<disk type='file' device='disk'>

<driver name='qemu' type='raw' cache='none' io='native'/>

<source file='/var/lib/libvirt/images/oracle652-4.img'/>

<target dev='vdc' bus='virtio'/>

<alias name='virtio-disk2'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>

</disk>

<disk type='file' device='disk'>

<driver name='qemu' type='raw' cache='none' io='native'/>

<source file='/var/lib/libvirt/images/oracle652-3.img'/>

<target dev='vdd' bus='virtio'/>

<alias name='virtio-disk3'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>

</disk>

<disk type='file' device='disk'>

<driver name='qemu' type='raw' cache='none' io='native'/>

<source file='/var/lib/libvirt/images/oracle652-1.img'/>

<target dev='vde' bus='virtio'/>

<alias name='virtio-disk4'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>

</disk>

<disk type='file' device='disk'>

<driver name='qemu' type='raw' cache='none' io='native'/>

<source file='/var/lib/libvirt/images/oracle652-2.img'/>

<target dev='vdf' bus='virtio'/>

<alias name='virtio-disk5'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/>

</disk>

<disk type='file' device='disk'>

<driver name='qemu' type='raw' cache='none' io='native'/> <-- Add cache and io settings for virtio-blk-data-plane

<source file='/var/lib/libvirt/images/oracle652-5.img'/>

<target dev='vdg' bus='virtio'/>

<alias name='virtio-disk6'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>

</disk>

<disk type='block' device='cdrom'>

<driver name='qemu' type='raw'/>

<target dev='hdc' bus='ide' tray='open'/>

<readonly/>

<alias name='ide0-1-0'/>

<address type='drive' controller='0' bus='1' target='0' unit='0'/>

</disk>

<controller type='usb' index='0'>

<alias name='usb0'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>

</controller>

<controller type='pci' index='0' model='pci-root'>

<alias name='pci.0'/>

</controller>

<controller type='ide' index='0'>

<alias name='ide0'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>

</controller>

<controller type='scsi' index='0'>

<alias name='scsi0'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>

</controller>

<interface type='bridge'>

<mac address='52:54:00:47:19:56'/>

<source bridge='sw1'/>

<virtualport type='openvswitch'>

<parameters interfaceid='3f227bc1-553a-43ad-a774-797e1bac6deb'/>

</virtualport>

<target dev='s6'/>

<model type='virtio'/>

<alias name='net0'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

</interface>

<interface type='bridge'>

<mac address='52:54:00:d0:30:ea'/>

<source bridge='sw2'/>

<virtualport type='openvswitch'>

<parameters interfaceid='376b5703-a814-4f6b-8388-2f7b10861b4d'/>

</virtualport>

<target dev='t6'/>

<model type='virtio'/>

<alias name='net1'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>

</interface>

<interface type='bridge'>

<mac address='52:54:00:a0:00:41'/>

<source bridge='sw3'/>

<virtualport type='openvswitch'>

<parameters interfaceid='6ad9dcd1-b318-45e1-a124-052acda1bcf5'/>

</virtualport>

<target dev='w6'/>

<model type='virtio'/>

<alias name='net2'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>

</interface>

<serial type='pty'>

<source path='/dev/pts/15'/>

<target port='0'/>

<alias name='serial0'/>

</serial>

<console type='pty' tty='/dev/pts/15'>

<source path='/dev/pts/15'/>

<target type='serial' port='0'/>

<alias name='serial0'/>

</console>

<input type='mouse' bus='ps2'/>

<input type='keyboard' bus='ps2'/>

<graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>

<listen type='address' address='127.0.0.1'/>

</graphics>

<sound model='ich6'>

<alias name='sound0'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

</sound>

<video>

<model type='cirrus' vram='9216' heads='1'/>

<alias name='video0'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

</video>

<memballoon model='virtio'>

<alias name='balloon0'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

</memballoon>

</devices>

<seclabel type='dynamic' model='apparmor' relabel='yes'>

<label>libvirt-cf142c15-9452-aa69-0d52-aa55a0bcacbf</label>

<imagelabel>libvirt-cf142c15-9452-aa69-0d52-aa55a0bcacbf</imagelabel>

</seclabel>

<qemu:commandline>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk0.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk0.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk0.x-data-plane=on'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk1.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk1.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk1.x-data-plane=on'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk2.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk2.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk2.x-data-plane=on'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk3.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk3.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk3.x-data-plane=on'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk4.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk4.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk4.x-data-plane=on'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk4.logical_block_size=512'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk4.physical_block_size=4096'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk5.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk5.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk5.x-data-plane=on'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk5.logical_block_size=512'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk5.physical_block_size=4096'/>

<qemu:arg value='-set'/> <-- This stanza sets virtio-blk-data-plane and sets logical and physical block size

<qemu:arg value='device.virtio-disk6.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk6.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk6.x-data-plane=on'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk6.logical_block_size=512'/> <-- Here logical block size of LUN is set to 512

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk6.physical_block_size=4096'/> <-- Here physical block size of LUN is set to 4096 (4K)

</qemu:commandline>

</domain>

gstanden@vmem1:~$

Optional: Verify Identity of LUN in KVM Guest

This typically would not be necessary in a simple system just using some file-backed KVM LUNs but if you had a complex system with file-backed LUNs and block-device LUNs etc. it might be desirable to verify the identity of the new LUN carefully before working with it, so some steps are presented here to do that. To be extra careful in systems with many many disks, one can match the physical disks at the KVM host level with the physical disks at the KVM guest level for absolute identification, the following command can be used to obtain PCI information about the KVM virtual file-backed LUNs as shown below. In this example, "drive-virtio-disk1" and "drive-virtio-disk2" had been previously dropped from the KVM guest. Therefore, when this new disk was added, the disk indexes that had been dropped were used first. In this example, KVM chose to use the lowest-numbered previously-used disk index, namely "drive-virtio-disk1" which is the default behavior of KVM. Note the "addr" value for "drive-virtio-disk1" in the "class SCSI controller" field.

Note that I added some material to this and these verification steps don't match up exactly as far as LUN names with regard to other sections, but the concepts and commands are valid and the same.

gstanden@vmem1:~$ sudo virsh qemu-monitor-command --hmp oracle651 'info qtree' | egrep 'class SCSI controller|drive-virtio-disk|logical_block|physical_block'

drive = "drive-virtio-disk4"

logical_block_size = 512 (0x200)

physical_block_size = 4096 (0x1000)

class SCSI controller, addr 00:0c.0, pci id 1af4:1001 (sub 1af4:0002)

drive = "drive-virtio-disk4"

logical_block_size = 512 (0x200)

physical_block_size = 4096 (0x1000)

drive = "drive-virtio-disk3"

logical_block_size = 512 (0x200)

physical_block_size = 4096 (0x1000)

class SCSI controller, addr 00:0b.0, pci id 1af4:1001 (sub 1af4:0002)

drive = "drive-virtio-disk3"

logical_block_size = 512 (0x200)

physical_block_size = 4096 (0x1000)

drive = "drive-virtio-disk1"

logical_block_size = 512 (0x200)

physical_block_size = 4096 (0x1000)

class SCSI controller, addr 00:07.0, pci id 1af4:1001 (sub 1af4:0002)

drive = "drive-virtio-disk1"

logical_block_size = 512 (0x200)

physical_block_size = 4096 (0x1000)

drive = "drive-virtio-disk0"

logical_block_size = 512 (0x200)

physical_block_size = 512 (0x200)

class SCSI controller, addr 00:05.0, pci id 1af4:1001 (sub 1af4:0002)

drive = "drive-virtio-disk0"

logical_block_size = 512 (0x200)

physical_block_size = 512 (0x200)

logical_block_size = 512 (0x200)

physical_block

This is the attribute that maps the actual device to the device node used by the KVM guest. Now the physical disk can be matched to the corresponding device node in "/dev" on the KVM guest as shown below.

[root@oracle651 ~]# ls -lrt /dev/disk/by-path

total 0

lrwxrwxrwx 1 root root 9 Aug 26 11:54 pci-0000:00:0c.0-virtio-pci-virtio7 -> ../../vdd

lrwxrwxrwx 1 root root 9 Aug 26 11:54 pci-0000:00:0b.0-virtio-pci-virtio6 -> ../../vdc

lrwxrwxrwx 1 root root 9 Aug 26 11:54 pci-0000:00:07.0-virtio-pci-virtio3 -> ../../vdb

lrwxrwxrwx 1 root root 9 Aug 26 11:54 pci-0000:00:05.0-virtio-pci-virtio1 -> ../../vda

lrwxrwxrwx 1 root root 10 Aug 26 11:54 pci-0000:00:05.0-virtio-pci-virtio1-part2 -> ../../vda2

lrwxrwxrwx 1 root root 10 Aug 26 11:54 pci-0000:00:05.0-virtio-pci-virtio1-part1 -> ../../vda1

lrwxrwxrwx 1 root root 9 Aug 26 11:54 pci-0000:00:01.1-scsi-0:0:0:0 -> ../../sr0

[root@oracle651 ~]#

Note that "/dev/vdb" has been used for the new disk. It can be verified a couple of ways. The first way is the match to the "pci-0000:00:07.0" which matches the output from the above qemu-monitor command.

The second way to match is to check the size of the LUN, and the logical and physical block sizes to be sure they match what was set in the XML of the KVM guest as shown below.

[root@oracle651 ~]# fdisk -l /dev/vdb

Disk /dev/vdb: 21.5 GB, 21474836480 bytes

16 heads, 63 sectors/track, 41610 cylinders

Units = cylinders of 1008 * 512 = 516096 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk identifier: 0x00000000

[root@oracle651 ~]#

It is also useful to list out all the currently assigned PV (physical volumes) to make sure that the new device node does not appear as an already-assigned PV as shown below. Note that "/dev/vdb" is not listed as a current PV as expected.

[root@oracle651 ~]# pvdisplay

--- Physical volume ---

PV Name /dev/vdc

VG Name vg_scst

PV Size 20.00 GiB / not usable 4.00 MiB

Allocatable yes (but full)

PE Size 4.00 MiB

Total PE 5119

Free PE 0

Allocated PE 5119

PV UUID MVPFlX-DlUw-QZtk-KHPi-dldw-0IIB-WHBmwG

--- Physical volume ---

PV Name /dev/vdd

VG Name vg_scst

PV Size 20.00 GiB / not usable 4.00 MiB

Allocatable yes (but full)

PE Size 4.00 MiB

Total PE 5119

Free PE 0

Allocated PE 5119

PV UUID sUVcv5-MIJG-sa7h-JCyS-N5Gh-m3y7-X3iAtg

--- Physical volume ---

PV Name /dev/vda2

VG Name vg_oracle651

PV Size 19.51 GiB / not usable 3.00 MiB

Allocatable yes (but full)

PE Size 4.00 MiB

Total PE 4994

Free PE 0

Allocated PE 4994

PV UUID EdMJuX-nhOz-5sfp-PvOA-d3w4-uVoB-yl8kL1

[root@oracle651 ~]#

Finally, do a "df -TH" to ensure that there is not a filesystem mounted using "/dev/vdb" as shown below.

[root@oracle651 ~]# df -TH

Filesystem Type Size Used Avail Use% Mounted on

/dev/mapper/vg_oracle651-lv_root

ext4 19G 17G 1.1G 95% /

tmpfs tmpfs 520M 74k 520M 1% /dev/shm

/dev/vda1 ext4 500M 117M 357M 25% /boot

/dev/mapper/vg_scst-lv_oracle631

ext4 21G 21G 0 100% /scst_oracle631

/dev/mapper/vg_scst-lv_oracle632

ext4 21G 447M 20G 3% /scst_oracle632

[root@oracle651 ~]#

After all these verification steps, it is safe to create "/dev/vdb" as a new PV and add it to the desired VG to be extended as shown below. First verify that "/dev/vdb" still maps to the actual disk that has been added using the pci information in /dev/disk/by-path correlated with the qemu-monitor-command, as shown below.

gstanden@vmem1:~$ ssh root@oracle651

root@oracle651's password:

Last login: Tue Aug 26 11:55:00 2014 from 10.207.39.1

[root@oracle651 ~]# clear

[root@oracle651 ~]# ls -lrt /dev/disk

total 0

drwxr-xr-x 2 root root 180 Aug 26 15:27 by-path

drwxr-xr-x 2 root root 140 Aug 26 15:27 by-uuid

drwxr-xr-x 2 root root 200 Aug 26 15:27 by-id

[root@oracle651 ~]# ls -lrt /dev/disk/by-path

total 0

lrwxrwxrwx 1 root root 9 Aug 26 15:27 pci-0000:00:0c.0-virtio-pci-virtio7 -> ../../vdd

lrwxrwxrwx 1 root root 9 Aug 26 15:27 pci-0000:00:0b.0-virtio-pci-virtio6 -> ../../vdc

lrwxrwxrwx 1 root root 9 Aug 26 15:27 pci-0000:00:07.0-virtio-pci-virtio3 -> ../../vdb

lrwxrwxrwx 1 root root 9 Aug 26 15:27 pci-0000:00:05.0-virtio-pci-virtio1 -> ../../vda

lrwxrwxrwx 1 root root 10 Aug 26 15:27 pci-0000:00:05.0-virtio-pci-virtio1-part1 -> ../../vda1

lrwxrwxrwx 1 root root 10 Aug 26 15:27 pci-0000:00:05.0-virtio-pci-virtio1-part2 -> ../../vda2

lrwxrwxrwx 1 root root 9 Aug 26 15:27 pci-0000:00:01.1-scsi-0:0:0:0 -> ../../sr0

Next check the current pv that are defined using pvs.

[root@oracle651 ~]# df -TH

Filesystem Type Size Used Avail Use% Mounted on

/dev/mapper/vg_oracle651-lv_root

ext4 40G 26G 12G 69% /

tmpfs tmpfs 520M 74k 520M 1% /dev/shm

/dev/vda1 ext4 500M 179M 295M 38% /boot

/dev/mapper/vg_scst-lv_oracle631

ext4 21G 18G 2.7G 87% /scst_oracle631

/dev/mapper/vg_scst-lv_oracle632

ext4 21G 18G 2.7G 87% /scst_oracle632

/dev/mapper/vg_scst-lv_vmem1

ext4 127G 19G 104G 15% /scst_vmem1

[root@oracle651 ~]#

Done.!