After a reboot of the REDFERN, the permissions of the shared disks are not being set so that ASM can use them.
In order to see what the current properties of /dev/xvdh are, I ran the following command as root:
udevadm info /dev/xvdh
The result is:
P: /devices/vbd-51824/block/xvdh N: xvdh E: DEVNAME=/dev/xvdh E: DEVPATH=/devices/vbd-51824/block/xvdh E: DEVTYPE=disk E: ID_FS_TYPE=oracleasm E: ID_FS_USAGE=filesystem E: MAJOR=202 E: MINOR=112 E: SUBSYSTEM=block E: TAGS=:systemd: E: USEC_INITIALIZED=52512
Following the procedure in 7.5 Modifying Udev Rules, I set my UDEV rule as follows for the voting disk (xvdh):
cat >/etc/udev/rules.d/10-local.rules <<DONE KERNEL=="xvdh", ACTION=="ADD", OWNER:="oracle", GROUP:="dba", MODE:="0660" DONE
Note: There is a mistake in this setup — the action should be in lowercase (add), not in uppercase (ADD).
Following the procedure in 7.5 Modifying Udev Rules, I ran the following command against the device path discovered above:
udevadm test /devices/vbd-51824/block/xvdh
And the output is:
calling: test version 208 This program is for debugging only, it does not run any program specified by a RUN key. It may show incorrect results, because some values may be different, or not available at a simulation run. === trie on-disk === tool version: 208 file size: 6029573 bytes header size 80 bytes strings 1330725 bytes nodes 4698768 bytes load module index read rules file: /usr/lib/udev/rules.d/10-dm.rules read rules file: /etc/udev/rules.d/10-local.rules read rules file: /usr/lib/udev/rules.d/100-balloon.rules read rules file: /usr/lib/udev/rules.d/11-dm-lvm.rules read rules file: /usr/lib/udev/rules.d/13-dm-disk.rules read rules file: /usr/lib/udev/rules.d/42-usb-hid-pm.rules read rules file: /usr/lib/udev/rules.d/50-udev-default.rules read rules file: /usr/lib/udev/rules.d/60-alias-kmsg.rules read rules file: /usr/lib/udev/rules.d/60-cdrom_id.rules read rules file: /usr/lib/udev/rules.d/60-keyboard.rules read rules file: /usr/lib/udev/rules.d/60-net.rules read rules file: /usr/lib/udev/rules.d/60-persistent-alsa.rules read rules file: /usr/lib/udev/rules.d/60-persistent-input.rules read rules file: /usr/lib/udev/rules.d/60-persistent-serial.rules read rules file: /usr/lib/udev/rules.d/60-persistent-storage-tape.rules read rules file: /usr/lib/udev/rules.d/60-persistent-storage.rules read rules file: /usr/lib/udev/rules.d/60-persistent-v4l.rules read rules file: /usr/lib/udev/rules.d/60-raw.rules read rules file: /usr/lib/udev/rules.d/61-accelerometer.rules read rules file: /usr/lib/udev/rules.d/64-btrfs.rules read rules file: /usr/lib/udev/rules.d/69-dm-lvm-metad.rules read rules file: /usr/lib/udev/rules.d/70-power-switch.rules read rules file: /usr/lib/udev/rules.d/70-uaccess.rules read rules file: /usr/lib/udev/rules.d/71-biosdevname.rules read rules file: /usr/lib/udev/rules.d/71-seat.rules read rules file: /usr/lib/udev/rules.d/73-seat-late.rules read rules file: /usr/lib/udev/rules.d/75-net-description.rules read rules file: /usr/lib/udev/rules.d/75-probe_mtd.rules read rules file: /usr/lib/udev/rules.d/75-tty-description.rules read rules file: /usr/lib/udev/rules.d/77-nm-olpc-mesh.rules read rules file: /usr/lib/udev/rules.d/78-sound-card.rules read rules file: /usr/lib/udev/rules.d/80-drivers.rules read rules file: /usr/lib/udev/rules.d/80-net-name-slot.rules read rules file: /usr/lib/udev/rules.d/81-kvm-rhel.rules read rules file: /usr/lib/udev/rules.d/90-alsa-tools-firmware.rules read rules file: /usr/lib/udev/rules.d/91-drm-modeset.rules read rules file: /usr/lib/udev/rules.d/95-dm-notify.rules read rules file: /usr/lib/udev/rules.d/95-udev-late.rules read rules file: /etc/udev/rules.d/98-kexec.rules read rules file: /usr/lib/udev/rules.d/99-systemd.rules rules contain 24576 bytes tokens (2048 * 12 bytes), 11185 bytes strings 1723 strings (21183 bytes), 1146 de-duplicated (10576 bytes), 578 trie nodes used GROUP 6 /usr/lib/udev/rules.d/50-udev-default.rules:51 IMPORT 'cdrom_id --lock-media /dev/xvdh' /usr/lib/udev/rules.d/60-cdrom_id.rules:16 starting 'cdrom_id --lock-media /dev/xvdh' 'cdrom_id --lock-media /dev/xvdh' [96018] exit with return code 1 IMPORT builtin 'path_id' /usr/lib/udev/rules.d/60-persistent-storage.rules:55 IMPORT builtin 'path_id' returned non-zero IMPORT builtin 'blkid' /usr/lib/udev/rules.d/60-persistent-storage.rules:70 probe /dev/xvdh raid offset=0 handling device node '/dev/xvdh', devnum=b202:112, mode=0660, uid=0, gid=6 preserve permissions /dev/xvdh, 060660, uid=0, gid=6 preserve already existing symlink '/dev/block/202:112' to '../xvdh' .ID_FS_TYPE_NEW=oracleasm ACTION=add DEVNAME=/dev/xvdh DEVPATH=/devices/vbd-51824/block/xvdh DEVTYPE=disk ID_FS_TYPE=oracleasm ID_FS_USAGE=filesystem MAJOR=202 MINOR=112 SUBSYSTEM=block TAGS=:systemd: USEC_INITIALIZED=52512 unload module index
The correct device is being selected, and the correct rules are being read. But the permissions are not being overridden.
There is no warning that the action is invalid.
Following the procedure in 7.5 Modifying Udev Rules, I set my UDEV rule as follows for the voting disk (xvdh):
cat >/etc/udev/rules.d/10-local.rules <<DONE KERNEL=="xvdh", ACTION=="add", OWNER:="oracle", GROUP:="dba", MODE:="0660" DONE
Note: The action is now in lowercase (add), not in uppercase (ADD).
And the UDEV simulation shows the correct output:
… probe /dev/xvdh raid offset=0 handling device node '/dev/xvdh', devnum=b202:112, mode=0660, uid=54321, gid=54322 preserve permissions /dev/xvdh, 060660, uid=54321, gid=54322 preserve already existing symlink '/dev/block/202:112' to '../xvdh' …
uid=54321 is for user oracle on my system.
gid=54322 is for group dba on my system.