Creating Required Directories for Oracle Database Files on Shared File Systems.
The EXAMPLE database successfully created on the PENRITH cluster after I corrected the file and directory permissions for the ASM files to allow the oracle user read/write permission on those files.
Corrections have been made based on information in 14 Set Up DirectNFS for Example Database.
The analysis put forward in 12 Remove DirectNFS in ASM was flawed because I should have realised that ASM itself was able to successfully write to the /u07 files because ASM was able to initialise and maintain its metadata there.
By reviewing Creating Required Directories for Oracle Database Files on Shared File Systems, I suspected that the security settings on the /u07 files were not correct. I was able to confirm this as follows:
[oracle@penrith1 ~]$ ls -ld /u07 drwxr-xr-x 2 asm asmdba 4 Jan 4 23:04 /u07 [oracle@penrith1 ~]$ sudo su - [root@penrith1 ~]# ls -l /u07 total 7340190 -rw-r--r-- 1 asm asmdba 3255435264 Jan 8 11:50 data.dbf -rw-r--r-- 1 asm asmdba 4259840000 Jan 8 11:50 recovery.dbf
The oracle user does not have write access to these files and this is a much simpler explanation of the ORA-15080: synchronous I/O operation to a disk failed message.
Change Permissions
Following Creating Required Directories for Oracle Database Files on Shared File Systems, I chose to use the oinstall group to allow the oracle user read/write access to the ASM files. I issued the following commands:
[root@penrith1 ~]# chmod 775 /u07 [root@penrith1 ~]# chown -R asm:oinstall /u07 [root@penrith1 ~]# chmod 664 /u07/*.dbf [root@penrith1 ~]# ls -ld /u07 drwxrwxr-x 2 asm oinstall 4 Jan 4 23:04 /u07 [root@penrith1 ~]# ls -l /u07 total 7340190 -rw-rw-r-- 1 asm oinstall 3255435264 Jan 8 11:51 data.dbf -rw-rw-r-- 1 asm oinstall 4259840000 Jan 8 11:51 recovery.dbf
Check ASM Access to Files
I restarted ASM on PENRITH1 in order to confirm that there were no ill effects:
[asm@penrith1 ~]$ srvctl stop asm -n penrith1 [asm@penrith1 ~]$ srvctl start asm -n penrith1 [asm@penrith1 ~]$ asmcmd lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Name MOUNTED EXTERN N 512 4096 1048576 3104 3010 0 3010 0 DATA/ MOUNTED EXTERN N 512 4096 1048576 4062 3969 0 3969 0 RECOVERY/
Create Database
The dbca command was able to complete successfully.
Enable DirectNFS for ASM
I enabled DirectNFS for ASM as follows, by following the procedure in 10 Configure ASM Storage,:
[asm@penrith1 dbs]$ cd /u01/app/asm/product/11.1.0/db_1/dbs [asm@penrith1 dbs]$ cat >oranfstab <<DONE > server: freenas > path: 192.168.2.1 > local: 192.168.2.2 > export: /mnt/PENRITH/ASMDATA > mount: /u07 > DONE [asm@penrith1 dbs]$ cat oranfstab server: freenas path: 192.168.2.1 local: 192.168.2.2 export: /mnt/PENRITH/ASMDATA mount: /u07 [asm@penrith1 dbs]$ srvctl stop database -d example [asm@penrith1 dbs]$ srvctl stop asm -n penrith1 [asm@penrith1 dbs]$ cd ../lib [asm@penrith1 lib]$ cp libodm11.so libodm11.so_stub [asm@penrith1 lib]$ ln -s libnfsodm11.so libodm11.so ln: creating symbolic link `libodm11.so' to `libnfsodm11.so': File exists [asm@penrith1 lib]$ ls -l libodm11.so lrwxrwxrwx 1 oracle oinstall 14 Jan 8 21:44 libodm11.so -> libnfsodm11.so [asm@penrith1 lib]$ srvctl start asm -n penrith1 [asm@penrith1 lib]$ srvctl start instance -d example -i example1
And the EXAMPLE database was successfully opened.
DirectNFS Information
The DirectNFS information is available on the ASM instance as follows:
[asm@penrith1 ~]$ sqlplus / as sysdba SQL*Plus: Release 11.1.0.6.0 - Production on Sun Jan 8 15:25:10 2012 Copyright (c) 1982, 2007, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - Production With the Partitioning, Real Application Clusters, OLAP, Data Mining and Real Application Testing options SQL> select * from v$dnfs_servers; ID SVRNAME DIRNAME MNTPORT NFSPORT WTMAX RTMAX ---------- ------------ -------------------- ---------- ---------- ---------- ---------- 1 freenas /mnt/PENRITH/ASMDATA 623 2049 32768 32768
Faulty analysis was done on my part (very passive voice here!). The file permissions was the core issue.
DirectNFS is now being used after the corrections from 14 Set Up DirectNFS for Example Database were applied.