Disabling Direct NFS Client Oracle Disk Management Control of NFS
Further thought on the problem described in 11 Create Database Failed led me to suspect that the problem was the DirectNFS driver being used by ASM.
Analysis
Since Oracle Clusterware appears to be using the ZFS Datasets on the FreeNAS just fine, I assume there must be some difference between how ASM was accessing the NFS files to the way by Oracle Clusterware. That difference is DirectNFS.
Although the DirectNFS ODM driver is installed in the Oracle Database home, there is no oranfstab file there to activate it.
First Attempt to Disable DirectNFS for ASM
I followed the first option in the procedure in Disabling Direct NFS Client Oracle Disk Management Control of NFS by removing the oranfstab file in ASM:
[asm@penrith1 ~]$ srvctl stop asm -n penrith1 [asm@penrith1 ~]$ cd $ORACLE_HOME/dbs [asm@penrith1 dbs]$ ls ab_+ASM1.dat init+ASM1.ora init.ora orapw+ASM1 hc_+ASM1.dat initdw.ora oranfstab [asm@penrith1 dbs]$ rm oranfstab [asm@penrith1 dbs]$ srvctl start asm -n penrith1 [asm@penrith1 dbs]$ asmcmd lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Name MOUNTED EXTERN N 512 4096 1048576 3104 3010 0 3010 0 DATA/ MOUNTED EXTERN N 512 4096 1048576 4062 3969 0 3969 0 RECOVERY/
Still get the same problem according to the alert log in /u01/app/oracle/diag/rdbms/example/example1/trace/alert_example1.log:
NOTE:Loaded library: System SUCCESS: diskgroup DATA was mounted Errors in file /u01/app/oracle/diag/rdbms/example/example1/trace/example1_ora_17659.trc: ORA-27041: unable to open file WARNING: IO Failed. subsys:System dg:1, disk:0x0.0xe9685a7f au:94 iop:0x2512000 bufp:0x5b2600 iosz:8192 oper:2 wait:0 result: 4 osderr:0x0 osderr1:0x0 pid:17659 Errors in file /u01/app/oracle/diag/rdbms/example/example1/trace/example1_ora_17659.trc: ORA-15080: synchronous I/O operation to a disk failed WARNING: failed to write mirror side 0 of virtual extent 0 of file 256 in group 1 2012-01-08 02:21:59.793000 +11:00 SUCCESS: diskgroup DATA was dismounted
Second Attempt to Remove DirectNFS
The first option should have disabled DirectNFS for the NFS files. However, I really wanted to make sure that DirectNFS was not being used by ASM.
In the second attempt to remove DirectNFS, I followed the section option in the procedure in Disabling Direct NFS Client Oracle Disk Management Control of NFS by restoring the libodm11.so stub file in ASM:
[asm@penrith1 ~]$ srvctl stop asm -n penrith1 [asm@penrith1 ~]$ cd $ORACLE_HOME/lib [asm@penrith1 lib]$ ls -l *odm* -rw-r--r-- 1 asm oinstall 44670 Aug 1 2007 libnfsodm11.so lrwxrwxrwx 1 asm oinstall 12 Jan 2 08:05 libodm11.so -> libodmd11.so -rw-r--r-- 1 asm oinstall 8841 Jan 4 21:11 libodm11.so_stub -rw-r--r-- 1 asm oinstall 8841 Aug 1 2007 libodmd11.so [asm@penrith1 lib]$ rm libodm11.so [asm@penrith1 lib]$ mv libodm11.so_stub libodm11.so [asm@penrith1 lib]$ srvctl start asm -n penrith1
And again, I get the same problem according to the alert log in /u01/app/oracle/diag/rdbms/example/example1/trace/alert_example1.log. See above for the style of error message.
The obvious error is the link error for libodm11.so — it should not be pointing to itself; it should be pointing to libnfsodm11.so.
DirectNFS in ASM was not the cause of the problem of write failures during database creation.
I have to consider other options.