Sunday 18 May, 2008 - 22:30
It would now appear that richmond1 has problems getting to runlevel 5. It tries several times before restarting automatically. I will have to rebuild the richmond1 image from the RHEL 3.7 1GB Base image.
On richmond2, I am having difficulties in getting the ora.example.example2.inst resource to come online. The $CRS_HOME/log/richmond2/crsd/crsd.log shows:
[ CRSOCR][388811696]0CAAOCR SET Debug Level[ora.example.example2.inst]: 2
[ CRSRES][525179824]0startRunnable: setting CLI values
[ CRSRES][525179824]0Attempting to start `ora.example.example2.inst` on member `richmond2`
[ CRSAPP][525179824]0StartResource error for ora.example.example2.inst error code = 1
[ CRSRES][525179824]0Start of `ora.example.example2.inst` on member `richmond2` failed.
[ CRSRES][525179824]0CRS-1006: No more members to consider
This was generated by the following commands:
# crsctl debug log res ora.example.example2.inst:2
# crs_start ora.example.example2.inst
There are no hits on Metalink for this problem (0StartResource).
I should rebuild the whole cluster from scratch.
This seemed too much like hard work, so I continued my investigations.
I tried starting example2 via srvctl to no avail, and via sqlplus. SQL*Plus did say that it could not find the pfile. This was unusual because it should be using the spfile in ASM.
asmcmd reported that both disk groups were not mounted. I tried mounting the diskgroups via SQL*Plus (ALTER DISKGROUP ALL MOUNT;) but I got an error. Checked the alert log in /u00/app/oracle/admin/+ASM/bdump/alert_+asm2.log :
ORA-15196: invalid ASM block header [kfr.c:6030] [endian_kfbh] [3] [10773] [33 != 1]
According to MetaLink DocID 194459.1, I should contact Oracle Support. Since I am running an unsupported configuration, I miss out.
So a rebuild is needed. I suppose I need the practice.