测试Exadata单个cell失败，模拟X2-2 3个cellserver中单个cell丢失的情况：
cell server
CellCLI> alter cell shutdown services all Stopping the RS, CELLSRV, and MS services... The SHUTDOWN of services was successful. ASM LOG: Sun Sep 02 09:08:48 2012 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_gmon_30756.trc: ORA-27603: Cell storage I/O error, I/O failed on disk o/192.168.64.131/DATA_DM01_CD_00_dm01cel01 at offset 8384512 for data length 4096 ORA-27626: Exadata error: 12 (Network error) ORA-27300: OS system dependent operation:rpc update timed out failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxp_path WARNING: Write Failed. group:1 disk:0 AU:1 offset:4190208 size:4096 WARNING: Hbeat write to PST disk 0.3940753198 (DATA_DM01_CD_00_DM01CEL01) in group 1 failed. WARNING: Write Failed. group:2 disk:0 AU:1 offset:4190208 size:4096 WARNING: Hbeat write to PST disk 0.3940753233 (DBFS_DG_CD_02_DM01CEL01) in group 2 failed. WARNING: Write Failed. group:3 disk:0 AU:1 offset:4190208 size:4096 WARNING: Hbeat write to PST disk 0.3940753265 (RECO_DM01_CD_00_DM01CEL01) in group 3 failed. Sun Sep 02 09:08:48 2012 NOTE: process _b000_+asm1 (21302) initiating offline of disk 0.3940753198 (DATA_DM01_CD_00_DM01CEL01) with mask 0x7e in group 1 NOTE: checking PST: grp = 1 Sun Sep 02 09:08:48 2012 NOTE: process _b001_+asm1 (21304) initiating offline of disk 0.3940753233 (DBFS_DG_CD_02_DM01CEL01) with mask 0x7e in group 2 NOTE: checking PST: grp = 2 GMON checking disk modes for group 1 at 10 for pid 30, osid 21302 Sun Sep 02 09:08:48 2012 NOTE: process _b002_+asm1 (21306) initiating offline of disk 0.3940753265 (RECO_DM01_CD_00_DM01CEL01) with mask 0x7e in group 3 NOTE: checking PST: grp = 3 WARNING: Read Failed. group:1 disk:11 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:1 disk:11 AU:1 offset:0 size:4096 WARNING: Read Failed. group:1 disk:10 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:1 disk:10 AU:1 offset:0 size:4096 WARNING: Read Failed. group:1 disk:9 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:1 disk:9 AU:1 offset:0 size:4096 WARNING: Read Failed. group:1 disk:8 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:1 disk:8 AU:1 offset:0 size:4096 WARNING: Read Failed. group:1 disk:7 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:1 disk:7 AU:1 offset:0 size:4096 WARNING: Read Failed. group:1 disk:6 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:1 disk:6 AU:1 offset:0 size:4096 WARNING: Read Failed. group:1 disk:5 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:1 disk:5 AU:1 offset:0 size:4096 WARNING: Read Failed. group:1 disk:4 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:1 disk:4 AU:1 offset:0 size:4096 WARNING: Read Failed. group:1 disk:3 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:1 disk:3 AU:1 offset:0 size:4096 WARNING: Read Failed. group:1 disk:2 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:1 disk:2 AU:1 offset:0 size:4096 WARNING: Read Failed. group:1 disk:1 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:1 disk:1 AU:1 offset:0 size:4096 WARNING: Write Failed. group:1 disk:1 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:1 disk:2 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:1 disk:3 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:1 disk:4 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:1 disk:5 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:1 disk:6 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:1 disk:7 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:1 disk:8 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:1 disk:9 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:1 disk:10 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:1 disk:11 AU:1 offset:4096 size:4096 WARNING: GMON has insufficient disks to maintain consensus. minimum required is 3 NOTE: group DATA_DM01: updated PST location: disk 0012 (PST copy 0) NOTE: group DATA_DM01: updated PST location: disk 0024 (PST copy 1) GMON checking disk modes for group 2 at 11 for pid 35, osid 21304 NOTE: checking PST for grp 1 done. WARNING: Read Failed. group:2 disk:9 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:2 disk:9 AU:1 offset:0 size:4096 WARNING: Read Failed. group:2 disk:8 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:2 disk:8 AU:1 offset:0 size:4096 WARNING: Read Failed. group:2 disk:7 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:2 disk:7 AU:1 offset:0 size:4096 WARNING: Read Failed. group:2 disk:6 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:2 disk:6 AU:1 offset:0 size:4096 WARNING: Read Failed. group:2 disk:5 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:2 disk:5 AU:1 offset:0 size:4096 WARNING: Read Failed. group:2 disk:4 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:2 disk:4 AU:1 offset:0 size:4096 WARNING: Read Failed. group:2 disk:3 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:2 disk:3 AU:1 offset:0 size:4096 WARNING: Read Failed. group:2 disk:2 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:2 disk:2 AU:1 offset:0 size:4096 WARNING: Read Failed. group:2 disk:1 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:2 disk:1 AU:1 offset:0 size:4096 WARNING: Write Failed. group:2 disk:1 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:2 disk:2 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:2 disk:3 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:2 disk:4 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:2 disk:5 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:2 disk:6 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:2 disk:7 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:2 disk:8 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:2 disk:9 AU:1 offset:4096 size:4096 WARNING: GMON has insufficient disks to maintain consensus. minimum required is 3 NOTE: group DBFS_DG: updated PST location: disk 0010 (PST copy 0) NOTE: group DBFS_DG: updated PST location: disk 0020 (PST copy 1) WARNING: Disk 0 (DATA_DM01_CD_00_DM01CEL01) in group 1 mode 0x7f is now being offlined NOTE: checking PST for grp 2 done. GMON checking disk modes for group 3 at 12 for pid 36, osid 21306 WARNING: Disk 0 (DATA_DM01_CD_00_DM01CEL01) in group 1 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Read Failed. group:3 disk:11 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:3 disk:11 AU:1 offset:0 size:4096 WARNING: Read Failed. group:3 disk:10 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:3 disk:10 AU:1 offset:0 size:4096 WARNING: Read Failed. group:3 disk:9 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:3 disk:9 AU:1 offset:0 size:4096 WARNING: Read Failed. group:3 disk:8 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:3 disk:8 AU:1 offset:0 size:4096 WARNING: Read Failed. group:3 disk:7 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:3 disk:7 AU:1 offset:0 size:4096 WARNING: Read Failed. group:3 disk:6 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:3 disk:6 AU:1 offset:0 size:4096 WARNING: Read Failed. group:3 disk:5 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:3 disk:5 AU:1 offset:0 size:4096 WARNING: Read Failed. group:3 disk:4 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:3 disk:4 AU:1 offset:0 size:4096 WARNING: Read Failed. group:3 disk:3 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:3 disk:3 AU:1 offset:0 size:4096 WARNING: Read Failed. group:3 disk:2 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:3 disk:2 AU:1 offset:0 size:4096 WARNING: Read Failed. group:3 disk:1 AU:1 offset:4096 size:4096 WARNING: Read Failed. group:3 disk:1 AU:1 offset:0 size:4096 WARNING: Write Failed. group:3 disk:1 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:3 disk:2 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:3 disk:3 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:3 disk:4 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:3 disk:5 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:3 disk:6 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:3 disk:7 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:3 disk:8 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:3 disk:9 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:3 disk:10 AU:1 offset:4096 size:4096 WARNING: Write Failed. group:3 disk:11 AU:1 offset:4096 size:4096 NOTE: group RECO_DM01: updated PST location: disk 0012 (PST copy 0) NOTE: group RECO_DM01: updated PST location: disk 0024 (PST copy 1) NOTE: checking PST for grp 3 done. WARNING: Disk 1 (DATA_DM01_CD_01_DM01CEL01) in group 1 mode 0x7f is now being offlined WARNING: Disk 1 (DATA_DM01_CD_01_DM01CEL01) in group 1 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 0 (DBFS_DG_CD_02_DM01CEL01) in group 2 mode 0x7f is now being offlined WARNING: Disk 0 (DBFS_DG_CD_02_DM01CEL01) in group 2 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 0 (RECO_DM01_CD_00_DM01CEL01) in group 3 mode 0x7f is now being offlined WARNING: Disk 0 (RECO_DM01_CD_00_DM01CEL01) in group 3 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 2 (DATA_DM01_CD_02_DM01CEL01) in group 1 mode 0x7f is now being offlined WARNING: Disk 2 (DATA_DM01_CD_02_DM01CEL01) in group 1 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 1 (DBFS_DG_CD_03_DM01CEL01) in group 2 mode 0x7f is now being offlined WARNING: Disk 1 (DBFS_DG_CD_03_DM01CEL01) in group 2 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 1 (RECO_DM01_CD_01_DM01CEL01) in group 3 mode 0x7f is now being offlined WARNING: Disk 1 (RECO_DM01_CD_01_DM01CEL01) in group 3 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 3 (DATA_DM01_CD_03_DM01CEL01) in group 1 mode 0x7f is now being offlined WARNING: Disk 3 (DATA_DM01_CD_03_DM01CEL01) in group 1 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 2 (DBFS_DG_CD_04_DM01CEL01) in group 2 mode 0x7f is now being offlined WARNING: Disk 2 (DBFS_DG_CD_04_DM01CEL01) in group 2 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 2 (RECO_DM01_CD_02_DM01CEL01) in group 3 mode 0x7f is now being offlined WARNING: Disk 2 (RECO_DM01_CD_02_DM01CEL01) in group 3 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 4 (DATA_DM01_CD_04_DM01CEL01) in group 1 mode 0x7f is now being offlined WARNING: Disk 4 (DATA_DM01_CD_04_DM01CEL01) in group 1 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 3 (DBFS_DG_CD_05_DM01CEL01) in group 2 mode 0x7f is now being offlined WARNING: Disk 3 (DBFS_DG_CD_05_DM01CEL01) in group 2 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 3 (RECO_DM01_CD_03_DM01CEL01) in group 3 mode 0x7f is now being offlined WARNING: Disk 3 (RECO_DM01_CD_03_DM01CEL01) in group 3 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 5 (DATA_DM01_CD_05_DM01CEL01) in group 1 mode 0x7f is now being offlined WARNING: Disk 5 (DATA_DM01_CD_05_DM01CEL01) in group 1 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 4 (DBFS_DG_CD_06_DM01CEL01) in group 2 mode 0x7f is now being offlined WARNING: Disk 4 (DBFS_DG_CD_06_DM01CEL01) in group 2 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 4 (RECO_DM01_CD_04_DM01CEL01) in group 3 mode 0x7f is now being offlined WARNING: Disk 4 (RECO_DM01_CD_04_DM01CEL01) in group 3 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 6 (DATA_DM01_CD_06_DM01CEL01) in group 1 mode 0x7f is now being offlined WARNING: Disk 6 (DATA_DM01_CD_06_DM01CEL01) in group 1 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 5 (DBFS_DG_CD_07_DM01CEL01) in group 2 mode 0x7f is now being offlined WARNING: Disk 5 (DBFS_DG_CD_07_DM01CEL01) in group 2 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 5 (RECO_DM01_CD_05_DM01CEL01) in group 3 mode 0x7f is now being offlined WARNING: Disk 5 (RECO_DM01_CD_05_DM01CEL01) in group 3 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 7 (DATA_DM01_CD_07_DM01CEL01) in group 1 mode 0x7f is now being offlined WARNING: Disk 7 (DATA_DM01_CD_07_DM01CEL01) in group 1 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 6 (DBFS_DG_CD_08_DM01CEL01) in group 2 mode 0x7f is now being offlined WARNING: Disk 6 (DBFS_DG_CD_08_DM01CEL01) in group 2 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 6 (RECO_DM01_CD_06_DM01CEL01) in group 3 mode 0x7f is now being offlined WARNING: Disk 6 (RECO_DM01_CD_06_DM01CEL01) in group 3 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 8 (DATA_DM01_CD_08_DM01CEL01) in group 1 mode 0x7f is now being offlined WARNING: Disk 8 (DATA_DM01_CD_08_DM01CEL01) in group 1 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 7 (DBFS_DG_CD_09_DM01CEL01) in group 2 mode 0x7f is now being offlined WARNING: Disk 7 (DBFS_DG_CD_09_DM01CEL01) in group 2 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 7 (RECO_DM01_CD_07_DM01CEL01) in group 3 mode 0x7f is now being offlined WARNING: Disk 7 (RECO_DM01_CD_07_DM01CEL01) in group 3 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 9 (DATA_DM01_CD_09_DM01CEL01) in group 1 mode 0x7f is now being offlined WARNING: Disk 9 (DATA_DM01_CD_09_DM01CEL01) in group 1 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 8 (DBFS_DG_CD_10_DM01CEL01) in group 2 mode 0x7f is now being offlined WARNING: Disk 8 (DBFS_DG_CD_10_DM01CEL01) in group 2 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 8 (RECO_DM01_CD_08_DM01CEL01) in group 3 mode 0x7f is now being offlined WARNING: Disk 8 (RECO_DM01_CD_08_DM01CEL01) in group 3 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 10 (DATA_DM01_CD_10_DM01CEL01) in group 1 mode 0x7f is now being offlined WARNING: Disk 10 (DATA_DM01_CD_10_DM01CEL01) in group 1 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 9 (DBFS_DG_CD_11_DM01CEL01) in group 2 mode 0x7f is now being offlined WARNING: Disk 9 (DBFS_DG_CD_11_DM01CEL01) in group 2 in mode 0x7f is now being taken offline on ASM inst 1 NOTE: initiating PST update: grp = 2, dsk = 0/0xeae31f51, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 2, dsk = 1/0xeae31f4a, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 2, dsk = 2/0xeae31f4c, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 2, dsk = 3/0xeae31f4f, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 2, dsk = 4/0xeae31f49, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 2, dsk = 5/0xeae31f4e, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 2, dsk = 6/0xeae31f50, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 2, dsk = 7/0xeae31f48, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 2, dsk = 8/0xeae31f4b, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 2, dsk = 9/0xeae31f4d, mask = 0x6a, op = clear WARNING: Disk 9 (RECO_DM01_CD_09_DM01CEL01) in group 3 mode 0x7f is now being offlined WARNING: Disk 9 (RECO_DM01_CD_09_DM01CEL01) in group 3 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 11 (DATA_DM01_CD_11_DM01CEL01) in group 1 mode 0x7f is now being offlined WARNING: Disk 11 (DATA_DM01_CD_11_DM01CEL01) in group 1 in mode 0x7f is now being taken offline on ASM inst 1 NOTE: initiating PST update: grp = 1, dsk = 0/0xeae31f2e, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 1, dsk = 1/0xeae31f29, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 1, dsk = 2/0xeae31f2f, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 1, dsk = 3/0xeae31f28, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 1, dsk = 4/0xeae31f33, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 1, dsk = 5/0xeae31f2d, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 1, dsk = 6/0xeae31f2c, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 1, dsk = 7/0xeae31f30, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 1, dsk = 8/0xeae31f31, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 1, dsk = 9/0xeae31f2b, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 1, dsk = 10/0xeae31f2a, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 1, dsk = 11/0xeae31f32, mask = 0x6a, op = clear WARNING: Disk 10 (RECO_DM01_CD_10_DM01CEL01) in group 3 mode 0x7f is now being offlined WARNING: Disk 10 (RECO_DM01_CD_10_DM01CEL01) in group 3 in mode 0x7f is now being taken offline on ASM inst 1 WARNING: Disk 11 (RECO_DM01_CD_11_DM01CEL01) in group 3 mode 0x7f is now being offlined WARNING: Disk 11 (RECO_DM01_CD_11_DM01CEL01) in group 3 in mode 0x7f is now being taken offline on ASM inst 1 NOTE: initiating PST update: grp = 3, dsk = 0/0xeae31f71, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 3, dsk = 1/0xeae31f6b, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 3, dsk = 2/0xeae31f6c, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 3, dsk = 3/0xeae31f74, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 3, dsk = 4/0xeae31f6d, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 3, dsk = 5/0xeae31f73, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 3, dsk = 6/0xeae31f70, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 3, dsk = 7/0xeae31f72, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 3, dsk = 8/0xeae31f6e, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 3, dsk = 9/0xeae31f6a, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 3, dsk = 10/0xeae31f6f, mask = 0x6a, op = clear NOTE: initiating PST update: grp = 3, dsk = 11/0xeae31f75, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 13 for pid 30, osid 21302 NOTE: group DATA_DM01: updated PST location: disk 0012 (PST copy 0) NOTE: group DATA_DM01: updated PST location: disk 0024 (PST copy 1) NOTE: group DATA_DM01: updated PST location: disk 0012 (PST copy 0) NOTE: group DATA_DM01: updated PST location: disk 0024 (PST copy 1) GMON updating disk modes for group 2 at 14 for pid 35, osid 21304 NOTE: group DBFS_DG: updated PST location: disk 0010 (PST copy 0) NOTE: group DBFS_DG: updated PST location: disk 0020 (PST copy 1) NOTE: group DBFS_DG: updated PST location: disk 0010 (PST copy 0) NOTE: group DBFS_DG: updated PST location: disk 0020 (PST copy 1) GMON updating disk modes for group 3 at 15 for pid 36, osid 21306 NOTE: PST update grp = 1 completed successfully NOTE: initiating PST update: grp = 1, dsk = 0/0xeae31f2e, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 1, dsk = 1/0xeae31f29, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 1, dsk = 2/0xeae31f2f, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 1, dsk = 3/0xeae31f28, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 1, dsk = 4/0xeae31f33, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 1, dsk = 5/0xeae31f2d, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 1, dsk = 6/0xeae31f2c, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 1, dsk = 7/0xeae31f30, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 1, dsk = 8/0xeae31f31, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 1, dsk = 9/0xeae31f2b, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 1, dsk = 10/0xeae31f2a, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 1, dsk = 11/0xeae31f32, mask = 0x7e, op = clear NOTE: group RECO_DM01: updated PST location: disk 0012 (PST copy 0) NOTE: group RECO_DM01: updated PST location: disk 0024 (PST copy 1) NOTE: group RECO_DM01: updated PST location: disk 0012 (PST copy 0) NOTE: group RECO_DM01: updated PST location: disk 0024 (PST copy 1) NOTE: PST update grp = 2 completed successfully NOTE: initiating PST update: grp = 2, dsk = 0/0xeae31f51, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 2, dsk = 1/0xeae31f4a, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 2, dsk = 2/0xeae31f4c, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 2, dsk = 3/0xeae31f4f, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 2, dsk = 4/0xeae31f49, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 2, dsk = 5/0xeae31f4e, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 2, dsk = 6/0xeae31f50, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 2, dsk = 7/0xeae31f48, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 2, dsk = 8/0xeae31f4b, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 2, dsk = 9/0xeae31f4d, mask = 0x7e, op = clear NOTE: PST update grp = 3 completed successfully NOTE: initiating PST update: grp = 3, dsk = 0/0xeae31f71, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 3, dsk = 1/0xeae31f6b, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 3, dsk = 2/0xeae31f6c, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 3, dsk = 3/0xeae31f74, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 3, dsk = 4/0xeae31f6d, mask = 0x7e, op = clear GMON updating disk modes for group 1 at 16 for pid 30, osid 21302 NOTE: initiating PST update: grp = 3, dsk = 5/0xeae31f73, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 3, dsk = 6/0xeae31f70, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 3, dsk = 7/0xeae31f72, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 3, dsk = 8/0xeae31f6e, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 3, dsk = 9/0xeae31f6a, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 3, dsk = 10/0xeae31f6f, mask = 0x7e, op = clear NOTE: initiating PST update: grp = 3, dsk = 11/0xeae31f75, mask = 0x7e, op = clear NOTE: group DATA_DM01: updated PST location: disk 0012 (PST copy 0) NOTE: group DATA_DM01: updated PST location: disk 0024 (PST copy 1) NOTE: group DATA_DM01: updated PST location: disk 0012 (PST copy 0) NOTE: group DATA_DM01: updated PST location: disk 0024 (PST copy 1) NOTE: cache closing disk 0 of grp 1: DATA_DM01_CD_00_DM01CEL01 NOTE: cache closing disk 1 of grp 1: DATA_DM01_CD_01_DM01CEL01 NOTE: cache closing disk 2 of grp 1: DATA_DM01_CD_02_DM01CEL01 NOTE: cache closing disk 3 of grp 1: DATA_DM01_CD_03_DM01CEL01 NOTE: cache closing disk 4 of grp 1: DATA_DM01_CD_04_DM01CEL01 NOTE: cache closing disk 5 of grp 1: DATA_DM01_CD_05_DM01CEL01 NOTE: cache closing disk 6 of grp 1: DATA_DM01_CD_06_DM01CEL01 NOTE: cache closing disk 7 of grp 1: DATA_DM01_CD_07_DM01CEL01 NOTE: cache closing disk 8 of grp 1: DATA_DM01_CD_08_DM01CEL01 NOTE: cache closing disk 9 of grp 1: DATA_DM01_CD_09_DM01CEL01 NOTE: cache closing disk 10 of grp 1: DATA_DM01_CD_10_DM01CEL01 NOTE: cache closing disk 11 of grp 1: DATA_DM01_CD_11_DM01CEL01 GMON updating disk modes for group 2 at 17 for pid 35, osid 21304 NOTE: group DBFS_DG: updated PST location: disk 0010 (PST copy 0) NOTE: group DBFS_DG: updated PST location: disk 0020 (PST copy 1) NOTE: group DBFS_DG: updated PST location: disk 0010 (PST copy 0) NOTE: group DBFS_DG: updated PST location: disk 0020 (PST copy 1) NOTE: cache closing disk 0 of grp 2: DBFS_DG_CD_02_DM01CEL01 NOTE: cache closing disk 1 of grp 2: DBFS_DG_CD_03_DM01CEL01 NOTE: cache closing disk 2 of grp 2: DBFS_DG_CD_04_DM01CEL01 NOTE: cache closing disk 3 of grp 2: DBFS_DG_CD_05_DM01CEL01 NOTE: cache closing disk 4 of grp 2: DBFS_DG_CD_06_DM01CEL01 NOTE: cache closing disk 5 of grp 2: DBFS_DG_CD_07_DM01CEL01 NOTE: cache closing disk 6 of grp 2: DBFS_DG_CD_08_DM01CEL01 NOTE: cache closing disk 7 of grp 2: DBFS_DG_CD_09_DM01CEL01 NOTE: cache closing disk 8 of grp 2: DBFS_DG_CD_10_DM01CEL01 NOTE: cache closing disk 9 of grp 2: DBFS_DG_CD_11_DM01CEL01 NOTE: PST update grp = 1 completed successfully GMON updating disk modes for group 3 at 18 for pid 36, osid 21306 NOTE: group RECO_DM01: updated PST location: disk 0012 (PST copy 0) NOTE: group RECO_DM01: updated PST location: disk 0024 (PST copy 1) NOTE: group RECO_DM01: updated PST location: disk 0012 (PST copy 0) NOTE: group RECO_DM01: updated PST location: disk 0024 (PST copy 1) NOTE: cache closing disk 0 of grp 3: RECO_DM01_CD_00_DM01CEL01 NOTE: cache closing disk 1 of grp 3: RECO_DM01_CD_01_DM01CEL01 NOTE: cache closing disk 2 of grp 3: RECO_DM01_CD_02_DM01CEL01 NOTE: cache closing disk 3 of grp 3: RECO_DM01_CD_03_DM01CEL01 NOTE: cache closing disk 4 of grp 3: RECO_DM01_CD_04_DM01CEL01 NOTE: cache closing disk 5 of grp 3: RECO_DM01_CD_05_DM01CEL01 NOTE: cache closing disk 6 of grp 3: RECO_DM01_CD_06_DM01CEL01 NOTE: cache closing disk 7 of grp 3: RECO_DM01_CD_07_DM01CEL01 NOTE: cache closing disk 8 of grp 3: RECO_DM01_CD_08_DM01CEL01 NOTE: cache closing disk 9 of grp 3: RECO_DM01_CD_09_DM01CEL01 NOTE: cache closing disk 10 of grp 3: RECO_DM01_CD_10_DM01CEL01 NOTE: cache closing disk 11 of grp 3: RECO_DM01_CD_11_DM01CEL01 NOTE: PST update grp = 2 completed successfully NOTE: PST update grp = 3 completed successfully Sun Sep 02 09:08:49 2012 NOTE: Attempting voting file refresh on diskgroup DBFS_DG NOTE: Voting file relocation is required in diskgroup DBFS_DG Sun Sep 02 09:08:51 2012 NOTE: successfully read ACD block gn=3 blk=11264 via retry read Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_30748.trc: ORA-15062: ASM disk is globally closed NOTE: Attempting voting file refresh on diskgroup DBFS_DG NOTE: Voting file relocation is required in diskgroup DBFS_DG NOTE: Attempting voting file relocation on diskgroup DBFS_DG NOTE: successfully read ACD block gn=3 blk=11264 via retry read Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_30748.trc: ORA-15062: ASM disk is globally closed Sun Sep 02 09:10:06 2012 NOTE: successfully read ACD block gn=3 blk=11264 via retry read Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_30748.trc: ORA-15062: ASM disk is globally closed NOTE: successfully read ACD block gn=3 blk=11264 via retry read Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_30748.trc: ORA-15062: ASM disk is globally closed Sun Sep 02 09:10:39 2012 NOTE: successfully read ACD block gn=1 blk=11264 via retry read Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_30748.trc: ORA-15062: ASM disk is globally closed NOTE: successfully read ACD block gn=1 blk=11264 via retry read Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_30748.trc: ORA-15062: ASM disk is globally closed NOTE: successfully read ACD block gn=1 blk=11264 via retry read Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_30748.trc: ORA-15062: ASM disk is globally closed *** 2012-09-02 09:08:48.881 Box name 0 - 192.168.64.132 OSS OS Pid - 12253 Reconnect: Attempts: 1 Last TS: 5693385570 Dumping SKGXP connection state: Band 0: port ID - 0x1267e4a8, connection - 0x126d98d0 Dumping SKGXP connection state: Band 1: port ID - 0x12714de8, connection - 0x126aab90 Dumping SKGXP connection state: Band 2: port ID - 0x126a20d8, connection - 0x126865a0 Dumping SKGXP connection state: Band 3: port ID - 0x126662b8, connection - 0x1269c680 Dumping SKGXP connection state: Band 4: port ID - 0x1271b048, connection - 0x126b2130 Dumping SKGXP connection state: Band 5: port ID - 0x12715188, connection - 0x126d2330 Dumping SKGXP connection state: Band 6: port ID - 0x126a26d8, connection - 0x12691610 Dumping SKGXP connection state: Band 7: port ID - 0x126a2728, connection - 0x126a35f0 Dumping SKGXP connection state: Band 8: port ID - 0x126a2778, connection - 0x126b96d0 Reconnecting to box 0x126c70a0 ... Storage box 0x126c70a0 Inc: 1 with the source id 3379307313 Box name 0 - 192.168.64.133 OSS OS Pid - 12308 Reconnect: Attempts: 1 Last TS: 5693385570 Dumping SKGXP connection state: Band 0: port ID - 0x126a2128, connection - 0x126a70c0 Dumping SKGXP connection state: Band 1: port ID - 0x126a2598, connection - 0x126950e0 Dumping SKGXP connection state: Band 2: port ID - 0x12666308, connection - 0x126e0e70 Dumping SKGXP connection state: Band 3: port ID - 0x1271b0a8, connection - 0x126d5e00 Dumping SKGXP connection state: Band 4: port ID - 0x126666c8, connection - 0x1270ed60 Dumping SKGXP connection state: Band 5: port ID - 0x12714ed8, connection - 0x1268a070 Dumping SKGXP connection state: Band 6: port ID - 0x1271afe8, connection - 0x12682ad0 Dumping SKGXP connection state: Band 7: port ID - 0x126661c8, connection - 0x126c0c70 Dumping SKGXP connection state: Band 8: port ID - 0x12666218, connection - 0x126c72c0 Reissuing requests for the box 0x126c70a0 Reissuing requests for the box 0x126a33d0 *** 2012-09-02 09:08:51.185 NOTE: successfully read ACD block gn=3 blk=11264 via retry read ORA-15062: ASM disk is globally closed
RDBMS ALERT.loG
Sun Sep 02 05:59:59 2012
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Sun Sep 02 05:59:59 2012
Starting background process VKRM
Sun Sep 02 05:59:59 2012
VKRM started with pid=83, OS id=5062
Sun Sep 02 09:08:48 2012
NOTE: disk 0 (DATA_DM01_CD_00_DM01CEL01) in group 1 (DATA_DM01) is offline for reads
NOTE: disk 1 (DATA_DM01_CD_01_DM01CEL01) in group 1 (DATA_DM01) is offline for reads
NOTE: disk 2 (DATA_DM01_CD_02_DM01CEL01) in group 1 (DATA_DM01) is offline for reads
NOTE: disk 3 (DATA_DM01_CD_03_DM01CEL01) in group 1 (DATA_DM01) is offline for reads
NOTE: disk 4 (DATA_DM01_CD_04_DM01CEL01) in group 1 (DATA_DM01) is offline for reads
NOTE: disk 5 (DATA_DM01_CD_05_DM01CEL01) in group 1 (DATA_DM01) is offline for reads
NOTE: disk 6 (DATA_DM01_CD_06_DM01CEL01) in group 1 (DATA_DM01) is offline for reads
NOTE: disk 7 (DATA_DM01_CD_07_DM01CEL01) in group 1 (DATA_DM01) is offline for reads
NOTE: disk 8 (DATA_DM01_CD_08_DM01CEL01) in group 1 (DATA_DM01) is offline for reads
NOTE: disk 9 (DATA_DM01_CD_09_DM01CEL01) in group 1 (DATA_DM01) is offline for reads
NOTE: disk 10 (DATA_DM01_CD_10_DM01CEL01) in group 1 (DATA_DM01) is offline for reads
NOTE: disk 11 (DATA_DM01_CD_11_DM01CEL01) in group 1 (DATA_DM01) is offline for reads
NOTE: disk 0 (RECO_DM01_CD_00_DM01CEL01) in group 3 (RECO_DM01) is offline for reads
NOTE: disk 1 (RECO_DM01_CD_01_DM01CEL01) in group 3 (RECO_DM01) is offline for reads
NOTE: disk 2 (RECO_DM01_CD_02_DM01CEL01) in group 3 (RECO_DM01) is offline for reads
NOTE: disk 3 (RECO_DM01_CD_03_DM01CEL01) in group 3 (RECO_DM01) is offline for reads
NOTE: disk 4 (RECO_DM01_CD_04_DM01CEL01) in group 3 (RECO_DM01) is offline for reads
NOTE: disk 5 (RECO_DM01_CD_05_DM01CEL01) in group 3 (RECO_DM01) is offline for reads
NOTE: disk 6 (RECO_DM01_CD_06_DM01CEL01) in group 3 (RECO_DM01) is offline for reads
NOTE: disk 7 (RECO_DM01_CD_07_DM01CEL01) in group 3 (RECO_DM01) is offline for reads
NOTE: disk 8 (RECO_DM01_CD_08_DM01CEL01) in group 3 (RECO_DM01) is offline for reads
NOTE: disk 9 (RECO_DM01_CD_09_DM01CEL01) in group 3 (RECO_DM01) is offline for reads
NOTE: disk 10 (RECO_DM01_CD_10_DM01CEL01) in group 3 (RECO_DM01) is offline for reads
NOTE: disk 11 (RECO_DM01_CD_11_DM01CEL01) in group 3 (RECO_DM01) is offline for reads
NOTE: disk 0 (DATA_DM01_CD_00_DM01CEL01) in group 1 (DATA_DM01) is offline for writes
NOTE: disk 1 (DATA_DM01_CD_01_DM01CEL01) in group 1 (DATA_DM01) is offline for writes
NOTE: disk 2 (DATA_DM01_CD_02_DM01CEL01) in group 1 (DATA_DM01) is offline for writes
NOTE: disk 3 (DATA_DM01_CD_03_DM01CEL01) in group 1 (DATA_DM01) is offline for writes
NOTE: disk 4 (DATA_DM01_CD_04_DM01CEL01) in group 1 (DATA_DM01) is offline for writes
NOTE: disk 5 (DATA_DM01_CD_05_DM01CEL01) in group 1 (DATA_DM01) is offline for writes
NOTE: disk 6 (DATA_DM01_CD_06_DM01CEL01) in group 1 (DATA_DM01) is offline for writes
NOTE: disk 7 (DATA_DM01_CD_07_DM01CEL01) in group 1 (DATA_DM01) is offline for writes
NOTE: disk 8 (DATA_DM01_CD_08_DM01CEL01) in group 1 (DATA_DM01) is offline for writes
NOTE: disk 9 (DATA_DM01_CD_09_DM01CEL01) in group 1 (DATA_DM01) is offline for writes
NOTE: disk 10 (DATA_DM01_CD_10_DM01CEL01) in group 1 (DATA_DM01) is offline for writes
NOTE: disk 11 (DATA_DM01_CD_11_DM01CEL01) in group 1 (DATA_DM01) is offline for writes
NOTE: disk 0 (RECO_DM01_CD_00_DM01CEL01) in group 3 (RECO_DM01) is offline for writes
NOTE: disk 1 (RECO_DM01_CD_01_DM01CEL01) in group 3 (RECO_DM01) is offline for writes
NOTE: disk 2 (RECO_DM01_CD_02_DM01CEL01) in group 3 (RECO_DM01) is offline for writes
NOTE: disk 3 (RECO_DM01_CD_03_DM01CEL01) in group 3 (RECO_DM01) is offline for writes
NOTE: disk 4 (RECO_DM01_CD_04_DM01CEL01) in group 3 (RECO_DM01) is offline for writes
NOTE: disk 5 (RECO_DM01_CD_05_DM01CEL01) in group 3 (RECO_DM01) is offline for writes
NOTE: disk 6 (RECO_DM01_CD_06_DM01CEL01) in group 3 (RECO_DM01) is offline for writes
NOTE: disk 7 (RECO_DM01_CD_07_DM01CEL01) in group 3 (RECO_DM01) is offline for writes
NOTE: disk 8 (RECO_DM01_CD_08_DM01CEL01) in group 3 (RECO_DM01) is offline for writes
NOTE: disk 9 (RECO_DM01_CD_09_DM01CEL01) in group 3 (RECO_DM01) is offline for writes
NOTE: disk 10 (RECO_DM01_CD_10_DM01CEL01) in group 3 (RECO_DM01) is offline for writes
NOTE: disk 11 (RECO_DM01_CD_11_DM01CEL01) in group 3 (RECO_DM01) is offline for writes
Sun Sep 02 09:08:48 2012
Errors in file /u01/app/oracle/diag/rdbms/dbm/dbm1/trace/dbm1_ckpt_31547.trc:
ORA-27603: Cell storage I/O error, I/O failed on disk o/192.168.64.131/RECO_DM01_CD_03_dm01cel01 at offset 16826368 for data length 16384
ORA-27626: Exadata error: 12 (Network error)
ORA-27300: OS system dependent operation:rpc update timed out failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxp_path
WARNING: Write Failed. group:3 disk:3 AU:4 offset:49152 size:16384
*** 2012-09-02 09:08:48.865
ORA-27626: Exadata error: 12 (Network error)
ORA-27300: OS system dependent operation:rpc update timed out failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxp_path
WARNING: Write Failed. group:3 disk:3 AU:4 offset:49152 size:16384
path:o/192.168.64.131/RECO_DM01_CD_03_dm01cel01
incarnation:0xeae31f74 asynchronous result:'I/O error'
subsys:OSS iop:0x2adfeda7d130 bufp:0x2adfed997e00 osderr:0xc osderr1:0x0
Exadata error:'Network error'
WARNING: (post-reap) disk offline and rejecting I/O
dsk: 3, au: 4, fn: 256, ext: 0
SQL> select status from v$datafile;
STATUS
-------
SYSTEM
ONLINE
ONLINE
ONLINE
ONLINE
ONLINE
6 rows selected.
丢失一个cell的情况下数据库所有数据文件仍ONLINE可操作状态
重启CELL，观察恢复状态：
CellCLI> alter cell startup services all
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services...
running
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...
The STARTUP of MS services was successful.
CellCLI> list griddisk
DATA_DM01_CD_00_dm01cel01 active
DATA_DM01_CD_01_dm01cel01 active
DATA_DM01_CD_02_dm01cel01 active
DATA_DM01_CD_03_dm01cel01 active
DATA_DM01_CD_04_dm01cel01 active
DATA_DM01_CD_05_dm01cel01 active
DATA_DM01_CD_06_dm01cel01 active
DATA_DM01_CD_07_dm01cel01 active
DATA_DM01_CD_08_dm01cel01 active
DATA_DM01_CD_09_dm01cel01 active
DATA_DM01_CD_10_dm01cel01 active
DATA_DM01_CD_11_dm01cel01 active
DBFS_DG_CD_02_dm01cel01 active
DBFS_DG_CD_03_dm01cel01 active
DBFS_DG_CD_04_dm01cel01 active
DBFS_DG_CD_05_dm01cel01 active
DBFS_DG_CD_06_dm01cel01 active
DBFS_DG_CD_07_dm01cel01 active
DBFS_DG_CD_08_dm01cel01 active
DBFS_DG_CD_09_dm01cel01 active
DBFS_DG_CD_10_dm01cel01 active
DBFS_DG_CD_11_dm01cel01 active
RECO_DM01_CD_00_dm01cel01 active
RECO_DM01_CD_01_dm01cel01 active
RECO_DM01_CD_02_dm01cel01 active
RECO_DM01_CD_03_dm01cel01 active
RECO_DM01_CD_04_dm01cel01 active
RECO_DM01_CD_05_dm01cel01 active
RECO_DM01_CD_06_dm01cel01 active
RECO_DM01_CD_07_dm01cel01 active
RECO_DM01_CD_08_dm01cel01 active
RECO_DM01_CD_09_dm01cel01 active
RECO_DM01_CD_10_dm01cel01 active
RECO_DM01_CD_11_dm01cel01 active
ASM ALERT.LOG
Sun Sep 02 09:15:18 2012
WARNING: Disk 0 (DATA_DM01_CD_00_DM01CEL01) in group 1 mode 0x1 is now being offlined
WARNING: Disk 1 (DATA_DM01_CD_01_DM01CEL01) in group 1 mode 0x1 is now being offlined
WARNING: Disk 2 (DATA_DM01_CD_02_DM01CEL01) in group 1 mode 0x1 is now being offlined
WARNING: Disk 3 (DATA_DM01_CD_03_DM01CEL01) in group 1 mode 0x1 is now being offlined
WARNING: Disk 4 (DATA_DM01_CD_04_DM01CEL01) in group 1 mode 0x1 is now being offlined
WARNING: Disk 5 (DATA_DM01_CD_05_DM01CEL01) in group 1 mode 0x1 is now being offlined
WARNING: Disk 6 (DATA_DM01_CD_06_DM01CEL01) in group 1 mode 0x1 is now being offlined
WARNING: Disk 7 (DATA_DM01_CD_07_DM01CEL01) in group 1 mode 0x1 is now being offlined
WARNING: Disk 8 (DATA_DM01_CD_08_DM01CEL01) in group 1 mode 0x1 is now being offlined
WARNING: Disk 9 (DATA_DM01_CD_09_DM01CEL01) in group 1 mode 0x1 is now being offlined
WARNING: Disk 10 (DATA_DM01_CD_10_DM01CEL01) in group 1 mode 0x1 is now being offlined
WARNING: Disk 11 (DATA_DM01_CD_11_DM01CEL01) in group 1 mode 0x1 is now being offlined
Sun Sep 02 09:15:19 2012
Starting background process XDWK
Sun Sep 02 09:15:19 2012
XDWK started with pid=35, OS id=26059
Sun Sep 02 09:15:19 2012
NOTE: disk validation pending for group 1/0x34e3ee5a (DATA_DM01)
NOTE: Found o/192.168.64.131/DATA_DM01_CD_03_dm01cel01 for disk DATA_DM01_CD_03_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DATA_DM01_CD_01_dm01cel01 for disk DATA_DM01_CD_01_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DATA_DM01_CD_10_dm01cel01 for disk DATA_DM01_CD_10_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DATA_DM01_CD_09_dm01cel01 for disk DATA_DM01_CD_09_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DATA_DM01_CD_06_dm01cel01 for disk DATA_DM01_CD_06_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DATA_DM01_CD_05_dm01cel01 for disk DATA_DM01_CD_05_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DATA_DM01_CD_00_dm01cel01 for disk DATA_DM01_CD_00_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DATA_DM01_CD_02_dm01cel01 for disk DATA_DM01_CD_02_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DATA_DM01_CD_07_dm01cel01 for disk DATA_DM01_CD_07_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DATA_DM01_CD_08_dm01cel01 for disk DATA_DM01_CD_08_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DATA_DM01_CD_11_dm01cel01 for disk DATA_DM01_CD_11_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DATA_DM01_CD_04_dm01cel01 for disk DATA_DM01_CD_04_DM01CEL01
WARNING: ignoring disk in deep discovery
SUCCESS: validated disks for 1/0x34e3ee5a (DATA_DM01)
NOTE: membership refresh pending for group 1/0x34e3ee5a (DATA_DM01)
Sun Sep 02 09:15:24 2012
NOTE: successfully read ACD block gn=1 blk=11264 via retry read
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_30748.trc:
ORA-15062: ASM disk is globally closed
GMON querying group 1 at 19 for pid 19, osid 30754
NOTE: cache opening disk 0 of grp 1: DATA_DM01_CD_00_DM01CEL01 path:o/192.168.64.131/DATA_DM01_CD_00_dm01cel01
NOTE: cache opening disk 1 of grp 1: DATA_DM01_CD_01_DM01CEL01 path:o/192.168.64.131/DATA_DM01_CD_01_dm01cel01
NOTE: cache opening disk 2 of grp 1: DATA_DM01_CD_02_DM01CEL01 path:o/192.168.64.131/DATA_DM01_CD_02_dm01cel01
NOTE: cache opening disk 3 of grp 1: DATA_DM01_CD_03_DM01CEL01 path:o/192.168.64.131/DATA_DM01_CD_03_dm01cel01
NOTE: cache opening disk 4 of grp 1: DATA_DM01_CD_04_DM01CEL01 path:o/192.168.64.131/DATA_DM01_CD_04_dm01cel01
NOTE: cache opening disk 5 of grp 1: DATA_DM01_CD_05_DM01CEL01 path:o/192.168.64.131/DATA_DM01_CD_05_dm01cel01
NOTE: cache opening disk 6 of grp 1: DATA_DM01_CD_06_DM01CEL01 path:o/192.168.64.131/DATA_DM01_CD_06_dm01cel01
NOTE: cache opening disk 7 of grp 1: DATA_DM01_CD_07_DM01CEL01 path:o/192.168.64.131/DATA_DM01_CD_07_dm01cel01
NOTE: cache opening disk 8 of grp 1: DATA_DM01_CD_08_DM01CEL01 path:o/192.168.64.131/DATA_DM01_CD_08_dm01cel01
NOTE: cache opening disk 9 of grp 1: DATA_DM01_CD_09_DM01CEL01 path:o/192.168.64.131/DATA_DM01_CD_09_dm01cel01
NOTE: cache opening disk 10 of grp 1: DATA_DM01_CD_10_DM01CEL01 path:o/192.168.64.131/DATA_DM01_CD_10_dm01cel01
NOTE: cache opening disk 11 of grp 1: DATA_DM01_CD_11_DM01CEL01 path:o/192.168.64.131/DATA_DM01_CD_11_dm01cel01
SUCCESS: refreshed membership for 1/0x34e3ee5a (DATA_DM01)
WARNING: Disk 0 (DBFS_DG_CD_02_DM01CEL01) in group 2 mode 0x1 is now being offlined
WARNING: Disk 1 (DBFS_DG_CD_03_DM01CEL01) in group 2 mode 0x1 is now being offlined
WARNING: Disk 2 (DBFS_DG_CD_04_DM01CEL01) in group 2 mode 0x1 is now being offlined
WARNING: Disk 3 (DBFS_DG_CD_05_DM01CEL01) in group 2 mode 0x1 is now being offlined
WARNING: Disk 4 (DBFS_DG_CD_06_DM01CEL01) in group 2 mode 0x1 is now being offlined
WARNING: Disk 5 (DBFS_DG_CD_07_DM01CEL01) in group 2 mode 0x1 is now being offlined
WARNING: Disk 6 (DBFS_DG_CD_08_DM01CEL01) in group 2 mode 0x1 is now being offlined
WARNING: Disk 7 (DBFS_DG_CD_09_DM01CEL01) in group 2 mode 0x1 is now being offlined
WARNING: Disk 8 (DBFS_DG_CD_10_DM01CEL01) in group 2 mode 0x1 is now being offlined
WARNING: Disk 9 (DBFS_DG_CD_11_DM01CEL01) in group 2 mode 0x1 is now being offlined
NOTE: Voting File refresh pending for group 1/0x34e3ee5a (DATA_DM01)
NOTE: Attempting voting file refresh on diskgroup DATA_DM01
NOTE: disk validation pending for group 2/0x34f3ee5b (DBFS_DG)
Sun Sep 02 09:15:31 2012
NOTE: Attempting voting file refresh on diskgroup DBFS_DG
NOTE: Voting file relocation is required in diskgroup DBFS_DG
NOTE: Attempting voting file relocation on diskgroup DBFS_DG
NOTE: Found o/192.168.64.131/DBFS_DG_CD_09_dm01cel01 for disk DBFS_DG_CD_09_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DBFS_DG_CD_06_dm01cel01 for disk DBFS_DG_CD_06_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DBFS_DG_CD_03_dm01cel01 for disk DBFS_DG_CD_03_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DBFS_DG_CD_10_dm01cel01 for disk DBFS_DG_CD_10_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DBFS_DG_CD_04_dm01cel01 for disk DBFS_DG_CD_04_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DBFS_DG_CD_11_dm01cel01 for disk DBFS_DG_CD_11_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DBFS_DG_CD_07_dm01cel01 for disk DBFS_DG_CD_07_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DBFS_DG_CD_05_dm01cel01 for disk DBFS_DG_CD_05_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DBFS_DG_CD_08_dm01cel01 for disk DBFS_DG_CD_08_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/DBFS_DG_CD_02_dm01cel01 for disk DBFS_DG_CD_02_DM01CEL01
WARNING: ignoring disk in deep discovery
SUCCESS: validated disks for 2/0x34f3ee5b (DBFS_DG)
NOTE: membership refresh pending for group 2/0x34f3ee5b (DBFS_DG)
NOTE: successfully read ACD block gn=2 blk=11264 via retry read
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_30748.trc:
ORA-15062: ASM disk is globally closed
NOTE: Attempting voting file refresh on diskgroup DBFS_DG
NOTE: Voting file relocation is required in diskgroup DBFS_DG
NOTE: Attempting voting file relocation on diskgroup DBFS_DG
Sun Sep 02 09:15:35 2012
GMON querying group 2 at 20 for pid 19, osid 30754
NOTE: cache opening disk 0 of grp 2: DBFS_DG_CD_02_DM01CEL01 path:o/192.168.64.131/DBFS_DG_CD_02_dm01cel01
NOTE: cache opening disk 1 of grp 2: DBFS_DG_CD_03_DM01CEL01 path:o/192.168.64.131/DBFS_DG_CD_03_dm01cel01
NOTE: cache opening disk 2 of grp 2: DBFS_DG_CD_04_DM01CEL01 path:o/192.168.64.131/DBFS_DG_CD_04_dm01cel01
NOTE: cache opening disk 3 of grp 2: DBFS_DG_CD_05_DM01CEL01 path:o/192.168.64.131/DBFS_DG_CD_05_dm01cel01
NOTE: cache opening disk 4 of grp 2: DBFS_DG_CD_06_DM01CEL01 path:o/192.168.64.131/DBFS_DG_CD_06_dm01cel01
NOTE: cache opening disk 5 of grp 2: DBFS_DG_CD_07_DM01CEL01 path:o/192.168.64.131/DBFS_DG_CD_07_dm01cel01
NOTE: cache opening disk 6 of grp 2: DBFS_DG_CD_08_DM01CEL01 path:o/192.168.64.131/DBFS_DG_CD_08_dm01cel01
NOTE: cache opening disk 7 of grp 2: DBFS_DG_CD_09_DM01CEL01 path:o/192.168.64.131/DBFS_DG_CD_09_dm01cel01
NOTE: cache opening disk 8 of grp 2: DBFS_DG_CD_10_DM01CEL01 path:o/192.168.64.131/DBFS_DG_CD_10_dm01cel01
NOTE: cache opening disk 9 of grp 2: DBFS_DG_CD_11_DM01CEL01 path:o/192.168.64.131/DBFS_DG_CD_11_dm01cel01
SUCCESS: refreshed membership for 2/0x34f3ee5b (DBFS_DG)
WARNING: Disk 0 (RECO_DM01_CD_00_DM01CEL01) in group 3 mode 0x1 is now being offlined
WARNING: Disk 1 (RECO_DM01_CD_01_DM01CEL01) in group 3 mode 0x1 is now being offlined
WARNING: Disk 2 (RECO_DM01_CD_02_DM01CEL01) in group 3 mode 0x1 is now being offlined
WARNING: Disk 3 (RECO_DM01_CD_03_DM01CEL01) in group 3 mode 0x1 is now being offlined
WARNING: Disk 4 (RECO_DM01_CD_04_DM01CEL01) in group 3 mode 0x1 is now being offlined
WARNING: Disk 5 (RECO_DM01_CD_05_DM01CEL01) in group 3 mode 0x1 is now being offlined
WARNING: Disk 6 (RECO_DM01_CD_06_DM01CEL01) in group 3 mode 0x1 is now being offlined
WARNING: Disk 7 (RECO_DM01_CD_07_DM01CEL01) in group 3 mode 0x1 is now being offlined
WARNING: Disk 8 (RECO_DM01_CD_08_DM01CEL01) in group 3 mode 0x1 is now being offlined
WARNING: Disk 9 (RECO_DM01_CD_09_DM01CEL01) in group 3 mode 0x1 is now being offlined
WARNING: Disk 10 (RECO_DM01_CD_10_DM01CEL01) in group 3 mode 0x1 is now being offlined
WARNING: Disk 11 (RECO_DM01_CD_11_DM01CEL01) in group 3 mode 0x1 is now being offlined
NOTE: Voting File refresh pending for group 2/0x34f3ee5b (DBFS_DG)
NOTE: Attempting voting file refresh on diskgroup DBFS_DG
NOTE: Voting file relocation is required in diskgroup DBFS_DG
NOTE: Attempting voting file relocation on diskgroup DBFS_DG
NOTE: voting file allocation on grp 2 disk DBFS_DG_CD_02_DM01CEL01
Sun Sep 02 09:16:09 2012
NOTE: disk validation pending for group 3/0x34f3ee5c (RECO_DM01)
NOTE: Found o/192.168.64.131/RECO_DM01_CD_09_dm01cel01 for disk RECO_DM01_CD_09_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/RECO_DM01_CD_01_dm01cel01 for disk RECO_DM01_CD_01_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/RECO_DM01_CD_02_dm01cel01 for disk RECO_DM01_CD_02_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/RECO_DM01_CD_04_dm01cel01 for disk RECO_DM01_CD_04_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/RECO_DM01_CD_08_dm01cel01 for disk RECO_DM01_CD_08_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/RECO_DM01_CD_10_dm01cel01 for disk RECO_DM01_CD_10_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/RECO_DM01_CD_06_dm01cel01 for disk RECO_DM01_CD_06_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/RECO_DM01_CD_00_dm01cel01 for disk RECO_DM01_CD_00_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/RECO_DM01_CD_07_dm01cel01 for disk RECO_DM01_CD_07_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/RECO_DM01_CD_05_dm01cel01 for disk RECO_DM01_CD_05_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/RECO_DM01_CD_03_dm01cel01 for disk RECO_DM01_CD_03_DM01CEL01
WARNING: ignoring disk in deep discovery
NOTE: Found o/192.168.64.131/RECO_DM01_CD_11_dm01cel01 for disk RECO_DM01_CD_11_DM01CEL01
WARNING: ignoring disk in deep discovery
SUCCESS: validated disks for 3/0x34f3ee5c (RECO_DM01)
Sun Sep 02 09:16:12 2012
NOTE: group RECO_DM01: updated PST location: disk 0012 (PST copy 0)
NOTE: group RECO_DM01: updated PST location: disk 0024 (PST copy 1)
NOTE: group RECO_DM01: updated PST location: disk 0000 (PST copy 2)
NOTE: membership refresh pending for group 3/0x34f3ee5c (RECO_DM01)
WARNING: GMON found an alien heartbeat (grp 3)
Sun Sep 02 09:16:15 2012
NOTE: successfully read ACD block gn=3 blk=11264 via retry read
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_30748.trc:
ORA-15062: ASM disk is globally closed
GMON querying group 3 at 21 for pid 19, osid 30754
NOTE: group RECO_DM01: updated PST location: disk 0012 (PST copy 0)
NOTE: group RECO_DM01: updated PST location: disk 0024 (PST copy 1)
NOTE: group RECO_DM01: updated PST location: disk 0000 (PST copy 2)
NOTE: cache opening disk 0 of grp 3: RECO_DM01_CD_00_DM01CEL01 path:o/192.168.64.131/RECO_DM01_CD_00_dm01cel01
NOTE: cache opening disk 1 of grp 3: RECO_DM01_CD_01_DM01CEL01 path:o/192.168.64.131/RECO_DM01_CD_01_dm01cel01
NOTE: cache opening disk 2 of grp 3: RECO_DM01_CD_02_DM01CEL01 path:o/192.168.64.131/RECO_DM01_CD_02_dm01cel01
NOTE: cache opening disk 3 of grp 3: RECO_DM01_CD_03_DM01CEL01 path:o/192.168.64.131/RECO_DM01_CD_03_dm01cel01
NOTE: cache opening disk 4 of grp 3: RECO_DM01_CD_04_DM01CEL01 path:o/192.168.64.131/RECO_DM01_CD_04_dm01cel01
NOTE: cache opening disk 5 of grp 3: RECO_DM01_CD_05_DM01CEL01 path:o/192.168.64.131/RECO_DM01_CD_05_dm01cel01
NOTE: cache opening disk 6 of grp 3: RECO_DM01_CD_06_DM01CEL01 path:o/192.168.64.131/RECO_DM01_CD_06_dm01cel01
NOTE: cache opening disk 7 of grp 3: RECO_DM01_CD_07_DM01CEL01 path:o/192.168.64.131/RECO_DM01_CD_07_dm01cel01
NOTE: cache opening disk 8 of grp 3: RECO_DM01_CD_08_DM01CEL01 path:o/192.168.64.131/RECO_DM01_CD_08_dm01cel01
NOTE: cache opening disk 9 of grp 3: RECO_DM01_CD_09_DM01CEL01 path:o/192.168.64.131/RECO_DM01_CD_09_dm01cel01
NOTE: cache opening disk 10 of grp 3: RECO_DM01_CD_10_DM01CEL01 path:o/192.168.64.131/RECO_DM01_CD_10_dm01cel01
NOTE: cache opening disk 11 of grp 3: RECO_DM01_CD_11_DM01CEL01 path:o/192.168.64.131/RECO_DM01_CD_11_dm01cel01
SUCCESS: refreshed membership for 3/0x34f3ee5c (RECO_DM01)
NOTE: Voting File refresh pending for group 3/0x34f3ee5c (RECO_DM01)
NOTE: Attempting voting file refresh on diskgroup RECO_DM01
RDBMS ALERT
Sun Sep 02 09:15:23 2012
NOTE: Found o/192.168.64.131/DATA_DM01_CD_00_dm01cel01 for disk DATA_DM01_CD_00_DM01CEL01
SUCCESS: disk DATA_DM01_CD_00_DM01CEL01 (0.3940753198) replaced in diskgroup DATA_DM01
NOTE: Found o/192.168.64.131/DATA_DM01_CD_01_dm01cel01 for disk DATA_DM01_CD_01_DM01CEL01
SUCCESS: disk DATA_DM01_CD_01_DM01CEL01 (1.3940753193) replaced in diskgroup DATA_DM01
NOTE: Found o/192.168.64.131/DATA_DM01_CD_02_dm01cel01 for disk DATA_DM01_CD_02_DM01CEL01
SUCCESS: disk DATA_DM01_CD_02_DM01CEL01 (2.3940753199) replaced in diskgroup DATA_DM01
NOTE: Found o/192.168.64.131/DATA_DM01_CD_03_dm01cel01 for disk DATA_DM01_CD_03_DM01CEL01
SUCCESS: disk DATA_DM01_CD_03_DM01CEL01 (3.3940753192) replaced in diskgroup DATA_DM01
NOTE: Found o/192.168.64.131/DATA_DM01_CD_04_dm01cel01 for disk DATA_DM01_CD_04_DM01CEL01
SUCCESS: disk DATA_DM01_CD_04_DM01CEL01 (4.3940753203) replaced in diskgroup DATA_DM01
NOTE: Found o/192.168.64.131/DATA_DM01_CD_05_dm01cel01 for disk DATA_DM01_CD_05_DM01CEL01
SUCCESS: disk DATA_DM01_CD_05_DM01CEL01 (5.3940753197) replaced in diskgroup DATA_DM01
NOTE: Found o/192.168.64.131/DATA_DM01_CD_06_dm01cel01 for disk DATA_DM01_CD_06_DM01CEL01
SUCCESS: disk DATA_DM01_CD_06_DM01CEL01 (6.3940753196) replaced in diskgroup DATA_DM01
NOTE: Found o/192.168.64.131/DATA_DM01_CD_07_dm01cel01 for disk DATA_DM01_CD_07_DM01CEL01
SUCCESS: disk DATA_DM01_CD_07_DM01CEL01 (7.3940753200) replaced in diskgroup DATA_DM01
NOTE: Found o/192.168.64.131/DATA_DM01_CD_08_dm01cel01 for disk DATA_DM01_CD_08_DM01CEL01
SUCCESS: disk DATA_DM01_CD_08_DM01CEL01 (8.3940753201) replaced in diskgroup DATA_DM01
NOTE: Found o/192.168.64.131/DATA_DM01_CD_09_dm01cel01 for disk DATA_DM01_CD_09_DM01CEL01
SUCCESS: disk DATA_DM01_CD_09_DM01CEL01 (9.3940753195) replaced in diskgroup DATA_DM01
NOTE: Found o/192.168.64.131/DATA_DM01_CD_10_dm01cel01 for disk DATA_DM01_CD_10_DM01CEL01
SUCCESS: disk DATA_DM01_CD_10_DM01CEL01 (10.3940753194) replaced in diskgroup DATA_DM01
NOTE: Found o/192.168.64.131/DATA_DM01_CD_11_dm01cel01 for disk DATA_DM01_CD_11_DM01CEL01
SUCCESS: disk DATA_DM01_CD_11_DM01CEL01 (11.3940753202) replaced in diskgroup DATA_DM01
NOTE: disk 0 (DATA_DM01_CD_00_DM01CEL01) in group 1 (DATA_DM01) is online for writes
NOTE: disk 1 (DATA_DM01_CD_01_DM01CEL01) in group 1 (DATA_DM01) is online for writes
NOTE: disk 2 (DATA_DM01_CD_02_DM01CEL01) in group 1 (DATA_DM01) is online for writes
NOTE: disk 3 (DATA_DM01_CD_03_DM01CEL01) in group 1 (DATA_DM01) is online for writes
NOTE: disk 4 (DATA_DM01_CD_04_DM01CEL01) in group 1 (DATA_DM01) is online for writes
NOTE: disk 5 (DATA_DM01_CD_05_DM01CEL01) in group 1 (DATA_DM01) is online for writes
NOTE: disk 6 (DATA_DM01_CD_06_DM01CEL01) in group 1 (DATA_DM01) is online for writes
NOTE: disk 7 (DATA_DM01_CD_07_DM01CEL01) in group 1 (DATA_DM01) is online for writes
NOTE: disk 8 (DATA_DM01_CD_08_DM01CEL01) in group 1 (DATA_DM01) is online for writes
NOTE: disk 9 (DATA_DM01_CD_09_DM01CEL01) in group 1 (DATA_DM01) is online for writes
NOTE: disk 10 (DATA_DM01_CD_10_DM01CEL01) in group 1 (DATA_DM01) is online for writes
NOTE: disk 11 (DATA_DM01_CD_11_DM01CEL01) in group 1 (DATA_DM01) is online for writes
NOTE: disk 0 (DATA_DM01_CD_00_DM01CEL01) in group 1 (DATA_DM01) is online for reads
NOTE: disk 1 (DATA_DM01_CD_01_DM01CEL01) in group 1 (DATA_DM01) is online for reads
NOTE: disk 2 (DATA_DM01_CD_02_DM01CEL01) in group 1 (DATA_DM01) is online for reads
NOTE: disk 3 (DATA_DM01_CD_03_DM01CEL01) in group 1 (DATA_DM01) is online for reads
NOTE: disk 4 (DATA_DM01_CD_04_DM01CEL01) in group 1 (DATA_DM01) is online for reads
NOTE: disk 5 (DATA_DM01_CD_05_DM01CEL01) in group 1 (DATA_DM01) is online for reads
NOTE: disk 6 (DATA_DM01_CD_06_DM01CEL01) in group 1 (DATA_DM01) is online for reads
NOTE: disk 7 (DATA_DM01_CD_07_DM01CEL01) in group 1 (DATA_DM01) is online for reads
NOTE: disk 8 (DATA_DM01_CD_08_DM01CEL01) in group 1 (DATA_DM01) is online for reads
NOTE: disk 9 (DATA_DM01_CD_09_DM01CEL01) in group 1 (DATA_DM01) is online for reads
NOTE: disk 10 (DATA_DM01_CD_10_DM01CEL01) in group 1 (DATA_DM01) is online for reads
NOTE: disk 11 (DATA_DM01_CD_11_DM01CEL01) in group 1 (DATA_DM01) is online for reads
Sun Sep 02 09:16:12 2012
NOTE: Found o/192.168.64.131/RECO_DM01_CD_00_dm01cel01 for disk RECO_DM01_CD_00_DM01CEL01
SUCCESS: disk RECO_DM01_CD_00_DM01CEL01 (0.3940753265) replaced in diskgroup RECO_DM01
NOTE: Found o/192.168.64.131/RECO_DM01_CD_01_dm01cel01 for disk RECO_DM01_CD_01_DM01CEL01
SUCCESS: disk RECO_DM01_CD_01_DM01CEL01 (1.3940753259) replaced in diskgroup RECO_DM01
NOTE: Found o/192.168.64.131/RECO_DM01_CD_02_dm01cel01 for disk RECO_DM01_CD_02_DM01CEL01
SUCCESS: disk RECO_DM01_CD_02_DM01CEL01 (2.3940753260) replaced in diskgroup RECO_DM01
NOTE: Found o/192.168.64.131/RECO_DM01_CD_03_dm01cel01 for disk RECO_DM01_CD_03_DM01CEL01
SUCCESS: disk RECO_DM01_CD_03_DM01CEL01 (3.3940753268) replaced in diskgroup RECO_DM01
NOTE: Found o/192.168.64.131/RECO_DM01_CD_04_dm01cel01 for disk RECO_DM01_CD_04_DM01CEL01
SUCCESS: disk RECO_DM01_CD_04_DM01CEL01 (4.3940753261) replaced in diskgroup RECO_DM01
NOTE: Found o/192.168.64.131/RECO_DM01_CD_05_dm01cel01 for disk RECO_DM01_CD_05_DM01CEL01
SUCCESS: disk RECO_DM01_CD_05_DM01CEL01 (5.3940753267) replaced in diskgroup RECO_DM01
NOTE: Found o/192.168.64.131/RECO_DM01_CD_06_dm01cel01 for disk RECO_DM01_CD_06_DM01CEL01
SUCCESS: disk RECO_DM01_CD_06_DM01CEL01 (6.3940753264) replaced in diskgroup RECO_DM01
NOTE: Found o/192.168.64.131/RECO_DM01_CD_07_dm01cel01 for disk RECO_DM01_CD_07_DM01CEL01
SUCCESS: disk RECO_DM01_CD_07_DM01CEL01 (7.3940753266) replaced in diskgroup RECO_DM01
NOTE: Found o/192.168.64.131/RECO_DM01_CD_08_dm01cel01 for disk RECO_DM01_CD_08_DM01CEL01
SUCCESS: disk RECO_DM01_CD_08_DM01CEL01 (8.3940753262) replaced in diskgroup RECO_DM01
NOTE: Found o/192.168.64.131/RECO_DM01_CD_09_dm01cel01 for disk RECO_DM01_CD_09_DM01CEL01
SUCCESS: disk RECO_DM01_CD_09_DM01CEL01 (9.3940753258) replaced in diskgroup RECO_DM01
NOTE: Found o/192.168.64.131/RECO_DM01_CD_10_dm01cel01 for disk RECO_DM01_CD_10_DM01CEL01
SUCCESS: disk RECO_DM01_CD_10_DM01CEL01 (10.3940753263) replaced in diskgroup RECO_DM01
NOTE: Found o/192.168.64.131/RECO_DM01_CD_11_dm01cel01 for disk RECO_DM01_CD_11_DM01CEL01
SUCCESS: disk RECO_DM01_CD_11_DM01CEL01 (11.3940753269) replaced in diskgroup RECO_DM01
NOTE: disk 0 (RECO_DM01_CD_00_DM01CEL01) in group 3 (RECO_DM01) is online for writes
NOTE: disk 1 (RECO_DM01_CD_01_DM01CEL01) in group 3 (RECO_DM01) is online for writes
NOTE: disk 2 (RECO_DM01_CD_02_DM01CEL01) in group 3 (RECO_DM01) is online for writes
NOTE: disk 3 (RECO_DM01_CD_03_DM01CEL01) in group 3 (RECO_DM01) is online for writes
NOTE: disk 4 (RECO_DM01_CD_04_DM01CEL01) in group 3 (RECO_DM01) is online for writes
NOTE: disk 5 (RECO_DM01_CD_05_DM01CEL01) in group 3 (RECO_DM01) is online for writes
NOTE: disk 6 (RECO_DM01_CD_06_DM01CEL01) in group 3 (RECO_DM01) is online for writes
NOTE: disk 7 (RECO_DM01_CD_07_DM01CEL01) in group 3 (RECO_DM01) is online for writes
NOTE: disk 8 (RECO_DM01_CD_08_DM01CEL01) in group 3 (RECO_DM01) is online for writes
NOTE: disk 9 (RECO_DM01_CD_09_DM01CEL01) in group 3 (RECO_DM01) is online for writes
NOTE: disk 10 (RECO_DM01_CD_10_DM01CEL01) in group 3 (RECO_DM01) is online for writes
NOTE: disk 11 (RECO_DM01_CD_11_DM01CEL01) in group 3 (RECO_DM01) is online for writes
NOTE: disk 0 (RECO_DM01_CD_00_DM01CEL01) in group 3 (RECO_DM01) is online for reads
NOTE: disk 1 (RECO_DM01_CD_01_DM01CEL01) in group 3 (RECO_DM01) is online for reads
NOTE: disk 2 (RECO_DM01_CD_02_DM01CEL01) in group 3 (RECO_DM01) is online for reads
NOTE: disk 3 (RECO_DM01_CD_03_DM01CEL01) in group 3 (RECO_DM01) is online for reads
NOTE: disk 4 (RECO_DM01_CD_04_DM01CEL01) in group 3 (RECO_DM01) is online for reads
NOTE: disk 5 (RECO_DM01_CD_05_DM01CEL01) in group 3 (RECO_DM01) is online for reads
NOTE: disk 6 (RECO_DM01_CD_06_DM01CEL01) in group 3 (RECO_DM01) is online for reads
NOTE: disk 7 (RECO_DM01_CD_07_DM01CEL01) in group 3 (RECO_DM01) is online for reads
NOTE: disk 8 (RECO_DM01_CD_08_DM01CEL01) in group 3 (RECO_DM01) is online for reads
NOTE: disk 9 (RECO_DM01_CD_09_DM01CEL01) in group 3 (RECO_DM01) is online for reads
NOTE: disk 10 (RECO_DM01_CD_10_DM01CEL01) in group 3 (RECO_DM01) is online for reads
NOTE: disk 11 (RECO_DM01_CD_11_DM01CEL01) in group 3 (RECO_DM01) is online for reads
[root@dm01cel01 trace]# pwd
/opt/oracle/cell11.2.3.1.1_LINUX.X64_120607/log/diag/asm/cell/dm01cel01/trace
CELL的ALERT日志如下：
Sun Sep 02 09:07:00 2012
[RS] Process /opt/oracle/cell11.2.3.1.1_LINUX.X64_120607/cellsrv/bin/cellrsmmt (pid: 10473) received clean shutdown signal from pid: 9334, uid: 0
Sun Sep 02 09:07:04 2012
[RS] Stopped Service MS with pid 10474
Sun Sep 02 09:07:04 2012
[RS] Process /opt/oracle/cell11.2.3.1.1_LINUX.X64_120607/cellsrv/bin/cellrsomt (pid: 12307) received clean shutdown signal from pid: 9334, uid: 0
Sun Sep 02 09:07:04 2012
Clean shutdown signal delivered to OSS<12308>
[RS] Stopped Service CELLSRV with pid 12308
Sun Sep 02 09:07:07 2012
[RS] Process /opt/oracle/cell11.2.3.1.1_LINUX.X64_120607/cellsrv/bin/cellrsbmt (pid: 9340) received clean shutdown signal from pid: 9334, uid: 0
[RS] Stopped Service RS_BACKUP
[RS] Stopped Service RS_MAIN
Sun Sep 02 09:07:08 2012
[RS] Process /opt/oracle/cell11.2.3.1.1_LINUX.X64_120607/cellsrv/bin/cellrsbkm (pid: 9342) received clean shutdown signal from pid: 9334, uid: 0
Sun Sep 02 09:07:09 2012
[RS] Process /opt/oracle/cell11.2.3.1.1_LINUX.X64_120607/cellsrv/bin/cellrssmt (pid: 9346) received clean shutdown signal from pid: 9334, uid: 0
Sun Sep 02 09:07:10 2012
[RS] Process /opt/oracle/cell11.2.3.1.1_LINUX.X64_120607/cellsrv/bin/cellrssrm (pid: 9334) received clean shutdown signal from pid: 9334, uid: 0
Sun Sep 02 09:13:40 2012
RS version=11.2.3.1.1,label=OSS_11.2.3.1.1_LINUX.X64_120607,Fri_Jun__8_12:49:44_PDT_2012
[RS] Started Service RS_MAIN with pid 18806
Sun Sep 02 09:13:40 2012
[RS] Started monitoring process /opt/oracle/cell11.2.3.1.1_LINUX.X64_120607/cellsrv/bin/cellrsbmt with pid 18812
Sun Sep 02 09:13:40 2012
[RS] Started monitoring process /opt/oracle/cell11.2.3.1.1_LINUX.X64_120607/cellsrv/bin/cellrsmmt with pid 18813
Sun Sep 02 09:13:40 2012
[RS] Started monitoring process /opt/oracle/cell11.2.3.1.1_LINUX.X64_120607/cellsrv/bin/cellrsomt with pid 18815
Sun Sep 02 09:13:40 2012
RSBK version=11.2.3.1.1,label=OSS_11.2.3.1.1_LINUX.X64_120607,Fri_Jun__8_12:49:44_PDT_2012
[RS] Started Service RS_BACKUP with pid 18814
Sun Sep 02 09:13:40 2012
[RS] Started monitoring process /opt/oracle/cell11.2.3.1.1_LINUX.X64_120607/cellsrv/bin/cellrssmt with pid 18844
Sun Sep 02 09:13:40 2012
Successfully setting event parameter -
Sun Sep 02 09:13:40 2012
Successfully setting event parameter -
CELLSRV process id=18817
CELLSRV cell host name=dm01cel01.acs.oracle.com
CELLSRV version=11.2.3.1.1,label=OSS_11.2.3.1.1_LINUX.X64_120607,Fri_Jun__8_12:49:44_PDT_2012
OS Hugepage status:
Total/free hugepages available=4001/155; hugepage size=2048KB
MS_ALERT HUGEPAGE CLEAR
Cache Allocation: Num 1MB hugepage buffers: 8000 Num 1MB non-hugepage buffers: 0
Cache Allocation: BufferSize: 512. Num buffers: 5000. Start Address: 2AACA2E00000
Cache Allocation: BufferSize: 2048. Num buffers: 5000. Start Address: 2AACA3072000
Cache Allocation: BufferSize: 4096. Num buffers: 5000. Start Address: 2AACA3A37000
Cache Allocation: BufferSize: 8192. Num buffers: 10000. Start Address: 2AACA4DC0000
Cache Allocation: BufferSize: 16384. Num buffers: 5000. Start Address: 2AACA9BE1000
Cache Allocation: BufferSize: 32768. Num buffers: 5000. Start Address: 2AACAEA02000
Cache Allocation: BufferSize: 65536. Num buffers: 5000. Start Address: 2AACB8643000
Cache Allocation: BufferSize: 10485760. Num buffers: 23. Start Address: 2AACCBEC4000
CELL communication is configured to use 1 interface(s):
192.168.64.131
IPC version: Oracle RDS/IP (generic)
IPC Vendor 1 Protocol 3
Version 4.1
CellDisk v0.6 name=CD_00_dm01cel01 status=NORMAL guid=d8227f04-4b2a-4a2b-b320-a73fe3211671 found on dev=/dev/sda3
GridDisk name=RECO_DM01_CD_00_dm01cel01 guid=27bc5e71-1662-419f-84ac-ac7936d051f9 (3428420588)
GridDisk name=DATA_DM01_CD_00_dm01cel01 guid=1215af58-90d9-4a74-a51b-3ddc6d54aede (3447961868)
CellDisk v0.6 name=FD_05_dm01cel01 status=NORMAL guid=a3718f03-8112-4643-932e-0837ea9d445f found on dev=/dev/sdaa
CellDisk v0.6 name=FD_06_dm01cel01 status=NORMAL guid=309f226c-8255-4c71-9991-ad7eab8e934b found on dev=/dev/sdab
CellDisk v0.6 name=FD_07_dm01cel01 status=NORMAL guid=af4b7aa9-486a-4458-a4e7-0ecfb17536ab found on dev=/dev/sdac
CellDisk v0.6 name=FD_13_dm01cel01 status=NORMAL guid=7bad74e5-d125-4702-882e-a11db0154588 found on dev=/dev/sdw
CellDisk v0.6 name=FD_14_dm01cel01 status=NORMAL guid=7588df10-a2e2-497d-967f-0e1ef5d07d74 found on dev=/dev/sdx
CellDisk v0.6 name=FD_15_dm01cel01 status=NORMAL guid=d70c7598-82e8-496e-ae8a-8f7c888374b8 found on dev=/dev/sdy
CellDisk v0.6 name=FD_04_dm01cel01 status=NORMAL guid=95662b6a-2eeb-40e4-81ca-d880b364fbac found on dev=/dev/sdz
FlashCache: allowing client IOs
Smart Flash Logging disabled due to lack of active Flash Log Stores
Cellsrv Incarnation is set: 2
CELLSRV Server startup complete
Sun Sep 2 09:13:43 2012
[RS] Started Service CELLSRV with pid 18817
[RS] Started Service MS with pid 18816
Sun Sep 02 09:13:47 2012
Heartbeat with diskmon started on dm01db02.acs.oracle.com
Heartbeat with diskmon started on dm01db01.acs.oracle.com
Sun Sep 02 09:13:47 2012
I/O Resource Manager enabled
Sun Sep 02 09:13:48 2012
Caching enabled on FlashCache dm01cel01_FLASHCACHE (1818411676), size=22GB, cdisk=FD_13_dm01cel01
Caching enabled on FlashCache dm01cel01_FLASHCACHE (3896713996), size=22GB, cdisk=FD_05_dm01cel01
Caching enabled on FlashCache dm01cel01_FLASHCACHE (4258309268), size=22GB, cdisk=FD_11_dm01cel01
Caching enabled on FlashCache dm01cel01_FLASHCACHE (2295236356), size=22GB, cdisk=FD_09_dm01cel01
Caching enabled on FlashCache dm01cel01_FLASHCACHE (2066597948), size=22GB, cdisk=FD_06_dm01cel01
Caching enabled on FlashCache dm01cel01_FLASHCACHE (3538503772), size=22GB, cdisk=FD_15_dm01cel01
Caching enabled on FlashCache dm01cel01_FLASHCACHE (1832946924), size=22GB, cdisk=FD_10_dm01cel01
Caching enabled on FlashCache dm01cel01_FLASHCACHE (542355212), size=22GB, cdisk=FD_00_dm01cel01
Caching enabled on FlashCache dm01cel01_FLASHCACHE (1736192156), size=22GB, cdisk=FD_04_dm01cel01
Caching enabled on FlashCache dm01cel01_FLASHCACHE (2921083820), size=22GB, cdisk=FD_14_dm01cel01
Caching enabled on FlashCache dm01cel01_FLASHCACHE (3957806660), size=22GB, cdisk=FD_12_dm01cel01
Caching enabled on FlashCache dm01cel01_FLASHCACHE (767858284), size=22GB, cdisk=FD_01_dm01cel01
Sun Sep 02 09:14:00 2012
Info: Assigning flash CD FD_00_dm01cel01 to group#1
Sun Sep 02 09:14:00 2012
Info: Assigning flash CD FD_01_dm01cel01 to group#1
Sun Sep 02 09:14:00 2012
Info: Assigning flash CD FD_02_dm01cel01 to group#1
Sun Sep 02 09:14:00 2012
Info: Assigning flash CD FD_03_dm01cel01 to group#1
Sun Sep 02 09:14:00 2012
Info: Assigning flash CD FD_04_dm01cel01 to group#2
Sun Sep 02 09:14:00 2012
Info: Assigning flash CD FD_05_dm01cel01 to group#2
Sun Sep 02 09:14:00 2012
Info: Assigning flash CD FD_06_dm01cel01 to group#2
Sun Sep 02 09:14:00 2012
Info: Assigning flash CD FD_07_dm01cel01 to group#2
Sun Sep 02 09:14:00 2012
Info: Assigning flash CD FD_08_dm01cel01 to group#4
Sun Sep 02 09:14:00 2012
