[Urgent] ORA-15042: ASM disk “76” is missing

2013/02/05 by Leave a Comment

If you cannot recover the data by yourself, ask Parnassusdata, the professional ORACLE database recovery team for help.

Parnassusdata Software Database Recovery Team

Service Hotline:  +86 13764045638 E-mail: service@parnassusdata.com

 

One customer has a ASM problem about ORA-15042.
O/S: Linux X86 64bit 2.6.18-194.el5
DB Version : 10.2.0.5

Although we can access the ASM header using kfed & dd, the asm instance cannot read these devices.
For example, the ASM instance can read the 75th disk, but cannot read the 76th disk.

Do you have this experience?

# Environment
LGEDGDMS01:/engn001/orasvc01/product/10.2.0] uname -a
Linux LGEDGDMS01 2.6.18-194.el5 #1 SMP Tue Mar 16 21:52:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

# Error
SQL> startup
ASM instance started

Total System Global Area  130023424 bytes
Fixed Size                  2094544 bytes
Variable Size             102763056 bytes
ASM Cache                  25165824 bytes
ORA-15042: ASM disk “23” is missing
ORA-15042: ASM disk “22” is missing
ORA-15040: diskgroup is incomplete
ORA-15066: offlining disk “” may result in a data loss
ORA-15042: ASM disk “88” is missing
…
ORA-15042: ASM disk “77” is missing
ORA-15042: ASM disk “76” is missing   ==> 76 th device
ORA-15040: diskgroup is incomplete
ORA-15066: offlining disk “” may result in a data loss
ORA-15042: ASM disk “88” is missing
ORA-15042: ASM disk “87” is missing
…
ORA-15042: ASM disk “81” is missing

SQL> show parameter asm_diskstring
NAME                                 TYPE                              VALUE
———————————— ——————————— ——————————
asm_diskstring                       string                    /dev/mapper/mpath_asm*

# v$asm_disks results.
select name, group_number,disk_number, path, state, header_status from v$asm_disk order by disk_number
/
NAME       GROUP_NUMBER DISK_NUMBER PATH                                     STATE                    HEADER_STATUS
———- ———— ———– —————————————- ———————— ————————————
0          66 /dev/mapper/mpath_asm129p1               NORMAL                   MEMBER
0          67 /dev/mapper/mpath_asm130p1               NORMAL                   MEMBER
0          68 /dev/mapper/mpath_asm131p1               NORMAL                   MEMBER
0          69 /dev/mapper/mpath_asm132p1               NORMAL                   MEMBER
0          70 /dev/mapper/mpath_asm133p1               NORMAL                   MEMBER
0          71 /dev/mapper/mpath_asm134p1               NORMAL                   MEMBER
0          72 /dev/mapper/mpath_asm135p1               NORMAL                   MEMBER
0          73 /dev/mapper/mpath_asm136p1               NORMAL                   MEMBER
0          74 /dev/mapper/mpath_asm137p1               NORMAL                   MEMBER
0          75 /dev/mapper/mpath_asm138p1               NORMAL                   MEMBER
0          89 /dev/mapper/mpath_asm063p1               NORMAL                   MEMBER   ==> Cannot see the 76 th device
0          90 /dev/mapper/mpath_asm064p1               NORMAL                   MEMBER
0          91 /dev/mapper/mpath_asm065p1               NORMAL                   MEMBER
0          92 /dev/mapper/mpath_asm066p1               NORMAL                   MEMBER
# Permission – OK
* 75th asm file (Good Device)
LGEDGDMS01:/engn001/orasvc01/product/10.2.0] ls -al /dev/mapper/mpath_asm138p1
brw-rw—- 1 orasvc01 dba 253, 248 Jan 30 17:06 /dev/mapper/mpath_asm138p1

* 76th the asm file (Cannot read this device)
LGEDGDMS01:/engn001/orasvc01/product/10.2.0] ls -al /dev/mapper/mpath_asm175
brw-rw—- 1 orasvc01 dba 253, 197 Jan 30 17:06 /dev/mapper/mpath_asm175

# kfed result – OK
* 75th asm file (Good Device)
+ /engn001/orasvc01/product/10.2.0/bin/kfed read /dev/mapper/mpath_asm138p1    
kfbh.endian:                          1 ; 0x000: 0x01                          
kfbh.hard:                          130 ; 0x001: 0x82                          
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD               
kfbh.datfmt:                          1 ; 0x003: 0x01                          
kfbh.block.blk:                       0 ; 0x004: T=0 NUMB=0x0                      
kfbh.block.obj:              2147483723 ; 0x008: TYPE=0x8 NUMB=0x4b
kfbh.check:                  2774762225 ; 0x00c: 0xa56382f1                    
kfbh.fcn.base:                        0 ; 0x010: 0x00000000                    
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000                    
kfbh.spare1:                          0 ; 0x018: 0x00000000                    
kfbh.spare2:                          0 ; 0x01c: 0x00000000                    
kfdhdb.driver.provstr:ORCLDISKASMDISK138 ; 0x000: length=18                     
kfdhdb.driver.reserved[0]:   1145918273 ; 0x008: 0x444d5341                    
kfdhdb.driver.reserved[1]:    827020105 ; 0x00c: 0x314b5349                    
kfdhdb.driver.reserved[2]:        14387 ; 0x010: 0x00003833                    
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000                    
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000                    
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000                    
kfdhdb.compat:                168820736 ; 0x020: 0x0a100000                    
kfdhdb.dsknum:                       75 ; 0x024: 0x004b                       ==> The 75th device
kfdhdb.grptyp:                        2 ; 0x026: KFDGTP_NORMAL                 
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER                 
kfdhdb.dskname:           DGDATA01_0075 ; 0x028: length=13                     
kfdhdb.grpname:                DGDATA01 ; 0x048: length=8                      
kfdhdb.fgname:            DGDATA01_0075 ; 0x068: length=13                     
kfdhdb.capname:                         ; 0x088: length=0                      
kfdhdb.crestmp.hi:             32973218 ; 0x0a8: HOUR=0x2 DAYS=0xd MNTH=0x8 YEAR=0x7dc
kfdhdb.crestmp.lo:           1898247168 ; 0x0ac: USEC=0x0 MSEC=0x13d SECS=0x12 MINS=0x1c
kfdhdb.mntstmp.hi:             32973219 ; 0x0b0: HOUR=0x3 DAYS=0xd MNTH=0x8 YEAR=0x7dc
kfdhdb.mntstmp.lo:           1163180032 ; 0x0b4: USEC=0x0 MSEC=0x12e SECS=0x15 MINS=0x11
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200                        
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000                        
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000                    
kfdhdb.mfact:                    113792 ; 0x0c0: 0x0001bc80                    
kfdhdb.dsksize:                   13892 ; 0x0c4: 0x00003644                    

* 76th the asm file (Cannot read this device)
+ /engn001/orasvc01/product/10.2.0/bin/kfed read /dev/mapper/mpath_asm175      
kfbh.endian:                          1 ; 0x000: 0x01                          
kfbh.hard:                          130 ; 0x001: 0x82                          
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD               
kfbh.datfmt:                          1 ; 0x003: 0x01                          
kfbh.block.blk:                       0 ; 0x004: T=0 NUMB=0x0                      
kfbh.block.obj:              2147483724 ; 0x008: TYPE=0x8 NUMB=0x4c
kfbh.check:                  2433973412 ; 0x00c: 0x91137ca4                    
kfbh.fcn.base:                        0 ; 0x010: 0x00000000                    
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000                    
kfbh.spare1:                          0 ; 0x018: 0x00000000                    
kfbh.spare2:                          0 ; 0x01c: 0x00000000                    
kfdhdb.driver.provstr:ORCLDISKASMDISK175 ; 0x000: length=18                     
kfdhdb.driver.reserved[0]:   1145918273 ; 0x008: 0x444d5341                    
kfdhdb.driver.reserved[1]:    827020105 ; 0x00c: 0x314b5349                    
kfdhdb.driver.reserved[2]:        13623 ; 0x010: 0x00003537                    
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000                    
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000                    
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000                    
kfdhdb.compat:                168820736 ; 0x020: 0x0a100000                    
kfdhdb.dsknum:                       76 ; 0x024: 0x004c            ==> the 76th device, ASM instance cannot read this device.
kfdhdb.grptyp:                        2 ; 0x026: KFDGTP_NORMAL                 
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER                 
kfdhdb.dskname:           DGDATA01_0076 ; 0x028: length=13                     
kfdhdb.grpname:                DGDATA01 ; 0x048: length=8                      
kfdhdb.fgname:            DGDATA01_0076 ; 0x068: length=13                     
kfdhdb.capname:                         ; 0x088: length=0                      
kfdhdb.crestmp.hi:             32982981 ; 0x0a8: HOUR=0x5 DAYS=0x1e MNTH=0x1 YEAR=0x7dd
kfdhdb.crestmp.lo:            366295040 ; 0x0ac: USEC=0x0 MSEC=0x14e SECS=0x1d MINS=0x5
kfdhdb.mntstmp.hi:             32982981 ; 0x0b0: HOUR=0x5 DAYS=0x1e MNTH=0x1 YEAR=0x7dd
kfdhdb.mntstmp.lo:            366307328 ; 0x0b4: USEC=0x0 MSEC=0x15a SECS=0x1d MINS=0x5
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200                        
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000                        
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000                    
kfdhdb.mfact:                    113792 ; 0x0c0: 0x0001bc80                    
kfdhdb.dsksize:                   55572 ; 0x0c4: 0x0000d914                    

# Check the dd results – OK
* the 75th asm device (Good)
dd if=/dev/mapper/mpath_asm138p1 bs=4096|od -tx1z|more                    
                              
0000000 01 82 01 01 00 00 00 00 4b 00 00 80 f1 82 63 a5  >……..K…..c.<
0000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >…………….<
0000040 4f 52 43 4c 44 49 53 4b 41 53 4d 44 49 53 4b 31  >ORCLDISKASMDISK1<
0000060 33 38 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >38…………..<
0000100 00 00 10 0a 4b 00 02 03 44 47 44 41 54 41 30 31  >….K…DGDATA01<
0000120 5f 30 30 37 35 00 00 00 00 00 00 00 00 00 00 00  >_0075………..<
0000140 00 00 00 00 00 00 00 00 44 47 44 41 54 41 30 31  >……..DGDATA01<
0000160 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >…………….<
0000200 00 00 00 00 00 00 00 00 44 47 44 41 54 41 30 31  >……..DGDATA01<
0000220 5f 30 30 37 35 00 00 00 00 00 00 00 00 00 00 00  >_0075………..<
0000240 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >…………….<
*                                                           
0000300 00 00 00 00 00 00 00 00 a2 21 f7 01 00 f4 24 71  >………!….$q<
0000320 a3 21 f7 01 00 b8 54 45 00 02 00 10 00 00 10 00  >.!….TE……..<
0000340 80 bc 01 00 44 36 00 00 02 00 00 00 01 00 00 00  >….D6……….<
0000360 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >…………….<
0000400 00 00 10 0a 14 cd f6 01 00 2c 95 00 00 00 00 00  >………,……<
0000420 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >…………….<
*                                                           
0010000 01 82 02 01 01 00 00 00 4b 00 00 80 de 63 17 81  >……..K….c..<
0010020 af e0 35 00 00 00 00 00 00 00 00 00 00 00 00 00  >..5………….<
0010040 00 00 00 00 fe 00 20 00 c0 01 00 01 c0 01 00 01  >…… ………<
0010060 c0 01 00 01 c0 01 00 01 c0 01 00 01 c0 01 01 01  >…………….<
0010100 c0 01 00 01 c0 01 00 01 c0 01 00 01 c0 01 00 01  >…………….<
0010120 c0 01 00 01 c0 01 01 01 c0 01 01 01 c0 01 01 01  >…………….<
0010140 c0 01 01 01 c0 01 01 01 c0 01 01 01 c0 01 01 01  >…………….<
*                                                           
0010240 c0 01 01 01 04 00 01 01 00 00 00 00 00 00 00 00  >…………….<
0010260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >…………….<
*                                                           
0020000 01 82 03 01 02 00 00 00 4b 00 00 80 ce 10 bd 80  >……..K…….<
0020020 df ad 1e 00 00 00 00 00 00 00 00 00 00 00 00 00  >…………….<
0020040 00 00 00 00 c0 01 00 00 08 00 08 00 00 00 c0 01  >…………….<
0020060 10 00 10 00 00 00 00 00 18 00 18 00 00 00 00 00  >…………….<
0020100 20 00 20 00 00 00 00 00 00 00 00 00 00 00 80 00  > . ………….<
0020120 00 00 00 00 00 00 80 00 d9 0b 00 00 18 01 80 00  >…………….<

* 76th device (Read Failure)                                                    
dd if=/dev/mapper/mpath_asm175 bs=4096|od -tx1z|more                    
                              
0000000 01 82 01 01 00 00 00 00 4c 00 00 80 a4 7c 13 91  >……..L….|..<
0000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >…………….<
0000040 4f 52 43 4c 44 49 53 4b 41 53 4d 44 49 53 4b 31  >ORCLDISKASMDISK1<
0000060 37 35 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >75…………..<
0000100 00 00 10 0a 4c 00 02 03 44 47 44 41 54 41 30 31  >….L…DGDATA01<
0000120 5f 30 30 37 36 00 00 00 00 00 00 00 00 00 00 00  >_0076………..<
0000140 00 00 00 00 00 00 00 00 44 47 44 41 54 41 30 31  >……..DGDATA01<
0000160 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >…………….<
0000200 00 00 00 00 00 00 00 00 44 47 44 41 54 41 30 31  >……..DGDATA01<
0000220 5f 30 30 37 36 00 00 00 00 00 00 00 00 00 00 00  >_0076………..<
0000240 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >…………….<
*                                                           
0000300 00 00 00 00 00 00 00 00 c5 47 f7 01 00 38 d5 15  >………G…8..<
0000320 c5 47 f7 01 00 68 d5 15 00 02 00 10 00 00 10 00  >.G…h……….<
0000340 80 bc 01 00 14 d9 00 00 02 00 00 00 01 00 00 00  >…………….<
0000360 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >…………….<
0000400 00 00 10 0a 14 cd f6 01 00 2c 95 00 00 00 00 00  >………,……<
0000420 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >…………….<
*                                                           
0000660 00 00 00 00 00 00 00 00 02 ec 44 ff 00 00 00 00  >……….D…..<
0000700 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >…………….<
*                                                           
0010000 01 82 02 01 01 00 00 00 4c 00 00 80 c3 62 4b 80  >……..L….bK.<
0010020 65 e0 35 00 00 00 00 00 00 00 00 00 00 00 00 00  >e.5………….<
0010040 00 00 00 00 fe 00 7d 00 c0 01 00 01 c0 01 00 01  >……}………<
0010060 c0 01 00 01 c0 01 00 01 c0 01 00 01 c0 01 00 01  >…………….<
*                                                           
0010460 c0 01 01 01 c0 01 01 01 c0 01 01 01 c0 01 01 01  >…………….<
*                                                           
0011020 c0 01 01 01 c0 01 01 01 14 00 01 01 00 00 00 00  >…………….<
0011040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >…………….<
*                                                           
0020000 01 82 03 01 02 00 00 00 4c 00 00 80 f0 45 ff 80  >……..L….E..<
0020020 9a d8 1c 00 00 00 00 00 00 00 00 00 00 00 00 00  >…………….<
0020040 00 00 00 00 c0 01 00 00 08 00 08 00 00 00 c0 01  >…………….<
0020060 10 00 10 00 00 00 00 00 18 00 18 00 00 00 00 00  >…………….<

 

kfod status=true asm_diskstring=’/dev/mapper/mpath_asm*’ disk=ALL
——————————————————————————–
Disk          Size Header    Path
================================================================================
1:      13893 Mb CANDIDATE /dev/mapper/mpath_asm001
2:      13892 Mb MEMBER    /dev/mapper/mpath_asm001p1
3:      13893 Mb CANDIDATE /dev/mapper/mpath_asm002
4:      13892 Mb MEMBER    /dev/mapper/mpath_asm002p1
5:      13893 Mb CANDIDATE /dev/mapper/mpath_asm003
6:      13892 Mb MEMBER    /dev/mapper/mpath_asm003p1
7:      13893 Mb CANDIDATE /dev/mapper/mpath_asm004
…
274:      13892 Mb MEMBER    /dev/mapper/mpath_asm137p1
275:      13893 Mb CANDIDATE /dev/mapper/mpath_asm138
276:      13892 Mb MEMBER    /dev/mapper/mpath_asm138p1   ==> MEMBER
277:      13893 Mb CANDIDATE /dev/mapper/mpath_asm139
278:      13892 Mb MEMBER    /dev/mapper/mpath_asm139p1
…
343:      62400 Mb CANDIDATE /dev/mapper/mpath_asm172
344:      62393 Mb MEMBER    /dev/mapper/mpath_asm172p1
345:      62400 Mb CANDIDATE /dev/mapper/mpath_asm173
346:      62393 Mb MEMBER    /dev/mapper/mpath_asm173p1
347:      62400 Mb CANDIDATE /dev/mapper/mpath_asm174
348:      62393 Mb MEMBER    /dev/mapper/mpath_asm174p1
349:      55572 Mb CANDIDATE /dev/mapper/mpath_asm175    ==> CANDIDATE~!
350:      55572 Mb CANDIDATE /dev/mapper/mpath_asm176
351:      55572 Mb CANDIDATE /dev/mapper/mpath_asm177
352:      55572 Mb CANDIDATE /dev/mapper/mpath_asm178
353:      55572 Mb CANDIDATE /dev/mapper/mpath_asm179

 

I could find out one. The added partition don’t have any partition tables, but the original asm disks do it.
Due to storage engineer fault, I suppose that the KFED results is “MEMBER” and the KFOD result is “CANDICATE’ status.
I’ll replace it to the additional disks with partition tables.
If it successful, I will reply it.

# Reference
(Doc ID 580153.1) How To Setup ASM on Linux Using ASMLIB Disks, Raw Devices or Block Devices?
In order to use a disk (e.g. SAN) in Automatic Storage Management, the disk must have a partition table.

Filed Under: Oracle, Oracle数据恢复、数据库恢复、灾难恢复

Comment

*

沪ICP备14014813号-2

沪公网安备 31010802001379号