针对11.2 RAC丢失OCR和Votedisk所在ASM Diskgroup的恢复手段

2012/08/09 by 9 Comments

之前有同学在我的Oracle Allstarts群里讨论关于丢失包含ocr和votedisk的ASM diskgroup导致11gR2 RAC cluster无法正常启动的问题，最早我在《在11gR2 RAC中修改ASM DISK Path磁盘路径》一文中介绍了，如何在不启动CRS的情况下启动11.2中的ASM实例并实施操作， 这里也需要用到同样的命令”crsctl start crs -excl -nocrs “;

 

如果自己搞不定可以找诗檀软件专业ORACLE数据库修复团队成员帮您恢复!

诗檀软件专业数据库修复团队

服务热线 ： 13764045638   QQ号:47079569    邮箱：service@parnassusdata.com

 

注意以下的恢复手段，针对ASM中单独的ocr或者单独的votedisk丢失也有效，因为11.2中普遍把ocr和votedisk存放在ASM中，而ASM的启动又依赖于ocr和votedisk，所以在丢失ocr或votedisk仍一一者都会导致cluter无法正常启动；这里我们仅仅讨论如何让CRS正常启动，如果丢失的diskgroup中还存放有数据库的话，数据的恢复不属于本篇文章的讨论范畴。

 

前提：恢复的前提是你仍有和故障前一样多的ASM LUN DISK，且你有OCR的自动备份，注意默认情况下每4个小时会自动备份一次，只要你没有删除$GI_HOME，一般都会有备份可用；不要求有votedisk备份

 

恢复场景： 利用dd命令清空ocr和votedisk所在diskgroup header，模拟diskgroup corruption:

 

 

1. 检查votedisk和 ocr备份

[root@vrh1 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   a853d6204bbc4feabfd8c73d4c3b3001 (/dev/asm-diskh) [SYSTEMDG]
 2. ONLINE   a5b37704c3574f0fbf21d1d9f58c4a6b (/dev/asm-diskg) [SYSTEMDG]
 3. ONLINE   36e5c51ff0294fc3bf2a042266650331 (/dev/asm-diski) [SYSTEMDG]
 4. ONLINE   af337d1512824fe4bf6ad45283517aaa (/dev/asm-diskj) [SYSTEMDG]
 5. ONLINE   3c4a349e2e304ff6bf64b2b1c9d9cf5d (/dev/asm-diskk) [SYSTEMDG]
Located 5 voting disk(s).

su - grid

[grid@vrh1 ~]$ ocrconfig -showbackup
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy

vrh1     2012/08/09 01:59:56     /g01/11.2.0/maclean/grid/cdata/vrh-cluster/backup00.ocr

vrh1     2012/08/08 21:59:56     /g01/11.2.0/maclean/grid/cdata/vrh-cluster/backup01.ocr

vrh1     2012/08/08 17:59:55     /g01/11.2.0/maclean/grid/cdata/vrh-cluster/backup02.ocr

vrh1     2012/08/08 05:59:54     /g01/11.2.0/grid/cdata/vrh-cluster/day.ocr

vrh1     2012/08/08 05:59:54     /g01/11.2.0/grid/cdata/vrh-cluster/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available

2. 彻底关闭所有节点上的clusterware ,OHASD 

crsctl stop has -f

3. GetAsmDH.sh ==> GetAsmDH.sh是ASM disk header的备份脚本 
请养成良好的习惯，做危险操作前备份asm header

[grid@vrh1 ~]$ ./GetAsmDH.sh 

############################################
 1) Collecting Information About the Disks:
############################################

SQL*Plus: Release 11.2.0.3.0 Production on Thu Aug 9 03:28:13 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

SQL> Connected.
SQL> SQL> SQL> SQL> SQL> SQL> SQL> 
           1           0 /dev/asm-diske
           1           1 /dev/asm-diskd
           2           0 /dev/asm-diskb
           2           1 /dev/asm-diskc
           2           2 /dev/asm-diskf
           3           0 /dev/asm-diskh
           3           1 /dev/asm-diskg
           3           2 /dev/asm-diski
           3           3 /dev/asm-diskj
           3           4 /dev/asm-diskk
SQL> SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

-rw-r--r-- 1 grid oinstall 1048 Aug  9 03:28 /tmp/HC/asmdisks.lst

############################################
 2) Generating asm_diskh.sh script.
############################################

-rwx------ 1 grid oinstall 666 Aug  9 03:28 /tmp/HC/asm_diskh.sh

############################################
 3) Executing  asm_diskh.sh script to 
    generate dd  dumps.
############################################

-rw-r--r-- 1 grid oinstall 1048576 Aug  9 03:28 /tmp/HC/dsk_1_0.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug  9 03:28 /tmp/HC/dsk_1_1.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug  9 03:28 /tmp/HC/dsk_2_0.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug  9 03:28 /tmp/HC/dsk_2_1.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug  9 03:28 /tmp/HC/dsk_2_2.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug  9 03:28 /tmp/HC/dsk_3_0.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug  9 03:28 /tmp/HC/dsk_3_1.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug  9 03:28 /tmp/HC/dsk_3_2.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug  9 03:28 /tmp/HC/dsk_3_3.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug  9 03:28 /tmp/HC/dsk_3_4.dd

############################################
 4) Compressing dd dumps in the next format:
    (asm_dd_header_all_.tar)
############################################

/tmp/HC/dsk_1_0.dd
/tmp/HC/dsk_1_1.dd
/tmp/HC/dsk_2_0.dd
/tmp/HC/dsk_2_1.dd
/tmp/HC/dsk_2_2.dd
/tmp/HC/dsk_3_0.dd
/tmp/HC/dsk_3_1.dd
/tmp/HC/dsk_3_2.dd
/tmp/HC/dsk_3_3.dd
/tmp/HC/dsk_3_4.dd
./GetAsmDH.sh: line 81: compress: command not found
ls: /tmp/HC/*.Z: No such file or directory
[grid@vrh1 ~]$

4. 使用dd 命令 破坏ocr和votedisk所在diskgroup

[root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00423853 seconds, 247 MB/s

[root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskg bs=1024k count=1 
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0045179 seconds, 232 MB/s

[root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diski bs=1024k count=1 
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00469976 seconds, 223 MB/s

[root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskj bs=1024k count=1 
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00344262 seconds, 305 MB/s
[root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskk bs=1024k count=1 
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0053518 seconds, 196 MB/s

5. 在一个节点上尝试重新启动HAS

[root@vrh1 ~]# crsctl start has
CRS-4123: Oracle High Availability Services has been started.

 

 

 

但是因为ocr和votedisk所在diskgroup丢失，所以CSS将无法正常启动，如以下日志所示：

 

 

alertvrh1.log 

[cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log
2012-08-09 03:35:41.207
[cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log
2012-08-09 03:35:56.240
[cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log
2012-08-09 03:36:11.284
[cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log
2012-08-09 03:36:26.305
[cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log
2012-08-09 03:36:41.328

ocssd.log
2012-08-09 03:40:26.662: [    CSSD][1078700352]clssnmReadDiscoveryProfile: voting file discovery string(/dev/asm*)
2012-08-09 03:40:26.662: [    CSSD][1078700352]clssnmvDDiscThread: using discovery string /dev/asm* for initial discovery
2012-08-09 03:40:26.662: [   SKGFD][1078700352]Discovery with str:/dev/asm*:

2012-08-09 03:40:26.662: [   SKGFD][1078700352]UFS discovery with :/dev/asm*:

2012-08-09 03:40:26.665: [   SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskf:

2012-08-09 03:40:26.665: [   SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskb:

2012-08-09 03:40:26.665: [   SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskj:

2012-08-09 03:40:26.665: [   SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskh:

2012-08-09 03:40:26.665: [   SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskc:

2012-08-09 03:40:26.665: [   SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskd:

2012-08-09 03:40:26.665: [   SKGFD][1078700352]Fetching UFS disk :/dev/asm-diske:

2012-08-09 03:40:26.665: [   SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskg:

2012-08-09 03:40:26.665: [   SKGFD][1078700352]Fetching UFS disk :/dev/asm-diski:

2012-08-09 03:40:26.665: [   SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskk:

2012-08-09 03:40:26.665: [   SKGFD][1078700352]OSS discovery with :/dev/asm*:

2012-08-09 03:40:26.665: [   SKGFD][1078700352]Handle 0xdf22a0 from lib :UFS:: for disk :/dev/asm-diskf:

2012-08-09 03:40:26.665: [   SKGFD][1078700352]Handle 0xf412a0 from lib :UFS:: for disk :/dev/asm-diskb:

2012-08-09 03:40:26.666: [   SKGFD][1078700352]Handle 0xf3a680 from lib :UFS:: for disk :/dev/asm-diskj:

2012-08-09 03:40:26.666: [   SKGFD][1078700352]Handle 0xf93da0 from lib :UFS:: for disk :/dev/asm-diskh:

2012-08-09 03:40:26.667: [    CSSD][1078700352]clssnmvDiskVerify: Successful discovery of 0 disks
2012-08-09 03:40:26.667: [    CSSD][1078700352]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2012-08-09 03:40:26.667: [    CSSD][1078700352]clssnmvFindInitialConfigs: No voting files found
2012-08-09 03:40:26.667: [    CSSD][1078700352](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds

 

 

 

正式的恢复ocr和votedisk所在diskgroup的步骤如下：

 

 

1. 以-excl -nocrs 方式启动cluster，这将可以启动ASM实例 但不启动CRS

 [root@vrh1 vrh1]# crsctl start crs -excl -nocrs 

CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'vrh1'
CRS-2676: Start of 'ora.mdnsd' on 'vrh1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'vrh1'
CRS-2676: Start of 'ora.gpnpd' on 'vrh1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'vrh1'
CRS-2672: Attempting to start 'ora.gipcd' on 'vrh1'
CRS-2676: Start of 'ora.cssdmonitor' on 'vrh1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'vrh1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'vrh1'
CRS-2672: Attempting to start 'ora.diskmon' on 'vrh1'
CRS-2676: Start of 'ora.diskmon' on 'vrh1' succeeded
CRS-2676: Start of 'ora.cssd' on 'vrh1' succeeded
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'vrh1'
CRS-2672: Attempting to start 'ora.ctssd' on 'vrh1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'vrh1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'vrh1'
CRS-2676: Start of 'ora.ctssd' on 'vrh1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'vrh1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'vrh1'
CRS-2676: Start of 'ora.asm' on 'vrh1' succeeded

 

 

 

2.重建原ocr和votedisk所在diskgroup,注意compatible.asm必须是11.2:

 

 

[root@vrh1 vrh1]# su - grid
[grid@vrh1 ~]$ sqlplus  / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Thu Aug 9 04:16:58 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> create diskgroup systemdg high redundancy disk '/dev/asm-diskh','/dev/asm-diskg','/dev/asm-diski','/dev/asm-diskj','/dev/asm-diskk'
 ATTRIBUTE 'compatible.rdbms' = '11.2', 'compatible.asm' = '11.2';

 

 

 

3.从ocr backup中恢复ocr并做ocrcheck检验:

 

 

[root@vrh1 ~]# ocrconfig -restore /g01/11.2.0/grid/cdata/vrh-cluster/backup00.ocr

[root@vrh1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3180
         Available space (kbytes) :     258940
         ID                       : 1238458014
         Device/File Name         :  +systemdg
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

 

 

 

4. 准备恢复votedisk ，可能会遇到下面的错误:

 

[grid@vrh1 ~]$ crsctl replace votedisk  +SYSTEMDG
CRS-4602: Failed 27 to add voting file 2e4e0fe285924f86bf5473d00dcc0388.
CRS-4602: Failed 27 to add voting file 4fa54bb0cc5c4fafbf1a9be5479bf389.
CRS-4602: Failed 27 to add voting file a109ead9ea4e4f28bfe233188623616a.
CRS-4602: Failed 27 to add voting file 042c9fbd71b54f5abfcd3ab3408f3cf3.
CRS-4602: Failed 27 to add voting file 7b5a8cd24f954fafbf835ad78615763f.
Failed to replace voting disk group with +SYSTEMDG.
CRS-4000: Command Replace failed, or completed with errors.

 

 

需要重新配置一下ASM的参数，并重启ASM:

 

 

SQL> alter system set asm_diskstring='/dev/asm*';

System altered.

SQL> create spfile from memory;

File created.

SQL>  startup force mount;
ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance
ASM instance started

Total System Global Area  283930624 bytes
Fixed Size                  2227664 bytes
Variable Size             256537136 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted

SQL> show parameter spfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      /g01/11.2.0/grid/dbs/spfile+AS
                                                 M1.ora

[grid@vrh1 trace]$  crsctl replace votedisk  +SYSTEMDG
CRS-4256: Updating the profile
Successful addition of voting disk 85edc0e82d274f78bfc58cdc73b8c68a.
Successful addition of voting disk 201ffffc8ba44faabfe2efec2aa75840.
Successful addition of voting disk 6f2a25c589964faabf6980f7c5f621ce.
Successful addition of voting disk 93eb315648454f25bf3717df1a2c73d5.
Successful addition of voting disk 3737240678964f88bfbfbd31d8b3829f.
Successfully replaced voting disk group with +SYSTEMDG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced

 

 

 

 

5. 重启has服务，检验cluster是否正常：

 

 

[root@vrh1 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

[root@vrh1 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   85edc0e82d274f78bfc58cdc73b8c68a (/dev/asm-diskh) [SYSTEMDG]
 2. ONLINE   201ffffc8ba44faabfe2efec2aa75840 (/dev/asm-diskg) [SYSTEMDG]
 3. ONLINE   6f2a25c589964faabf6980f7c5f621ce (/dev/asm-diski) [SYSTEMDG]
 4. ONLINE   93eb315648454f25bf3717df1a2c73d5 (/dev/asm-diskj) [SYSTEMDG]
 5. ONLINE   3737240678964f88bfbfbd31d8b3829f (/dev/asm-diskk) [SYSTEMDG]
Located 5 voting disk(s).

[root@vrh1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.BACKUPDG.dg
               ONLINE  ONLINE       vrh1                                         
ora.DATA.dg
               ONLINE  ONLINE       vrh1                                         
ora.LISTENER.lsnr
               ONLINE  ONLINE       vrh1                                         
ora.LSN_MACLEAN.lsnr
               ONLINE  ONLINE       vrh1                                         
ora.SYSTEMDG.dg
               ONLINE  ONLINE       vrh1                                         
ora.asm
               ONLINE  ONLINE       vrh1                     Started             
ora.gsd
               OFFLINE OFFLINE      vrh1                                         
ora.net1.network
               ONLINE  ONLINE       vrh1                                         
ora.ons
               ONLINE  ONLINE       vrh1                                         
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr  https://www.askmac.cn
      1        ONLINE  ONLINE       vrh1                                         
ora.cvu
      1        OFFLINE OFFLINE                                                   
ora.oc4j
      1        OFFLINE OFFLINE                                                   
ora.scan1.vip
      1        ONLINE  ONLINE       vrh1                                         
ora.vprod.db
      1        ONLINE  OFFLINE                                                   
      2        ONLINE  OFFLINE                                                   
ora.vrh1.vip
      1        ONLINE  ONLINE       vrh1                                         
ora.vrh2.vip
      1        ONLINE  INTERMEDIATE vrh1                     FAILED OVER
Filed Under: Oracle, Oracle 11g新特性, Oracle RAC Tagged With: ,

11.2 中Oracle Cluster Registry(OCR)可选的存储设备

2011/10/30 by Leave a Comment

在11.2中ocr和votedisk 可以存放在ASM中了，该版本中Oracle Cluster Registry(OCR)可选的存储设备包括:

 

2011-10-29 00:13:18.828: [  OCROSD][1087046128]utstoragetypecommon:
Oracle Cluster Registry does not support the storage type configured.
OCR can be configured on: ASM, OCFS, OCFS2, NFS, Block Device, Character Device, VxFS
2011-10-29 00:13:18.829: [  OCROSD][1087046128]utopen:6m'': OCR location # [0] [/g01/ocrlocal]
configured is not a valid storage type. Return code [37].
2011-10-29 00:13:18.829: [  OCROSD][1087046128]utopen:7: failed to open any OCR file/disk,
errno=11, os err string=Resource temporarily unavailable
2011-10-29 00:13:18.829: [  OCRRAW][1087046128]proprinit: Could not open raw device
2011-10-29 00:13:18.829: [ default][1087046128]a_init:7!: Backend init unsuccessful : [26]
2011-10-29 00:13:19.831: [  OCROSD][1087046128]utstoragetypecommon:
Oracle Cluster Registry does not support the storage type configured.
OCR can be configured on: ASM, OCFS, OCFS2, NFS, Block Device, Character Device, VxFS

 

ASM, OCFS, OCFS2, NFS, Block Device, Character Device, VxFS都可以用来存放ocr文件。 虽然NFS是一种可用的选项，但是实际不推荐在产品环境使用网络文件系统， Mos Note<Mount Options for Oracle files when used with NAS devices [ID 359515.1]>介绍了mount NFS时的一些注意事项。

Filed Under: Oracle, Oracle 11g新特性, Oracle RAC Tagged With: , ,

几个关于oracle 11g ASM的问题

2011/02/21 by 4 Comments

Question:

1.11g Oracle Clusterware需要的OCR和Voting disk可以存储在ASM或者集群文件系统或者NFS中。对于全新安装，裸设备不再被支持(是否有办法使用裸设备？)。

2.使用ASM时，若相关存储上的磁盘路径(disk path)名前后不一致，是否仍然可以使用？需要什么调整？

Answer:

1.在11gr2 Grid Infrastructure全新安装时是没有办法使用裸设备的(You cannot install OCR or voting disk files on raw partitions. You can install only on Oracle ASM, or on supported network-attached storage or cluster file systems. The only use for raw devices is as Oracle ASM disks.);但可以通过后续的手段将OCR和VOTING DISK移动到裸设备上，如:

替换OCR:
ocrconfig -add rawdevice
ocrconfig -replace

替换voting disk
crsctl add votedisk css  -force
crsctl delete votedisk css  -force

具体可以参考Metalink文档<How to ADD/REMOVE/REPLACE/MOVE Oracle Cluster Registry (OCR) and Voting Disk>
实际上强烈不建议这样做。因为如果出现问题,Oracle GCS可以拒绝提供建议。

2.ASM是通过读取磁盘头部来了解磁盘内容的；磁盘路径名在安装时需要在所有节点一致，在安装完成后即便路径名改变也不会影响到ASM的使用。
需要注意的是在AIX操作平台上分配给ASM的磁盘(ASM DISK)，如果直接是HDISK形式的LUN则该HDISK不应当具有PVID(If the disk device has a PVID, then ASM will fail to mount the diskgroup created on the disk device.)。如果是裸的逻辑卷，那么所建VG应当是scaleable volume group。

Filed Under: Oracle, Oracle ASM Tagged With: , , ,

沪ICP备14014813号-2

沪公网安备 31010802001379号