GI

RAC Grid Infrastructure安装11.2.0.3.5 14727347 PSU GI-RDBMS补丁

2013/02/13 by admin 6 Comments

在《【视频教学】Maclean教你用Vbox在Linux 6.3上安装Oracle 11gR2 RAC》中我们介绍了在11.2.0.3 Grid Infrastructure GI环境下安装11.2.0.3.5 14727347 补丁的步骤；由于该11.2.0.3.5的opatch auto安装会有问题，所以我们使用手动的opatch apply安装该补丁，以下是检验步骤：

14727347解压后包含了2个补丁目录14727310和15876003 ，已包含11.2.0.3.5 DB的PSU，下载了该GI PSU后无需再去下载DB 的PSU了。

PSU 14727347的下载地址

1） rootcrs.pl 停止本节点的服务，若有RDBMS DB在运行则首先关闭该实例

su – oracle

$ <ORACLE_HOME>/bin/srvctl stop database –d <db-unique-name>

su – root

$GRID_HOME/crs/install/rootcrs.pl -unlock

2) 给GI HOME打补丁

AIX上： su – root; slibclean

su – grid

opatch napply -oh $GRID_HOME -local /tmp/patch/14727310
opatch napply -oh $GRID_HOME -local /tmp/patch/15876003/

3) 给RDBMS DB HOME打补丁

su – oracle

[oracle@vmac1 scripts]$ /tmp/patch/15876003/custom/server/15876003/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME
/tmp/patch/15876003/custom/server/15876003/custom/scripts/prepatch.sh completed successfully.

opatch napply -oh $ORACLE_HOME -local /tmp/patch/15876003/custom/server/15876003
opatch napply -oh $ORACLE_HOME -local /tmp/patch/14727310
/tmp/patch/15876003/custom/server/15876003/custom/scripts/postpatch.sh -dbhome $ORACLE_HOME

4)执行rootcrs.pl -patch

su – root

[root@vmac1 ~]# /g01/11ggrid/app/11.2.0/grid/rdbms/install/rootadd_rdbms.sh
[root@vmac1 ~]# /g01/11ggrid/app/11.2.0/grid/crs/install/rootcrs.pl -patch

5) 在2节点上重复以上步骤

6)对于现有的DB 执行升级字典操作

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> QUIT

7)执行utlrp 脚本并重启DB

SQL> @?/rdbms/admin/utlrp
SQL> shutdown immediate;
SQL> startup;

8) 重启应用程序

Filed Under: Oracle, Oracle补丁patch Tagged With: GI, psu

在11gR2 RAC中修改ASM DISK Path磁盘路径

2012/07/15 by admin 8 Comments

有同学在T.askmac.cn上提问关于修改11gR2中ASM DISK的路径问题，具体问题如下：

aix 6.1，grid 11.2.0.3+asm11.2.0.3+rac

建数据库的时候使用的是aix自带的多路径软件mpio，建了diskgroup

现在改造成veritas dmp多路径，已经修改了asm的disk_strings＝/dev/vx/rdmp/*，crs/asm启动的时候已经可以识别到磁盘/dev/vx/rdmp/开头的磁盘，但是读取不回原来的diskgroup信息。

crs启动的报错日志：
2012-07-13 15:07:29.748: [ GPNP][1286]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2108 clsgpnp_profileCallUrlInt] get-profile call to url “ipc://GPNPD_ggtest1” disco “” [f=0 claimed- host: cname: seq: auth:]
2012-07-13 15:07:29.762: [ GPNP][1286]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2236 clsgpnp_profileCallUrlInt] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote “ipc://GPNPD_ggtest1” disco “”
2012-07-13 15:07:29.762: [ CSSD][1286]clssnmReadDiscoveryProfile: voting file discovery string(/dev/vx/rdmp/*)
2012-07-13 15:07:29.762: [ CSSD][1286]clssnmvDDiscThread: using discovery string /dev/vx/rdmp/* for initial discovery
2012-07-13 15:07:29.762: [ SKGFD][1286]Discovery with str:/dev/vx/rdmp/*:

2012-07-13 15:07:29.762: [ SKGFD][1286]UFS discovery with :/dev/vx/rdmp/*:

2012-07-13 15:07:29.769: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_919:

2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_212:

2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_211:

2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_210:

2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_209:

2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_181:

2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_180:

2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_3:

2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_2:

2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_1:

2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_0:

2012-07-13 15:07:29.771: [ SKGFD][1286]OSS discovery with :/dev/vx/rdmp/*:

2012-07-13 15:07:29.771: [ SKGFD][1286]Handle 1115e7510 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_916:

2012-07-13 15:07:29.772: [ SKGFD][1286]Handle 1118758b0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_912:

2012-07-13 15:07:29.773: [ SKGFD][1286]Handle 1118d9cf0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_908:

2012-07-13 15:07:29.773: [ SKGFD][1286]Handle 1118da450 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_904:

2012-07-13 15:07:29.773: [ SKGFD][1286]Handle 1118dad70 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_903:

2012-07-13 15:07:29.802: [ CLSF][1286]checksum failed for disk:/dev/vx/rdmp/v_df8000_916:
2012-07-13 15:07:29.803: [ SKGFD][1286]Lib :UFS:: closing handle 1115e7510 for disk :/dev/vx/rdmp/v_df8000_916:

2012-07-13 15:07:29.803: [ SKGFD][1286]Lib :UFS:: closing handle 1118758b0 for disk :/dev/vx/rdmp/v_df8000_912:

2012-07-13 15:07:29.804: [ SKGFD][1286]Handle 1115e6710 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_202:

2012-07-13 15:07:29.808: [ SKGFD][1286]Handle 1115e7030 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_201:

2012-07-13 15:07:29.809: [ SKGFD][1286]Handle 1115e7ad0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_200:

2012-07-13 15:07:29.809: [ SKGFD][1286]Handle 1118733f0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_199:

2012-07-13 15:07:29.816: [ CLSF][1286]checksum failed for disk:/dev/vx/rdmp/v_df8000_186:
2012-07-13 15:07:29.816: [ SKGFD][1286]Lib :UFS:: closing handle 1118de5d0 for disk :/dev/vx/rdmp/v_df8000_186:

2012-07-13 15:07:29.816: [ CSSD][1286]clssnmvDiskVerify: Successful discovery of 0 disks
2012-07-13 15:07:29.816: [ CSSD][1286]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2012-07-13 15:07:29.816: [ CSSD][1286]clssnmvFindInitialConfigs: No voting files found
2012-07-13 15:07:29.816: [ CSSD][1286](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds
2012-07-13 15:07:30.169: [ CSSD][1029]clssgmExecuteClientRequest(): type(37) size(80) only connect and exit messages are allowed before lease acquisition proc(1115e4870) client(0)

其需求为修改ASM DISK PATH的磁盘设备路径，但是由于11gR2 RAC+ASM的特殊性，导致CRS无法正常启动，虽然使用crsctl start crs -excl -nocrs的方式可以启动CSS服务和ASM实例，

但是最后还是报(clssnmCompleteInitVFDiscovery: Voting file not found)，这是由于Voteing file投票磁盘的位置没有合理更新导致的。

这里我们来学习一下，如何正确的修改11gR2 RAC+ASM下的ASM DISK路径：

1.我们来营造一个修改ASM DISK路径的环境，这里我们使用UDEV的设备名绑定服务，并利用UDEV将原本的ASM DISK从/dev/asm-disk* 修改为 /dev/rasm-disk*的形式，这只需要修改udev rule文件即可实现：

[grid@maclean1 ~]$ export  ORACLE_HOME=/g01/grid/app/11.2.0/grid

[grid@maclean1 ~]$ /g01/grid/app/11.2.0/grid/bin/sqlplus  / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Sun Jul 15 04:09:28 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> show parameter diskstri 

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskstring                       string      /dev/asm*

可以看到当前ASM实例使用的asm_diskstring 为/dev/asm*，切换到root用户修改UDEV RULE文件：

[root@maclean1 rules.d]# cp 99-oracle-asmdevices.rules  99-oracle-asmdevices.rules.bak
[root@maclean1 rules.d]# vi 99-oracle-asmdevices.rules

[root@maclean1 rules.d]# cat 99-oracle-asmdevices.rules

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB09cadb31-cfbea255_", NAME="rasm-diskb", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB5f097069-59efb82f_", NAME="rasm-diskc", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB4e1a81c0-20478bc4_", NAME="rasm-diskd", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VBdcce9285-b13c5a27_", NAME="rasm-diske", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB82effe1a-dbca7dff_", NAME="rasm-diskf", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB950d279f-c581cb51_", NAME="rasm-diskg", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB14400d81-651672d7_", NAME="rasm-diskh", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB31b1237b-78aa22bb_", NAME="rasm-diski", OWNER="grid", GROUP="asmadmin", MODE="0660"

以上修改了原始99-oracle-asmdevices.rules的UDEV RULE规则文件，生成的设备名被修改为/dev/rasm-disk*的形式，不同于之前的ASM DISK设备名，这要求我们后续的一系列操作来保证RAC CRS可以正常启动。

当前运行时的votedisk和ocr 存放位置:

[root@maclean1 rules.d]# /g01/grid/app/11.2.0/grid/bin/crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   6896bfc3d1464f9fbf0ea9df87e023ad (/dev/asm-diskb) [SYSTEMDG]
 2. ONLINE   58eb81b656084ff2bfd315d9badd08b7 (/dev/asm-diskc) [SYSTEMDG]
 3. ONLINE   6bf7324625c54f3abf2c942b1e7f70d9 (/dev/asm-diskd) [SYSTEMDG]
 4. ONLINE   43ad8ae20c354f5ebf7083bc30bf94cc (/dev/asm-diske) [SYSTEMDG]
 5. ONLINE   4c225359d51b4f93bfba01080664b3d7 (/dev/asm-diskf) [SYSTEMDG]
Located 5 voting disk(s).

[root@maclean1 rules.d]# /g01/grid/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2844
         Available space (kbytes) :     259276
         ID                       :  879001605
         Device/File Name         :  +SYSTEMDG
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

因为votedisk file的位置具体指向一个ASM DISK，所以后续我们会需要crsctl replace votedisk，我们现在重启LINUX OS:

[root@maclean1 rules.d]# init 6

rebooting ............

[root@maclean1 dev]# ls -l *asm*
brw-rw---- 1 grid asmadmin 8,  16 Jul 15 04:15 rasm-diskb
brw-rw---- 1 grid asmadmin 8,  32 Jul 15 04:15 rasm-diskc
brw-rw---- 1 grid asmadmin 8,  48 Jul 15 04:15 rasm-diskd
brw-rw---- 1 grid asmadmin 8,  64 Jul 15 04:15 rasm-diske
brw-rw---- 1 grid asmadmin 8,  80 Jul 15 04:15 rasm-diskf
brw-rw---- 1 grid asmadmin 8,  96 Jul 15 04:15 rasm-diskg
brw-rw---- 1 grid asmadmin 8, 112 Jul 15 04:15 rasm-diskh
brw-rw---- 1 grid asmadmin 8, 128 Jul 15 04:15 rasm-diski

重启后自动获得了形如/dev/rasm-disk*的ASM DISK,查阅ASM日志可以发现css服务仍搜索/dev/asm*路径来获得ASM DISK，但是这将导致找不到任何有效的ASM DISK:

more /g01/grid/app/11.2.0/grid/log/maclean1/cssd/ocssd.log

2012-07-15 04:17:45.208: [ SKGFD][1099548992]Discovery with str:/dev/asm*:

2012-07-15 04:17:45.208: [   SKGFD][1099548992]UFS discovery with :/dev/asm*:

2012-07-15 04:17:45.208: [   SKGFD][1099548992]OSS discovery with :/dev/asm*:

2012-07-15 04:17:45.208: [ CSSD][1099548992]clssnmvDiskVerify: Successful discovery of 0 disks
2012-07-15 04:17:45.208: [    CSSD][1099548992]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2012-07-15 04:17:45.208: [    CSSD][1099548992]clssnmvFindInitialConfigs: No voting files found
2012-07-15 04:17:45.208: [    CSSD][1099548992](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. 
Retrying discovery in 15 seconds
2012-07-15 04:17:45.251: [    CSSD][1096661312]clssgmExecuteClientRequest(): type(37) size(80) only connect and 
exit messages are allowed before lease acquisition proc(0x26a8ba0) client((nil))
2012-07-15 04:17:45.251: [    CSSD][1096661312]clssgmDeadProc: proc 0x26a8ba0
2012-07-15 04:17:45.251: [    CSSD][1096661312]clssgmDestroyProc: cleaning up proc(0x26a8ba0) con(0xfe6) skgpid  
ospid 3751 with 0 clients, refcount 0
2012-07-15 04:17:45.252: [    CSSD][1096661312]clssgmDiscEndpcl: gipcDestroy 0xfe6
2012-07-15 04:17:45.829: [    CSSD][1096661312]clssscSelect: cookie accept request 0x2318ea0
2012-07-15 04:17:45.829: [    CSSD][1096661312]clssgmAllocProc: (0x2659480) allocated
2012-07-15 04:17:45.830: [    CSSD][1096661312]clssgmClientConnectMsg: properties of cmProc 0x2659480 - 1,2,3,4,5
2012-07-15 04:17:45.830: [    CSSD][1096661312]clssgmClientConnectMsg: Connect from con(0x114e) proc(0x2659480) pid(3751) 
version 11:2:1:4, properties: 1,2,3,4,5
2012-07-15 04:17:45.830: [    CSSD][1096661312]clssgmClientConnectMsg: msg flags 0x0000
2012-07-15 04:17:45.939: [    CSSD][1096661312]clssscSelect: cookie accept request 0x253ddd0
2012-07-15 04:17:45.939: [    CSSD][1096661312]clssscevtypSHRCON: getting client with cmproc 0x253ddd0
2012-07-15 04:17:45.939: [    CSSD][1096661312]clssgmRegisterClient: proc(3/0x253ddd0), client(61/0x26877b0)
2012-07-15 04:17:45.939: [    CSSD][1096661312]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are 
allowed before lease acquisition proc(0x253ddd0) client(0x26877b0)
2012-07-15 04:17:45.939: [    CSSD][1096661312]clssgmDiscEndpcl: gipcDestroy 0x1174
2012-07-15 04:17:46.070: [    CSSD][1096661312]clssscSelect: cookie accept request 0x26368a0
2012-07-15 04:17:46.070: [    CSSD][1096661312]clssscevtypSHRCON: getting client with cmproc 0x26368a0
2012-07-15 04:17:46.070: [    CSSD][1096661312]clssgmRegisterClient: proc(5/0x26368a0), client(50/0x26877b0)

由于11gR2中CRS服务依赖于ASM，因为ocr存放在ASM中，所以ASM若无法有效启动，这导致CRS服务也无法正常工作:

[root@maclean1 ~]# crsctl check has
CRS-4638: Oracle High Availability Services is online

[root@maclean1 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager

2. 以上是修改ASM DISK PATH磁盘路径的现状，我们需要通过以下操作来恢复CRS:

首先彻底关闭OHASD服务：

[root@maclean1 ~]# crsctl stop has -f 

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'maclean1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'maclean1'
CRS-2673: Attempting to stop 'ora.crf' on 'maclean1'
CRS-2677: Stop of 'ora.mdnsd' on 'maclean1' succeeded
CRS-2677: Stop of 'ora.crf' on 'maclean1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'maclean1'
CRS-2677: Stop of 'ora.gipcd' on 'maclean1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'maclean1'
CRS-2677: Stop of 'ora.gpnpd' on 'maclean1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'maclean1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

3. 以-excl -nocrs方式启动CRS，这将仅启动ASM 实例而不会启动CRS服务：

 [root@maclean1 ~]# crsctl start crs -excl -nocrs 

CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'maclean1'
CRS-2676: Start of 'ora.mdnsd' on 'maclean1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'maclean1'
CRS-2676: Start of 'ora.gpnpd' on 'maclean1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'maclean1'
CRS-2672: Attempting to start 'ora.gipcd' on 'maclean1'
CRS-2676: Start of 'ora.cssdmonitor' on 'maclean1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'maclean1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'maclean1'
CRS-2672: Attempting to start 'ora.diskmon' on 'maclean1'
CRS-2676: Start of 'ora.diskmon' on 'maclean1' succeeded
CRS-2676: Start of 'ora.cssd' on 'maclean1' succeeded
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'maclean1'
CRS-2672: Attempting to start 'ora.ctssd' on 'maclean1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'maclean1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'maclean1'
CRS-2676: Start of 'ora.ctssd' on 'maclean1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'maclean1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'maclean1'
CRS-2676: Start of 'ora.asm' on 'maclean1' succeeded	

#建议同时修改CRS_HOME所在的ORACLE_BASE为777权限，避免可能的问题
[root@maclean1 ~]# chmod 777 /g01

4.修改ASM实例的disk_strings为当前的ASM DISK PATH信息:

[root@maclean1 ~]# su - grid

[grid@maclean1 ~]$ sqlplus  / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Sun Jul 15 04:40:40 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter system set asm_diskstring='/dev/rasm*';

System altered.

SQL> alter diskgroup systemdg mount;

Diskgroup altered.

SQL> create spfile from memory;

File created.

SQL> startup force mount;
ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance
ASM instance started

Total System Global Area  283930624 bytes
Fixed Size                  2227664 bytes
Variable Size             256537136 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted

SQL> show parameter spfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      /g01/grid/app/11.2.0/grid/dbs/
                                                 spfile+ASM1.ora

SQL> show parameter disk

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups                       string      SYSTEMDG
asm_diskstring                       string      /dev/rasm*

SQL> create pfile from spfile;

File created.

SQL> create spfile='+SYSTEMDG' from pfile;

File created.

SQL> startup force;
ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance
ASM instance started

Total System Global Area  283930624 bytes
Fixed Size                  2227664 bytes
Variable Size             256537136 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted
SQL> show parameter spfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +SYSTEMDG/maclean-cluster/asmp
                                                 arameterfile/registry.253.7886
                                                 82933

以上成功修改了asm_diskstring ，且更新了ASM DISKGROUP上的SPFILE ，由于ASM使用共享的SPFILE所以其他节点上一般无需在做其他操作。

5. crsctl replace votedisk 命令将votedisk重置位置：

[root@maclean1 ~]# crsctl replace votedisk +systemdg

Successful addition of voting disk 864a00efcfbe4f42bfd0f4f6b60472a0.
Successful addition of voting disk ab14d6e727614f29bf53b9870052a5c8.
Successful addition of voting disk 754c03c168854f46bf2daee7287bf260.
Successful addition of voting disk 9ed58f37f3e84f28bfcd9b101f2af9f3.
Successful addition of voting disk 4ce7b7c682364f12bf4df5ce1fb7814e.
Successfully replaced voting disk group with +systemdg.
CRS-4266: Voting file(s) successfully replaced		

[root@maclean1 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   864a00efcfbe4f42bfd0f4f6b60472a0 (/dev/rasm-diskb) [SYSTEMDG]
 2. ONLINE   ab14d6e727614f29bf53b9870052a5c8 (/dev/rasm-diskc) [SYSTEMDG]
 3. ONLINE   754c03c168854f46bf2daee7287bf260 (/dev/rasm-diskd) [SYSTEMDG]
 4. ONLINE   9ed58f37f3e84f28bfcd9b101f2af9f3 (/dev/rasm-diske) [SYSTEMDG]
 5. ONLINE   4ce7b7c682364f12bf4df5ce1fb7814e (/dev/rasm-diskf) [SYSTEMDG]
Located 5 voting disk(s).

[root@maclean1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2844
         Available space (kbytes) :     259276
         ID                       :  879001605
         Device/File Name         :  +SYSTEMDG
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

以上replace了votedisk到新的 ASM DISK上，并确认votedisk和OCR均为可用状态。

6.重启CRS服务：

[root@maclean1 ~]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'maclean1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'maclean1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'maclean1'
CRS-2673: Attempting to stop 'ora.asm' on 'maclean1'
CRS-2677: Stop of 'ora.mdnsd' on 'maclean1' succeeded
CRS-2677: Stop of 'ora.asm' on 'maclean1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'maclean1'
CRS-2677: Stop of 'ora.ctssd' on 'maclean1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'maclean1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'maclean1'
CRS-2677: Stop of 'ora.cssd' on 'maclean1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'maclean1'
CRS-2677: Stop of 'ora.gipcd' on 'maclean1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'maclean1'
CRS-2677: Stop of 'ora.gpnpd' on 'maclean1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'maclean1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@maclean1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.BACKUPDG.dg
               ONLINE  ONLINE       maclean1                                     
ora.DATA.dg
               ONLINE  ONLINE       maclean1                                     
ora.LISTENER.lsnr
               ONLINE  ONLINE       maclean1                                     
ora.SYSTEMDG.dg
               ONLINE  ONLINE       maclean1                                     
ora.asm
               ONLINE  ONLINE       maclean1                 Started             
ora.gsd
               OFFLINE OFFLINE      maclean1                                     
ora.net1.network
               ONLINE  ONLINE       maclean1                                     
ora.ons
               ONLINE  ONLINE       maclean1                                     
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       maclean1                                     
ora.cvu
      1        ONLINE  ONLINE       maclean1                                     
ora.maclean1.vip
      1        ONLINE  ONLINE       maclean1                                     
ora.maclean2.vip
      1        ONLINE  INTERMEDIATE maclean1                 FAILED OVER         
ora.oc4j
      1        ONLINE  OFFLINE                               STARTING            
ora.prod.db
      1        ONLINE  OFFLINE                               Instance Shutdown,S 
                                                             TARTING             
      2        ONLINE  OFFLINE                                                   
ora.scan1.vip
      1        ONLINE  ONLINE       maclean1

因为上面更新了ASM共享使用的SPFILE，所以其他节点上一般不会存在问题，直接重启后CRS即可正常工作。

由于11gR2 RAC+ASM的启动依赖较为复杂，虽然通过以上操作但你仍可能在修改ASM DISK PATH路径时遇到一些问题，若无法解决请去该问题的原帖提问，谢谢配合！

Filed Under: Oracle, Oracle 11g新特性, Oracle ASM Tagged With: 11gR2 RAC, crsctl replace votedisk, GI

Upgrade GI/CRS 11.1.0.7 to 11.2.0.2. Rootupgrade.sh Hanging

2011/09/11 by admin Leave a Comment

Upgrade grid 11.1.0.7 to 11.2.0.2. Rootupgrade.sh Hanging

We installed 11gR2 GI software and applied PSU2 patches upon getting runupgrade.sh prompt.runupgrade.sh hang on the first node.

[root@vrh8 client]# uname -a
Linux vrh8 2.6.18-238.5.1.el5 #1 SMP Mon Feb 21 05:52:39 EST 2011 x86_64

x86_64 x86_64 GNU/Linux
cluvfy passed with 2 ignorable errors:

[root@vrh8 vrh8]# cd /tmp
[root@vrh8 tmp]# df -lh .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg0-tmp 992M 263M 679M 28% /tmp

[root@vrh8 grid]# grep fail cluvfy_during_inst.log
/tmp l118464lwap1049 /tmp 713MB 1GB failed
Result: Free disk space check failed for “l118464lwap1049:/tmp”
/tmp vrh8 /tmp 692.131MB 1GB failed
Result: Free disk space check failed for “vrh8:/tmp”
Result: Check for multiple users with UID value 0 failed

[root@vrh8 vrh8]# cd /tmp
[root@vrh8 tmp]# df -lh .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg0-tmp 992M 263M 679M 28% /tmp

We installed 11gR2 GI software and applied PSU2 patches upon getting runupgrade.sh prompt.

runupgrade.sh hang on the first node. We followed “How to Proceed from Failed Upgrade to 11gR2

Grid Infrastructure on Linux/Unix [ID 969254.1]” 1A section, it didn’t help.

[root@vrh8 bin]# ./crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.1.0.7.0]

rootupgrade.sh output:

[root@vrh8 11.2.0.2]# ./rootupgrade.sh
Running Oracle 11g root script…

The following environment variables are set as:
ORACLE_OWNER= oracrs
ORACLE_HOME= /d22/oracrs/11.2.0.2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of “dbhome” have not changed. No need to overwrite.
The contents of “oraenv” have not changed. No need to overwrite.
The contents of “coraenv” have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /d22/oracrs/11.2.0.2/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
OLR initialization – successful
Adding daemon to inittab
ACFS-9200: Supported
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies – this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.

****hanging here for more than 2 hrs, so we cancelled it

INT at /d22/oracrs/11.2.0.2/crs/install/crsconfig_lib.pm line 1173.
/d22/oracrs/11.2.0.2/perl/bin/perl -I/d22/oracrs/11.2.0.2/perl/lib –

I/d22/oracrs/11.2.0.2/crs/install /d22/oracrs/11.2.0.2/crs/install/rootcrs.pl execution failed
Oracle root script execution aborted!

1. The below logs are required to analyze this issue.

NEW_GRID_HOME/cfgtoollogs/crsconfig/*.*
NEW_GRID_HOME/log/<nodename>/*.*

Please upload the logs under the above directories. Zip and upload the files including the subdirectories.

2. When the rootupgrade was handing, did you check the usage of /tmp. Was free space exhausting?

=== ODM Research ===

There has been multiple root script run for upgrade. I have taken the first incident from the file
rootcrs_vrh8.log:
—————————————–

2011-02-13 13:07:55: Successfully started requested Oracle stack daemons
2011-02-13 13:07:55: Upgrading the existing voting disks!
2011-02-13 13:07:55: Executing /d22/oracrs/11.2.0.2/bin/cssvfupgd
2011-02-13 13:07:55: Executing cmd: /d22/oracrs/11.2.0.2/bin/cssvfupgd <<<<<<<<<<<<<<< The root script seems to hang at this point.
2011-02-13 15:01:16: ###### Begin DIE Stack Trace ######
2011-02-13 15:01:16: Package File Line Calling
2011-02-13 15:01:16: ————— ——————– —- ———-
2011-02-13 15:01:16: 1: main rootcrs.pl 325 crsconfig_lib::dietrap
2011-02-13 15:01:16: 2: crsconfig_lib crsconfig_lib.pm 9301 main::__ANON__
2011-02-13 15:01:16: 3: crsconfig_lib crsconfig_lib.pm 9301 (eval)
2011-02-13 15:01:16: 4: crsconfig_lib crsconfig_lib.pm 9260 crsconfig_lib::system_cmd_capture1
2011-02-13 15:01:16: 5: crsconfig_lib crsconfig_lib.pm 9247 crsconfig_lib::system_cmd_capture
2011-02-13 15:01:16: 6: crsconfig_lib crsconfig_lib.pm 924 crsconfig_lib::system_cmd
2011-02-13 15:01:16: 7: oracss oracss.pm 275 crsconfig_lib::run_crs_cmd
2011-02-13 15:01:16: 8: crsconfig_lib crsconfig_lib.pm 1019 oracss::CSS_upgrade
2011-02-13 15:01:16: 9: crsconfig_lib crsconfig_lib.pm 1006 crsconfig_lib::start_cluster
2011-02-13 15:01:16: 10: main rootcrs.pl 697 crsconfig_lib::perform_start_cluster
2011-02-13 15:01:16: ####### End DIE Stack Trace #######

cssvfupgd.log:
——————–
Oracle Database 11g Clusterware Release 11.2.0.2.0 – Production Copyright 1996, 2010 Oracle. All rights reserved.
2011-02-13 13:07:55.356: [ OCRRAW][3605955376]prgval:buffer passed is too small
2011-02-13 13:07:55.361: [CSSVFUPG][3605955376]cssvfupgd_GetVFList: found voting file /s01/app/ocrvot/VOTEDISK/UAT2_vdisk1.dat
2011-02-13 13:07:55.365: [ OCRRAW][3605955376]prgval:buffer passed is too small
2011-02-13 13:07:55.369: [CSSVFUPG][3605955376]cssvfupgd_GetVFList: found voting file /s01/app/ocrvot/VOTEDISK/UAT2_vdisk2.dat
2011-02-13 13:07:55.373: [ OCRRAW][3605955376]prgval:buffer passed is too small
2011-02-13 13:07:55.377: [CSSVFUPG][3605955376]cssvfupgd_GetVFList: found voting file /s01/app/ocrvot/VOTEDISK/UAT2_vdisk3.dat
2011-02-13 13:07:55.402: [CSSVFUPG][3605955376]cssvfupgd_SetNum: Processing SYSTEM.css.misscount
2011-02-13 13:07:55.404: [CSSVFUPG][3605955376]cssvfupgd_SetNum: Processing SYSTEM.css.disktimeout
2011-02-13 13:07:55.406: [CSSVFUPG][3605955376]cssvfupgd_SetNum: Processing SYSTEM.css.reboottime
2011-02-13 13:07:55.408: [CSSVFUPG][3605955376]cssvfupgd_SetNum: Processing SYSTEM.css.diagwait
2011-02-13 13:07:55.414: [CSSVFUPG][3605955376]cssvfupgd_SetNum: Processing SYSTEM.css.pollinterval
2011-02-13 13:07:55.416: [CSSVFUPG][3605955376]cssvfupgd_GetGUID: Fetching GUID for /s01/app/ocrvot/VOTEDISK/UAT2_vdisk1.dat
2011-02-13 13:07:55.419: [ SKGFD][3605955376]NOTE: No asm libraries found in the system

2011-02-13 13:07:55.419: [ CLSF][3605955376]Allocated CLSF context
2011-02-13 13:07:55.419: [ SKGFD][3605955376]Discovery with str:/s01/app/ocrvot/VOTEDISK/UAT2_vdisk1.dat:

2011-02-13 13:07:55.419: [ SKGFD][3605955376]UFS discovery with :/s01/app/ocrvot/VOTEDISK/UAT2_vdisk1.dat:

2011-02-13 13:07:55.420: [ SKGFD][3605955376]Fetching UFS disk :/s01/app/ocrvot/VOTEDISK/UAT2_vdisk1.dat:

2011-02-13 13:07:55.420: [ SKGFD][3605955376]OSS discovery with :/s01/app/ocrvot/VOTEDISK/UAT2_vdisk1.dat:

2011-02-13 13:07:55.421: [ SKGFD][3605955376]Handle 0x124de360 from lib :UFS:: for disk :/s01/app/ocrvot/VOTEDISK/UAT2_vdisk1.dat:

2011-02-13 14:19:31.132: [ SKGFD][3605955376]WARNING:io_getevents timed out 2226 sec >>>>>>>>>>>>>>>>>>>> After about one hour it shows time out error.

2011-02-13 14:19:31.132: [ SKGFD][3605955376]WARNING:io_getevents timed out 2226 sec

The script has stalled at the voting disk upgrade phase. Please provide me the below details.

1. What cluster file system are you using for the voting files? provide its details and the mount options used.

for ocfs, get its mount options
mount | grep ocfs

3. Voting disks details
ls -l /s01/app/ocrvot/VOTEDISK/UAT2_vdisk*

4. Get the diagwait detail.
OLD_CRS_HOME/bin/crsctl get css diagwait

1. What cluster file system are you using for the voting files? provide its details and the mount options used
/dev/emcpowera1 on /s01/app/ocrvot type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)

2. Voting disks details

[root@vrh8 11.2.0.2]# ls -l /s01/app/ocrvot/VOTEDISK/UAT2_vdisk*
-rw-r—– 1 oracrs oinstall 21004288 Jun 11 07:31 /s01/app/ocrvot/VOTEDISK/UAT2_vdisk1.dat
-rw-r—– 1 oracrs oinstall 21004288 Jun 11 07:31 /s01/app/ocrvot/VOTEDISK/UAT2_vdisk2.dat
-rw-r—– 1 oracrs oinstall 21004288 Jun 11 07:31 /s01/app/ocrvot/VOTEDISK/UAT2_vdisk3.dat

3. Get the diagwait detail

crsctl get css diagwait
Failure 33 in main Oracle Cluster Registry context initialization: PROC-33: Oracle Cluster Registry is not configured Operating System error [No such file or directory] [2]

owc may not be required now as the issue we face is clear.

The diagwait should not error out, as explained in the following note,
11gR2 rootupgrade.sh Fails as cssvfupgd Can not Upgrade Voting Disk (Doc ID 1102283.1)

Make sure you are running ‘crsctl get css diagwait’ from the old crs home. You can also check it in multiple node. If it errors out, this has to be fixed as explained in the above note.

according to that note ,When I ./oprocd stop ,get error:
[root@l118464lwap1049 bin]# ./oprocd stop
Jun 16 23:24:42.966 | ERR | failed to connect to daemon, errno(111)

ACFS-9200: Supported
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies – this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.

cssvfupgd.log
2011-02-13 23:36:49.311: [ OCRRAW][3394941744]prgval:buffer passed is too small
2011-02-13 23:36:49.315: [CSSVFUPG][3394941744]cssvfupgd_GetVFList: found voting
file /s01/app/ocrvot/VOTEDISK/UAT2_vdisk2.dat
2011-02-13 23:36:49.319: [ OCRRAW][3394941744]prgval:buffer passed is too small
2011-02-13 23:36:49.323: [CSSVFUPG][3394941744]cssvfupgd_GetVFList: found voting
file /s01/app/ocrvot/VOTEDISK/UAT2_vdisk3.dat
2011-02-13 23:36:49.351: [CSSVFUPG][3394941744]cssvfupgd_SetNum: Processing SYST
EM.css.misscount
2011-02-13 23:36:49.354: [CSSVFUPG][3394941744]cssvfupgd_SetNum: Processing SYST
EM.css.disktimeout
2011-02-13 23:36:49.356: [CSSVFUPG][3394941744]cssvfupgd_SetNum: Processing SYST
EM.css.reboottime
2011-02-13 23:36:49.358: [CSSVFUPG][3394941744]cssvfupgd_SetNum: Processing SYST
EM.css.diagwait
2011-02-13 23:36:49.367: [CSSVFUPG][3394941744]cssvfupgd_SetNum: Processing SYST
EM.css.pollinterval
2011-02-13 23:36:49.369: [CSSVFUPG][3394941744]cssvfupgd_GetGUID: Fetching GUID
for /s01/app/ocrvot/VOTEDISK/UAT2_vdisk1.dat
2011-02-13 23:36:49.371: [ SKGFD][3394941744]NOTE: No asm libraries found in t
he system

2011-02-13 23:36:49.372: [ CLSF][3394941744]Allocated CLSF context
2011-02-13 23:36:49.372: [ SKGFD][3394941744]Discovery with str:/s01/app/ocrvo
t/VOTEDISK/UAT2_vdisk1.dat:

2011-02-13 23:36:49.372: [ SKGFD][3394941744]UFS discovery with :/s01/app/ocrv
ot/VOTEDISK/UAT2_vdisk1.dat:

2011-02-13 23:36:49.372: [ SKGFD][3394941744]Fetching UFS disk :/s01/app/ocrvo
t/VOTEDISK/UAT2_vdisk1.dat:

2011-02-13 23:36:49.372: [ SKGFD][3394941744]OSS discovery with :/s01/app/ocrv
ot/VOTEDISK/UAT2_vdisk1.dat:

2011-02-13 23:36:49.372: [ SKGFD][3394941744]Handle 0x98c4360 from lib :UFS::
for disk :/s01/app/ocrvot/VOTEDISK/UAT2_vdisk1.dat:

Question:
in Your update about cssvfupgd.log You stated it was hanging there.
Is there an entry after about 70 minutes about a timeout in that log file like:

2011-02-13 23:36:49.372: [ SKGFD][3394941744]Handle 0x98c4360 from lib :UFS::
for disk :/s01/app/ocrvot/VOTEDISK/UAT2_vdisk1.dat:
2011-02-17 0:48:19.372: [ SKGFD][3394941744]WARNING:io_getevents timed out 4294 sec <<<< present ???

Please provide the following outputs:
rpm -qa|grep ocfs2
uname -a
cat /etc/redhat-release

[root@vrh8 ~]# rpm -qa|grep ocfs2
ocfs2console-1.4.4-1.el5
ocfs2-tools-1.4.4-1.el5
ocfs2-2.6.18-238.5.1.el5-1.4.7-1.el5
[root@vrh8 ~]# uname -a
Linux vrh8 2.6.18-238.5.1.el5 #1 SMP Mon Feb 21 05:52:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
[root@vrh8 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.6 (Tikanga)
[root@vrh8 ~]#

Combinations that install SUCCESSFUL:

OEL5.4+ocfs2-1.4.7-1+ocfs2-tools-1.4.4
OEL5.6+ocfs2-1.4.8-1+ocfs2-tools-1.6.3
OEL5.6+ocfs2-1.4.7-1+ocfs2-tools-1.4.4
RHLE5.6+OEL kernel(redhat compatible kernel)+ocfs2-1.4.8-1+ocfs2-tools-1.6.3
RHLE5.6+OEL kernel(redhat compatible kernel)+ocfs2-1.4.7-1+ocfs2-tools-1.4.4
RHEL5.4

Combinations that failed:
RHLE5.6(redhat kernel)+ocfs2-1.4.7-1+ocfs2-tools-1.4.4
RHLE5.6(redhat kernel)+ocfs2-1.4.8-1+ocfs2-tools-1.6.3

Problem reproduces with redhat kernel — RHEL 5.6 with 2.6.18-2xx kernels

Please review the following Note to change the location of your voting disk
Note 428681.1
Title: How to ADD/REMOVE/REPLACE/MOVE Oracle Cluster Registry (OCR) and Voting Disk

Pasting info from —
Oracle? Clusterware Administration and Deployment Guide
11g Release 2 (11.2)

3 Managing Oracle Cluster Registry and Voting Disks
Oracle Universal Installer for Oracle Clusterware 11g release 2 (11.2), does not support the use of raw or block devices. However, if you upgrade from a previous Oracle Clusterware release, then you can continue to use raw or block devices.

[oracrs@vrh8 grid]$ grep fail cluvfy_during_inst_061711.log
/tmp l118464lwap1049 /tmp 706MB 1GB failed
Result: Free disk space check failed for “l118464lwap1049:/tmp”
/tmp vrh8 /tmp 927.1312MB 1GB failed
Result: Free disk space check failed for “vrh8:/tmp”
Result: Check for multiple users with UID value 0 failed
PRVF-5431 : Oracle Cluster Voting Disk configuration check failed

[oracrs@vrh8 grid]$ ./runcluvfy.sh stage -pre crsinst -n vrh8,l118464lwap1049 -verbose|tee cluvfy_during_inst.log

Please upload the following Cluvfy trace log —
$ORA_CRS_HOME/cv/log/cvutrace.log.0

Please download the latest CVU from OTN:
http://www.oracle.com/technetwork/database/clustering/downloads/cvu-download-homepage-099973.html

Please upload
/s02/app/crs/11.2.0.2/log/vrh8/agent/ohasd/oraagent_oracrs/oraagent_oracrs.log

In addition pls upload
/s02/app/crs/11.2.0.2/log/vrh8/agent/ohasd/oracssdagent_root/oracssdagent_root.log

Please run this command on both the new setup and your existing production setup for a quick comparison —
rpm -qa|grep ocfs2

Server with issue:
[root@vrh8 ohasd]# rpm -qa|grep ocfs2
ocfs2console-1.4.4-1.el5
ocfs2-tools-1.4.4-1.el5
ocfs2-2.6.18-238.5.1.el5-1.4.7-1.el5

Prod:

[root@vrh9 bin]# rpm -qa|grep ocfs2
ocfs2-2.6.18-194.el5-1.4.7-1.el5
ocfs2console-1.4.4-1.el5
ocfs2-tools-1.4.4-1.el5
ocfs2-2.6.18-194.8.1.el5-1.4.7-1.el5

[root@vrh8 ~]# uname -a
Linux vrh8 2.6.18-238.5.1.el5 #1 SMP Mon Feb 21 05:52:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

[root@vrh8 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.6 (Tikanga)

rpm -qa|grep ocfs2
ocfs2console-1.4.4-1.el5
ocfs2-tools-1.4.4-1.el5
ocfs2-2.6.18-238.5.1.el5-1.4.7-1.el5

@ . from Bug 11876815 (Doc ID 1321757.1)
@ combinations that install SUCCESSFUL:
@ .
@ OEL5.4+ocfs2-1.4.7-1+ocfs2-tools-1.4.4
@ OEL5.6+ocfs2-1.4.8-1+ocfs2-tools-1.6.3
@ OEL5.6+ocfs2-1.4.7-1+ocfs2-tools-1.4.4
@ RHLE5.6+OEL kernel(redhat compatible kernel)+ocfs2-1.4.8-1+ocfs2-tools-1.6.3
@ RHLE5.6+OEL kernel(redhat compatible kernel)+ocfs2-1.4.7-1+ocfs2-tools-1.4.4
@ RHEL5.4
@ .
@ combinations that failed:
@ RHLE5.6(redhat kernel)+ocfs2-1.4.7-1+ocfs2-tools-1.4.4
@ RHLE5.6(redhat kernel)+ocfs2-1.4.8-1+ocfs2-tools-1.6.3
@ .
@ .
@ So that is clear that , it is redhat kernel’s problem.Since RHEL5.6 redhat
@ provided 2.6.18-2xx kernels, we can’t fix redhat kernels, please use Oracle
@ Enterprise kernel (redhat compatible) for installation.

As per last action plan (conveyed if any) you need to contact REDHAT support to know the cause of this issue. Workaround is to not use OCFS and go for raw device for upgrade to succeed.
A Oracle bug 11876815 was logged internally for this hang issue and few combinations of OEL, RHEL, OCFS2 were tried and tested and the combination you are using has not worked for us too (per bug internal updates given above)
The solution provided by Oracle bug developer is to use OEL and not RHEL or contact RHEL support for identifying the cause and solution (incase they have already tested this setup).
Let me know if RHEL support is already engaged and provide the case id so that I can open internal SR for Oracle/Red Hat Joint Escalation Team (JET) Engagement for both vendors to work together internally.

+ the SR issue of grid upgrade from 11.1 to 11.2.0.2.2 is resolved
– voting disk was moved from ocfs to raw device – as a workaround for Bug 11876815
– set TMP and TEMP env to new dir with availabe space before running the installer and prechecks to succeed
– applied GIPSU#2 before the rootupgrade.sh step
– rootupgrade.sh step was successful on all nodes
– verified post upgrade checks and logs to confirm GI upgrade was success !

+ DB upgrade to 11.2.0.2 Plus PSU#2 will be resumed shorlty

Filed Under: Oracle, Oracle 11g新特性, Oracle RAC Tagged With: 11.2, CRS, GI, OCFS, Oracle, Oracle Bug, raw, Redhat, rootupgrade.sh, upgrade

Slide:Upgrade 11.2.0.1 GI/CRS to 11.2.0.2 in Linux

2011/09/08 by admin Leave a Comment

Upgrade 11.2.0.1 gi crs to 11.2.0.2 in linux

View more documents from Maclean Liu

Filed Under: Oracle, Oracle RAC Tagged With: 11.2.0.2, CRS, GI, RAC, slide, upgrade

Oracle安装与操作系统用户组

2011/09/05 by admin Leave a Comment

Oracle软件在安装维护过程中长要和操作用户组(OS user group)打交道，从早前的只有oracle用户和dba组发展到今天11gr2中的grid用户和asm组，Oracle管理的日新月异可见一斑。

我们在单实例(single-instance)环境中常用的三个操作用户组，分别是:

oinstall用户组

oinstall 组是Oracle推荐创建的OS用户组之一，建议在系统第一次安装oracle软件产品之前创建该oinstall组，理论上该oinstall组应当拥有oracle软件产品目录(例如$CRS_HOME和$ORACLE_HOME)和oracle Inventory信息目录仓库，oracle Inventory信息目录记录了系统上安装过的oracle产品的记录。关于oracle Inventory产品信息仓库更多内容可以参考<深入理解Oracle Universal Installer (OUI)>一文。

若系统中已有安装过oracle产品软件，则现有的oracle Inventory目录的所有组必须是今后用来安装新oracle软件产品的用户的主组(primary group)。

现有的oracle Inventory拥有者组可以通过/etc/oraInst.loc位置文件了解:

inventory_loc=/u01/app/oracle/oraInventory
inst_group=oinstall

若/etc/oraInst.loc(少数平台不在该位置)位置文件不存在，那么建议创建oinstall用户组，注意在RAC环境中要保持各节点上用户组的gid一致:

# /usr/sbin/groupadd -g GID oinstall

OSDBA用户组(dba)

OSDBA是我们必须要创建的一种系统DBA用户组(dba)，若没有该用户组我们将无法安装数据库软件及执行管理数据库的任务。

OSOPER用户组(oper)

OSOPER是一种额外的用户组(oper)，我们可以选择要不要创建该用户组，创建该用户组可以满足让os用户行使某些数据库管理权限(包括SYSOPER角色权限)的目的。注意SYSOPER的权限包括startup和shutdown，所以要小心为该用户组添加成员。

创建OSOPER用户组的方法:

# /usr/sbin/groupadd oper

综上所述在单机环境(single-instance)中oracle软件拥有者用户(常见的oracle或者orauser)，因该同时是oinstall、dba、oper用户组的成员。同时该用户的主用户组必须是oinstall。

Oracle Database 11g release 2中选择Privileged Operating System Groups

而在11.2的GI/CRS环境中数据库软件拥有者用户(oracle或orauser)还必须是asmdba用户组的成员。

usermod -g oinstall -G dba,oper,asmdba [oracle|orauser]

id oracle
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),701(asmdba),54324(oper)

注意OSDBA和OSOPER用户组都受到$ORACLE_HOME/rdbms/lib/config.c 源文件的影响，该文件定义了默认的 SS_DBA_GRP “dba” 和SS_OPER_GRP “oper”,该源文件内容如下:

/*  Refer to the Installation and User's Guide for further information.  */

/* IMPORTANT: this file needs to be in sync with
              rdbms/src/server/osds/config.c, specifically regarding the
              number of elements in the ss_dba_grp array.
 */

#define SS_DBA_GRP "dba"
#define SS_OPER_GRP "oper"
#define SS_ASM_GRP ""

char *ss_dba_grp[] = {SS_DBA_GRP, SS_OPER_GRP, SS_ASM_GRP};
~

11g release2中oracle建议独立地管理Grid Infrastructure和ASM实例，因此有必要创建更多的os用户组以满足不同的权限分配。

我们在11.2的GI中常用的ASM用户组有以下三个：

OSASM(asmadmin)用户组

如果使用ASM，那么我们必须创建osasm(asmadmin)用户组，该OSASM用户组的成员将被赋予SYSASM权限，以满足组成员管理Oracle Clusterware和Oracle ASM的权限需求。

OSDBA for ASM group(asmdba)用户组

OSDBA(asmdba)用户组的成员将被赋予读写访问ASM文件的权限。GI/CRS拥有者用户和所有oracle数据库软件的拥有者必须是该组的成员。同时所有OSDBA(dba)用户组的成员也必须是asmdba组的成员。

OSOPER for ASM(asmoper)用户组

asmoper和osoper类似都是额外的可选择创建的用户组，创建该独立的用户组以满足赋予用户一套受限的ASM实例管理权限(ASM的SYSOPER角色)，该权限包括了启动和停止ASM实例，默认情况下OSASM(asmadmin)组成员将拥有所有SYSOPER的ASM管理权限。

在11.2的GI/CRS环境中一般会创建grid或griduser用户来管理GI软件和ASM实例，以如下方式创建grid用户:

 useradd -g oinstall -G asmadmin,asmdba,asmoper grid  

 id grid
 uid=54322(grid) gid=54321(oinstall) groups=54321(oinstall),700(asmadmin),701(asmdba),55000(asmoper)

Oracle 11g release2 Grid Infrastructure中选择Privileged Operating System Groups:

综合上述OS用户和用户组间的关系：

更多内容可以参考下文:

The OSDBA group (typically, dba)

You must create this group the first time you install Oracle Database software on the system. This group identifies operating system user accounts that have database administrative privileges (the SYSDBA privilege). If you do not create separate OSDBA, OSOPER and OSASM groups for the Oracle ASM instance, then operating system user accounts that have the SYSOPER and SYSASM privileges must be members of this group. The name used for this group in Oracle code examples is dba. If you do not designate a separate group as the OSASM group, then the OSDBA group you define is also by default the OSASM group.

To specify a group name other than the default dba group, then you must choose the Advanced installation type to install the software or start Oracle Universal Installer (OUI) as a user that is not a member of this group. In this case, OUI prompts you to specify the name of this group.

Members of the OSDBA group formerly were granted SYSASM privileges on Oracle ASM instances, including mounting and dismounting disk groups. This privileges grant is removed with Oracle Grid Infrastructure 11g release 2, if different operating system groups are designated as the OSDBA and OSASM groups. If the same group is used for both OSDBA and OSASM, then the privilege is retained.

The OSOPER group for Oracle Database (typically, oper)

This is an optional group. Create this group if you want a separate group of operating system users to have a limited set of database administrative privileges (the SYSOPER privilege). By default, members of the OSDBA group also have all privileges granted by the SYSOPER privilege.

To use the OSOPER group to create a database administrator group with fewer privileges than the default dba group, then you must choose the Advanced installation type to install the software or start OUI as a user that is not a member of the dba group. In this case, OUI prompts you to specify the name of this group. The usual name chosen for this group is oper.

The Oracle Automatic Storage Management Group (typically asmadmin)

This is a required group. Create this group as a separate group if you want to have separate administration privilege groups for Oracle ASM and Oracle Database administrators. In Oracle documentation, the operating system group whose members are granted privileges is called the OSASM group, and in code examples, where there is a group specifically created to grant this privilege, it is referred to as asmadmin.

If you have multiple databases on your system, and use multiple OSDBA groups so that you can provide separate SYSDBA privileges for each database, then you should create a separate OSASM group, and use a separate user from the database users to own the Oracle Grid Infrastructure installation (Oracle Clusterware and Oracle ASM). Oracle ASM can support multiple databases.

Members of the OSASM group can use SQL to connect to an Oracle ASM instance as SYSASM using operating system authentication. The SYSASM privileges permit mounting and dismounting disk groups, and other storage administration tasks. SYSASM privileges provide no access privileges on an RDBMS instance.

The Oracle ASM Database Administrator group (OSDBA for ASM, typically asmdba)

Members of the Oracle ASM Database Administrator group (OSDBA for ASM) are granted read and write access to files managed by Oracle ASM. The Oracle Grid Infrastructure installation owner and all Oracle Database software owners must be a member of this group, and all users with OSDBA membership on databases that have access to the files managed by Oracle ASM must be members of the OSDBA group for ASM.

Members of the Oracle ASM Operator Group (OSOPER for ASM, typically asmoper)

This is an optional group. Create this group if you want a separate group of operating system users to have a limited set of Oracle ASM instance administrative privileges (the SYSOPER for ASM privilege), including starting up and stopping the Oracle ASM instance. By default, members of the OSASM group also have all privileges granted by the SYSOPER for ASM privilege.

To use the Oracle ASM Operator group to create an ASM administrator group with fewer privileges than the default asmadmin group, then you must choose the Advanced installation type to install the software, In this case, OUI prompts you to specify the name of this group. In code examples, this group is asmoper.

An Oracle central inventory group, or oraInventory group (oinstall). Members who have the central inventory group as their primary group, are granted the OINSTALL permission to write to the oraInventory directory.

A single system privileges group that is used as the OSASM, OSDBA, OSDBA for ASM, and OSOPER for ASM group (dba), whose members are granted the SYSASM and SYSDBA privilege to administer Oracle Clusterware, Oracle ASM, and Oracle Database, and are granted SYSASM and OSOPER for ASM access to the Oracle ASM storage.

An Oracle grid installation for a cluster owner (grid), with the oraInventory group as its primary group, and with the OSASM group as the secondary group, with its Oracle base directory /u01/app/grid.

An Oracle Database owner (oracle) with the oraInventory group as its primary group, and the OSDBA group as its secondary group, with its Oracle base directory /u01/app/oracle.

/u01/app owned by grid:oinstall with 775 permissions before installation, and by root after the root.sh script is run during installation. This ownership and permissions enables OUI to create the Oracle Inventory directory, in the path /u01/app/oraInventory.

/u01 owned by grid:oinstall before installation, and by root after the root.sh script is run during installation.

/u01/app/11.2.0/grid owned by grid:oinstall with 775 permissions. These permissions are required for installation, and are changed during the installation process.

/u01/app/grid owned by grid:oinstall with 775 permissions before installation, and 755 permissions after installation.

/u01/app/oracle owned by oracle:oinstall with 775 permissions.

An Oracle central inventory group, or oraInventory group (oinstall), whose members that have this group as their primary group are granted permissions to write to the oraInventory directory.

A separate OSASM group (asmadmin), whose members are granted the SYSASM privilege to administer Oracle Clusterware and Oracle ASM.

A separate OSDBA for ASM group (asmdba), whose members include grid, oracle1 and oracle2, and who are granted access to Oracle ASM.

A separate OSOPER for ASM group (asmoper), whose members are granted limited Oracle ASM administrator privileges, including the permissions to start and stop the Oracle ASM instance.

An Oracle grid installation for a cluster owner (grid), with the oraInventory group as its primary group, and with the OSASM (asmadmin), OSDBA for ASM (asmdba) group as a secondary group.

Two separate OSDBA groups for two different databases (dba1 and dba2) to establish separate SYSDBA privileges for each database.

Two Oracle Database software owners (oracle1 and oracle2), to divide ownership of the Oracle database binaries, with the OraInventory group as their primary group, and the OSDBA group for their database (dba1 or dba2) and the OSDBA for ASM group (asmdba) as their secondary groups.

An OFA-compliant mount point /u01 owned by grid:oinstall before installation.

An Oracle base for the grid installation owner /u01/app/grid owned by grid:oinstall with 775 permissions, and changed during the installation process to 755 permissions.

An Oracle base /u01/app/oracle1 owned by oracle1:oinstall with 775 permissions.

An Oracle base /u01/app/oracle 2 owned by oracle2:oinstall with 775 permissions.

A Grid home /u01/app/11.2.0/grid owned by grid:oinstall with 775 (drwxdrwxr-x) permissions. These permissions are required for installation, and are changed during the installation process to root:oinstall with 755 permissions (drwxr-xr-x).

/u01/app/oraInventory. This path remains owned by grid:oinstall, to enable other Oracle software owners to write to the central inventory.

Filed Under: Oracle Tagged With: ASM, asmadmin, GI, installation, os group, os user, software