Question:
1.11g Oracle Clusterware需要的OCR和Voting disk可以存储在ASM或者集群文件系统或者NFS中。对于全新安装，裸设备不再被支持(是否有办法使用裸设备？)。
2.使用ASM时，若相关存储上的磁盘路径(disk path)名前后不一致，是否仍然可以使用？需要什么调整？
Answer:
1.在11gr2 Grid Infrastructure全新安装时是没有办法使用裸设备的(You cannot install OCR or voting disk files on raw partitions. You can install only on Oracle ASM, or on supported network-attached storage or cluster file systems. The only use for raw devices is as Oracle ASM disks.);但可以通过后续的手段将OCR和VOTING DISK移动到裸设备上，如:
替换OCR: ocrconfig -add rawdevice ocrconfig -replace 替换voting disk crsctl add votedisk css -force crsctl delete votedisk css -force
具体可以参考Metalink文档<How to ADD/REMOVE/REPLACE/MOVE Oracle Cluster Registry (OCR) and Voting Disk>
实际上强烈不建议这样做。因为如果出现问题,Oracle GCS可以拒绝提供建议。
2.ASM是通过读取磁盘头部来了解磁盘内容的；磁盘路径名在安装时需要在所有节点一致，在安装完成后即便路径名改变也不会影响到ASM的使用。
需要注意的是在AIX操作平台上分配给ASM的磁盘(ASM DISK)，如果直接是HDISK形式的LUN则该HDISK不应当具有PVID(If the disk device has a PVID, then ASM will fail to mount the diskgroup created on the disk device.)。如果是裸的逻辑卷，那么所建VG应当是scaleable volume group。
How to ADD/REMOVE/REPLACE/MOVE Oracle Cluster Registry (OCR) and Voting Disk
Purpose
To provide step to add, remove, replace or move an OCR or voting disk in Oracle Real Application Clusters (RAC) 10g Realese 2 (10.2.0.1 and later) environments.
SCOPE & APPLICATION
This article is intended for DBAs and Support Engineers who need to modify, or move OCR and voting disk locations. Customers, who have an existing clustered environment deployed on a storage array, might want to migrate to a new storage array, with minimal downtime. Typically, one would simply cp or dd the files once the new storage has been presented to the hosts. In this case, it is a little more difficult because,
1. The clusterware has the OCR and voting disks open and is actively using them. (Both the primary and the mirrors)
2. There is a “cluster api” provided for this function (ocrconfig, and crsctl), which is a better interface than typical cp and/or dd commands.
It is highly recommended to take a backup of the voting disk, and OCR device before making any changes.
Examples are included at the end of OCR and voting disk command.
ADD/REMOVE/REPLACE OCR Device
Note: You must be logged in as the root user, because root own the OCR files.
Make sure there is a recent copy of the OCR file before making any changes:
ocrconfig –showbackup
If there is not a recent backup copy of the OCR file, an export can be taken for the current OCR file. Use the following command to generate a an export of the online OCR file:
ocrconfig –export -s online
If you need to recover using this file, the following command can be used:
ocrconfig import
1. To add an OCR device:
To add an OCR device, provide the full path including file name.
ocrconfig -replace ocr
To add an OCR mirror device, provide the full path including file name.
ocrconfig -replace ocrmirror
2. To remove an OCR device:
To remove an OCR device:
ocrconfig -replace ocr
To remove an OCR mirror device
ocrconfig -replace ocrmirror
3. To replace or move the location of an OCR device:
To replace the OCR device with , provide the full path including file name.
ocrconfig -replace ocr
To replace the OCR mirror device with , provide the full path including file name.
ocrconfig -replace ocrmirror
Example moving OCR file from OCFS to raw devices
The OCR disk must be owned by root, must be in the oinstall group, and must have permissions set to 640. Provide at least 100 MB disk space for the OCR.
In this example the OCR file are located in the ocfs2 file system:
/ocfs2/ocr1
/ocfs2/ocr2
Create raw device files of at least 100 MB. In this example the new OCR file will be on the following devices:
/dev/raw/raw1
/dev/raw/raw2
Once the raw devices are created, use the dd command to zero out the device and make sure no data is written to the raw devices:
dd if=/dev/zero of=/dev/raw/raw1
dd if=/dev/zero of=/dev/raw/raw2
Note: Use UNIX man pages for additional information on the dd command.
Now you are ready to move/replace the OCR file to the new storage location.
Move/Replace the OCR device
ocrconfig -replace ocr /dev/raw/raw1
Add /dev/raw/raw2 as OCR mirror device
ocrconfig -replace ocr /dev/raw/raw2
Example of adding an OCR device file
If you have upgraded your environment from a previous version, where you only had 1 OCR device file, you can use the following step to add an additional OCR file.
In this example a second OCR device file is added:
Add /dev/raw/raw2 as OCR mirror device
ocrconfig -replace ocr /dev/raw/raw2
ADD/DELETE/MOVE Voting Disk
Note: Shutdown the Oracle Clusterware (crsctl stop crs as root) on all nodes before making any modification to the voting disk. Determine the current voting disk location using:
crsctl query css votedisk
Take a backup of all voting disk:
dd if=voting_disk_name of=backup_file_name
Note: Use UNIX man pages for additional information on the dd command.
The following can be used to restore the voting disk from the backup file created.
dd if=backup_file_name of=voting_disk_name
1. To add a Voting Disk, provide the full path including file name.:
crsctl add votedisk css -force
2. To delete a Voting Disk, provide the full path including file name.:
crsctl delete votedisk css -force
3. To move a Voting Disk, provide the full path including file name.:
crsctl delete votedisk css –force
crsctl add votedisk css –force
After modifying the voting disk, start the Oracle Clusterware stack on all nodes
crsctl start crs
Verify the voting disk location using
crsctl query css votedisk
Example moving Voting Disk from OCFS to raw devices
The voting disk is a partition that Oracle Clusterware uses to verify cluster
node membership and status.
The voting disk must be owned by the oracle user, must be in the dba
group, and must have permissions set to 644. Provide at least 20 MB disk
space for the voting disk.
In this example the Voting Disks are located in the ocfs2 file system:
/ocfs2/voting1
/ocfs2/voting2
/ocfs2/voting3
Create raw device files of at least 20 MB. In this example the new voting disks will be on the following devices:
/dev/raw/raw3
/dev/raw/raw4
/dev/raw/raw5
Once the raw devices are created, use the dd command to zero out the device and make sure no data is written to the raw devices:
dd if=/dev/zero of=/dev/raw/raw3
dd if=/dev/zero of=/dev/raw/raw4
dd if=/dev/zero of=/dev/raw/raw5
Note: Use UNIX man pages for additional information on the dd command.
Now you are ready to move/replace the voting disks to the new storage location.
To move a Voting Disk to new storage location:
crsctl delete votedisk css /ocfs2/voting1 –force
crsctl add votedisk css /dev/raw/raw3 –force
crsctl delete votedisk css /ocfs2/voting2 –force
crsctl add votedisk css /dev/raw/raw4 –force
crsctl delete votedisk css /ocfs2/voting3 –force
crsctl add votedisk css /dev/raw/raw5 –force
Example of adding Voting Disks
If you have upgraded your environment from a previous version, where you only had 1 voting disk, you can use the following steps to add additional voting disk.
In this example 2 additional Voting Disks are added:
crsctl add votedisk css /dev/raw/raw4 –force
crsctl add votedisk css /dev/raw/raw5 –force
After modifying the voting disk, start the Oracle Clusterware stack on all nodes
crsctl start crs
Verify the voting disk location using
crsctl query css votedisk
good.
btw, Failed to Start CRS stack After Adding a New Voting Disk [ID 460874.1]