RHEL 7.4装Oracle 11.2.0.4 RAC的一些问题

cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)

一、要安装compat-libcap1.x86_64包：

yum install compat-libcap1.x86_64

否则root.sh会因为缺少库文件而报错

clscfg.bin: error while loading shared libraries: libcap.so.1: cannot open shared object file: No such file or directory Failed to create keys in the OLR, rc = 127, Message:

Failed to write the checkpoint:” with status:FAIL.Error code is 256

二、 在执行root.sh前要给GRID_HOME打补丁 ，补丁号18370031 否则因为RHEL 7修改了一些RC脚本会导致安装不上

报错如下：

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME=  /u01/app/11.2.0/grid
 
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of “dbhome” have not changed. No need to overwrite.
The contents of “oraenv” have not changed. No need to overwrite.
The contents of “coraenv” have not changed. No need to overwrite.
 
Creating /etc/oratab file…
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization – successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding Clusterware entries to inittab
ohasd failed to start
Failed to start the Clusterware. Last 20 lines of the alert log follow:
2018-03-08 10:20:24.544:
[client(17856)]CRS-2101:The OLR was formatted using version 3.

因为补丁18370031(Patch 18370031: RC SCRIPTS (/ETC/RC.D/RC.* , /ETC/INIT.D/* ) ON OL7 FOR CLUSTERWARE)   被 20433339( Patch 24333766: MERGE REQUEST ON TOP OF 11.2.0.4.0 FOR BUGS 18370031 20954311)
包含了， 所以目前应该只能下载到20433339了。

opatch napply -local 20433339



opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.23
Copyright (c) 2020, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.23
OUI version       : 11.2.0.4.0
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2020-03-06_09-37-02AM_1.log

Lsinventory Output file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2020-03-06_09-37-02AM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: rac1
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1): 

Oracle Grid Infrastructure 11g                                       11.2.0.4.0
There are 1 products installed in this Oracle Home.


Interim patches (1) :

Patch  24333766     : applied on Fri Mar 06 08:15:04 EST 2020
Unique Patch ID:  20433339
Patch description:  "OCW Interim patch for 24333766"
   Created on 30 Nov 2016, 12:56:34 hrs PST8PDT
   Bugs fixed:
     18370031, 20954311



--------------------------------------------------------------------------------

OPatch succeeded.

但是注意要打以上补丁先要将Opatch升级到最新，所以要安装前先去support.oracle.com下载2个包p6880880_112000_Linux-x86-64.zip和p24333766_112040_Linux-x86-64.zip。

三、安装fuser 命令

yum install psmisc

四、对于RHEL 7下的udev配置可以看这篇文章 https://gruffdba.wordpress.com/2017/02/20/udev-rules-for-asm-disks-on-rhel7/

这里顺便贴出来：

On this blog and elsewhere you will find UDEV rules examples for setting device ownership and naming consistency on older versions of Linux.

With RHEL7 some of the syntax has changed slightly.

This example was created using OEL7 with the Red Hat kernel, but should also work on Red Hat and CentOS.


First, log in as root and check the block device is visible on the Linux host:

[root@unirac02 ~]# ls /dev/sd*
/dev/sda /dev/sda1 /dev/sda2 /dev/sdb /dev/sdb1
In this example I have created a device sdb, and as you can see I have created a partition header on it.

Next, make sure we can see the device’s SCSI ID:

[root@unirac02 ~]# /lib/udev/scsi_id -g -u /dev/sdb
36006016004503e0017f99d58603c7c1e
Next, we are going to create a UDEV rule for this SCSI ID in the file /etc/udev/rules.d/99-oracleasm.rules.

[root@unirac01 ~]# cat /etc/udev/rules.d/99-oracleasm.rules
KERNEL=="sd?", ENV{ID_SERIAL}=="36006016004503e0017f99d58603c7c1e", SYMLINK+="oracleasm/grid1", OWNER="oracle", GROUP="oinstall", MODE="0660"
If you have several devices to add, you can use the following script to automate the rule generation.

[root@unirac02 ~]# mydevs="sdb sdc sdd" ; export count=0 ; for mydev in $mydevs; do ((count+=1)) ; /lib/udev/scsi_id -g -u /dev/$mydev | awk '{print "KERNEL==\"sd?\", ENV{ID_SERIAL}==\""$1"\", SYMLINK+=\"oracleasm/disk"ENVIRON["count"]"\", OWNER=\"oracle\", GROUP=\"oinstall\", MODE=\"0660\""}' ; done
KERNEL=="sd?", ENV{ID_SERIAL}=="36006016004503e0017f99d58603c7c1e", SYMLINK+="oracleasm/disk1", OWNER="oracle", GROUP="oinstall", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="36006016004503e0017f99d58603d1a87", SYMLINK+="oracleasm/disk2", OWNER="oracle", GROUP="oinstall", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="36006016004503e0017f99d58603d246a", SYMLINK+="oracleasm/disk3", OWNER="oracle", GROUP="oinstall", MODE="0660"
With RHEL7, restarting the UDEV rules is slightly difference than previous releases:

[root@unirac02 ~]# /sbin/udevadm control --reload-rules
[root@unirac02 ~]# /sbin/udevadm trigger
Now check, and a new device should be visible under /dev/oracleasm

[root@unirac02 ~]# ls -al /dev/oracleasm/*
lrwxrwxrwx. 1 root root 6 Feb 20 18:38 /dev/oracleasm/grid1 -> ../sdb

一些相关文档：

Requirements for Installing Oracle 11.2.0.4 RDBMS on OL7 or RHEL7 64-bit (x86-64) (Doc ID 1962100.1)

APPLIES TO:

Oracle Database – Standard Edition – Version 11.2.0.4 to 11.2.0.4 [Release 11.2]
Oracle Database – Enterprise Edition – Version 11.2.0.4 to 11.2.0.4 [Release 11.2]
Oracle Database Cloud Schema Service – Version N/A and later
Oracle Database Exadata Cloud Machine – Version N/A and later
Oracle Cloud Infrastructure – Database Service – Version N/A and later
Linux x86-64

PURPOSE

This note explains the requirements that need to be met for a successful installation of Oracle 11gR2 RDBMS release 11.2.0.4 on Red Hat Enterprise Linux 7.0 (or higher 7.x version), 64-bit (x86-64).  These guidelines apply to cluster (RAC) or standalone / single instances.

It is NOT the purpose of this NOTE to repeat every “how-to” step that is presented in the 11gR2 Installation Guide manual. For example this NOTE does not include how to create the Linux OS account named “oracle”, nor does it cover how to set environment variables. Both are adequately covered in Chapter 2 “Oracle Database Pre-installation Requirements” of the 11gR2 Installation Guide manual.

You can download Oracle 11.2.0.4 software from My Oracle Support (patch 13390677)

SCOPE

This procedure is meant for those planning/installing Oracle 11gR2 RDBMS release 11.2.0.4.0 (or higher 11.2.0.x version) on Red Hat Enterprise Linux 7.0 (or higher 7.x version) on the 64-bit (x86-64) platform. Since it is the expressed goal to keep Oracle Linux (OL) functionally IDENTICAL to RHEL, this NOTE is also completely applicable to 64-bit (x86-64) OL 7.0 (or higher 7.x version).

This procedure is not meant for those planning/installing Grid Infrastructure (GI) or any other Oracle products.

DETAILS

Requirements for installing Oracle 11gR2 RDBMS release 11.2.0.4 64-bit on RHEL7 or OL7 64-bit (x86_64)

I. Hardware:
===========
1. Minimum Hardware Requirements
a.) At least 1.0 GB (1024MB) of physical RAM
b.) Swap disk space proportional to the system’s physical memory as follows:

NOTE: The above recommendations (from the 11.2 Database installation guide) are MINIMUM recommendations for installations. Further RAM and swap space may be required to tune/improve RDBMS performance.

c.) 1.0 GB (1024MB) of disk space (and less than 2TB of disk space) in the /tmp directory.
d.) approximately 4.4 GB of local disk space for the database software.
e.) approximately 1.7 GB of disk space for a preconfigured database that uses file system storage (optional).

2. Refer Note:236826.1 for details on certified filesystems for Oracle Database.

II. Software:
============
1. As is specified in section 1.3.2 of the Oracle Database Installation Guide for 11gR2 on Linux (part number E24321-02), Oracle recommends that you install the Linux operating system with the default software packages (RPMs) and do not customize the RPMs during installation. For additional information on “default-RPMs”, please see Note 376183.1, “Defining a “default RPMs” installation of the RHEL OS” or Note 401167.1, “Defining a “default RPMs” installation of the Oracle Enterprise Linux (OEL) OS”.

2.Linux Kernel Requirements

Oracle Linux 7.0 

• Oracle Linux 7 with Unbreakable Enterprise Kernel : 3.8.13-33.el7uek.x86_64 or later
• Oracle Linux 7 with the Red Hat Compatible kernel : 3.10.0-54.0.1.el7.x86_64 or later
  

Red Hat Enterprise Linux Server 7.0

• Red Hat Enterprise Linux 7 : 3.10.0-54.0.1.el7.x86_64 or later

NOTE:

• RHEL7 servers must be running Red Hat kernel 3.10.0-54.0.1.el7 (x86_64) or higher or 3.8.13-33.el7uek (x86_64) or higher with UEK kernel. OL7 servers must also be running kernel 3.8.13-33.el7uek (x86_64) or higher version. The product RHEL does not deliver UEK Kernel. Only in OL 7 UEK and RHCK Kernel is included.
• It is observed there are hang issues in RHEL 7 with many CPU cores and more RAM, due NUMA was enabled. As a work around it is recommended to turn off NUMA.

3. Required OS Components (per Release Notes, and Install Guide)

a.) The exact version number details of this list are based upon 64-bit (x86_64) RHEL 7.0. When a higher “update” level is used, the RPM release numbers (such as 4.4.4-13) may be slightly different. Since updates of RHEL 7 are certified, this is fine so long as you are still using 64-bit Linux (x86_64) RHEL 7 RPMs.
b.) Some of the Install Guide requirements will already be present from the “default-RPMs” foundation of Linux that you started with:

4. Additional Required OS Components (per the runInstaller OUI)
a.) intentionally blank

5. Additional Required OS Components (per this NOTE)
a.) Please do not rush, skip, or minimize this critical step. This list is based upon a “default-RPMs” installation of 64-bit (x86_64) RHEL 6. Additional RPMs (beyond anything known to Oracle) may be needed if a “less-than-default-RPMs” installation of 64-bit (x86_64) RHEL Server 6 is performed. For more information, please refer to Note 376183.1, “Defining a “default RPMs” installation of the RHEL OS” or Note 401167.1, “Defining a “default RPMs” installation of the Oracle Enterprise Linux (OEL) OS”.
b.) Several RPMs will be required as prerequisites to those listed in section II.3.c:  

 6. Oracle Global Customer Support has noticed a recent trend with install problems that originates from installing too many RPMs. For example:
a.) installing your own JDK version (prior to execute the Oracle Software runInstaller) is not needed on Linux, and is not recommended on Linux. A pre-existing JDK often interferes with the correct JDK that the Linux Oracle Software runInstaller will place and use.
b.) installing more than the required version of the gcc / g++ RPMs often leads to accidentally using (aka enabling or activating) the incorrect one. If you have multiple RDBMS versions installed on the same Linux machine, then you will likely have to manage multiple versions of gcc /g++ . For more information, please see Note 444084.1, “Multiple gcc / g++ Versions in Linux”

7. All of the RPMs in section II. are on the Red Hat Enterprise Linux 7 64-bit (x86_64) distribution media.

III. Environment:
================
1. Modify your kernel settings in /etc/sysctl.conf (RedHat) as follows. If the current value for any parameter is higher than the value listed in this table, do not change the value of that parameter. Range values (such as net.ipv4.ip_local_port_range) must match exactly. 

2. To activate these new settings into the running kernel space, run the “sysctl -p” command as root.

3. Set Shell Limits for the oracle User. Assuming that the “oracle” Unix user will perform the installation, do the following:

a.) Add the following settings to /etc/security/limits.conf

b.) Verify the latest version of PAM is loaded, then add or edit the following line in the /etc/pam.d/login file, if it does not already exist: 

c.) Verify the current ulimits, and raise if needed.  This can be done many ways…adding the following lines to /etc/profile is the recommended method: 

4. The gcc-4.4.4 and gcc-c++-4.4.4 RPM items above will ensure that the correct gcc / g++ versions are installed. It is also required that you ensure that these correct gcc / g++ versions are active, and in-use. Ensure that the commands “gcc –version” and “g++ –version” each return “4.8.2”.

5. If any Java packages are installed on the system, unset the Java environment variables, for example JAVA_HOME.

6. The oracle account that is used to install Oracle 11.2.0.4 should not have the Oracle install related variables set by default. For example setting ORACLE_HOME, PATH, LD_LIBRARY_PATH to include Oracle binaries in .profile, .login file and /etc/profile.d should be completely avoided.
a.) Setting $ORACLE_BASE (not $ORACLE_HOME) is recommended, since it eases a few prompts in the OUI runInstaller tool.
b.) Following the successful install, it is recommended to set $ORACLE_HOME, and to set $PATH to include $ORACLE_HOME/bin at the beginning of the $PATH string.

7. By default, RHEL 7 x86_64 Linux is installed with SELinux as “enforcing”. This is fine for the 11gR2 installation process. However, to subsequently run “sqlplus”, switch SELinux to the “Permissive” mode. See NOTE 454196.1, “./sqlplus: error on libnnz11.so: cannot restore segment prot after reloc” for more details.

UPDATE: Internal testing suggests that there is no problem running “sqlplus” with SELinux in “enforcing” mode on RHEL7/OL7. The problem only affects RHEL5/OL5.

8. Log in as Oracle user and start the installation as follows: 

a.) It is best practice not to use any form of “su” to start the runInstaller, in order to avoid potential display-related problems.
b.) When performing the 11.2.0.4 installation, make sure to use the “runInstaller” version that comes with 11.2.0.4 software.
c.) When performing any subsequent 11.2.0.x patchset, make sure to use the “runInstaller” version that comes with the patchset.

Known Issue :

01) The installer needs to be launched with “-ignorePrereq” option due to unpublished bug 19947777. This issue occurs since Oracle Linux 7 was not released when Oracle database 11.2.0.4 was made available and hence was not certified. However, Oracle 11.2.0.4 is now certified on OL7. Refer Note 1962046.1 for details.

02) Compilation fails for target ‘relink_exe’ fails with “undefined reference to symbol ‘B_DestroyKeyObject’” error and is reported in unpublished bug 19692824. The solution is to install patch 19692824 as documented in Note 1965691.1.

ADDITIONAL NOTES
—————-
1. Supported distributions of the 32-bit (x86) Linux OS can run on on AMD64/EM64T and Intel Processor Chips that adhere to the x86_64 architecture
a.) Oracle 32-bit Database Server running on AMD64/EM64T with 32-bit OS is supported, but is NOT covered by this NOTE.
b.) Oracle 32-bit Database Server running on AMD64/EM64T with 64-bit OS is not certified and is not supported.
c.) Oracle 32-bit Database Client running on AMD64/EM64T with 64-bit OS is expected to be supported, but is NOT covered by this NOTE.

2. Asynchronous I/O on ext2 and ext3 file systems is supported if your scsi/fc driver supports that functionality. 

3. No extra patch is required for the DIRECTIO support for x86_64.

4. No LD_ASSUME_KERNEL value should be used with the 11gR2 product.

5. The following rpm command can be used to distinguish between a 32-bit or 64-bit package.   

# cat /etc/yum.repos.d/public-yum-ol7.repo
[ol7_latest]
name=Oracle Linux $releasever Latest ($basearch)
baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1

[ol7_addons]
name=Oracle Linux $releasever Add ons ($basearch)
baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL7/addons/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1

yum install device-mapper-multipath

[root@xxxx ~]# systemctl start multipathd
[root@xxxx ~]# systemctl status multipathd


SELinux
Disable, e.g.

setenforce 0