環境
虛擬機VMware12:兩臺Oracle Linux R6 U3 x86_64
Oracle Database software: Oracle11gR2
Cluster software: Oracle grid infrastructure 11gR2
共享存儲:ASM
IP規劃:
Host Name | Type | address | desc |
tyrac1 | public | 192.168.26.77 | tyrac1 |
private | 192.168.65.77 | tyrac1-priv | |
vip | 192.168.26.78 | tyrac1-vip | |
tyrac2 | public | 192.168.26.88 | tyrac2 |
private | 192.168.65.88 | tyrac2-priv | |
vip | 192.168.26.89 | tyrac2-vip | |
scan | scan ip | 192.168.26.90 | scan-ip |
共享磁盤準備
這裏利用 vmware 實現共享磁盤的效果
打開 cmd ,cd 到 wmware 的安裝目錄,執行如下命令:
vmware-vdiskmanager.exe -c -s 5000Mb -a lsilogic -t 2 "D:\VM_OS\share_disk"\tyrac1.vmdk
vmware-vdiskmanager.exe -c -s 5000Mb -a lsilogic -t 2 "D:\VM_OS\share_disk"\tyrac2.vmdk
vmware-vdiskmanager.exe -c -s 5000Mb -a lsilogic -t 2 "D:\VM_OS\share_disk"\tyrac3.vmdk
2臺虛擬機關機,添加硬盤
這裏選擇 ” 使用現有虛擬磁盤 "
這裏輸入的路徑就是之前創建好的 3 塊盤
選擇保持現有格式。
然後選中新加的磁盤,點擊高級
選擇 SCSI 1:1 ,不要和系統磁盤在一個通道上面
點擊 OK 即可完成。
依次將 3 塊盤都加上去即可。
然後編輯 vmx 文件,添加如下內容:
disk.locking="FALSE"
scsi1:1.SharedBus="Virtual"
scsi1:2.SharedBus="Virtual"
scsi1:3.SharedBus="Virtual"
至此完成一臺機器的操作,另外一臺操作相同。
開機,fdisk -l 可查看到新加的磁盤即可。
安裝需要的包
yum install -y openssh-clients compat-libcap1-1.10 libX11 glibc-devel libstdc++-devel ksh gcc gcc-c* sysstat compat-libstdc++-33 libaio libaio-devel unzip screen smartmontools
更新 host 文件
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
#public
192.168.26.77 rac1
192.168.26.88 rac2
#private
192.168.65.77 rac1-priv
192.168.65.88 rac2-priv
#vip
192.168.26.78 rac1-vip
192.168.26.89 rac2-vip
#scan
192.168.26.90 scan-ip
關閉SELinux
vi /etc/sysconfig/selinux
SELINUX=disabled
關閉防火牆
chkconfig --level 2345 iptables off
重啓虛擬機
創建必要的用戶、組和目錄,並授權
groupadd -g 1000 oinstall
groupadd -g 1020 asmadmin
groupadd -g 1021 asmdba
groupadd -g 1022 asmoper
groupadd -g 1031 dba
groupadd -g 1032 oper
useradd -u 1100 -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid
useradd -u 1101 -g oinstall -G dba,asmdba,oper oracle
passwd grid
passwd oracle
mkdir -p /u01/app/11.2.0/grid
mkdir -p /u01/app/grid
mkdir /u01/app/oracle
chown -R grid:oinstall /u01
chown oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/
修改grid及oracle用戶的資源限制
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
grid soft stack 10240
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
配置操作系統內核參數
net.ipv4.ip_local_port_range = 9000 65500
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 1928120320
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
sysctl -p
配置grid和oracle用戶環境變量
grid:
export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_SID=+ASM1 # RAC1
export ORACLE_SID=+ASM2 # RAC2
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/11.2.0/grid
export PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
oracle:
export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_SID=orcl1 # RAC1
export ORACLE_SID=orcl2 # RAC2
export ORACLE_UNQNAME=orcl
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1
export TNS_ADMIN=$ORACLE_HOME/network/admin
export PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
配置SSH等效性(需同時配置grid用戶和oracle用戶的等效性)
1)在節點一上以grid用戶身份生成用戶的公鑰和私鑰:
[grid@xzrac01 ~]$ mkdir ~/.ssh
[grid@xzrac01 ~]$ ssh-keygen -t rsa
[grid@xzrac01 ~]$ ssh-keygen -t dsa
2)在節點二上執行相同的操作,生成grid用戶的公鑰和私鑰:
[grid@xzrac02 ~]$ mkdir ~/.ssh
[grid@xzrac02 ~]$ ssh-keygen -t rsa
[grid@xzrac02 ~]$ ssh-keygen -t dsa
3)在節點一上使用grid用戶執行以下操作:
[grid@xzrac01 ~]$ cat ~/.ssh/id_rsa.pub >> ./.ssh/authorized_keys
[grid@xzrac01 ~]$ cat ~/.ssh/id_dsa.pub >> ./.ssh/authorized_keys
[grid@xzrac01 ~]$ ssh rac02 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[grid@xzrac01 ~]$ ssh rac02 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[grid@xzrac01 ~]$ scp ~/.ssh/authorized_keys xzrac02:~/.ssh/authorized_keys
4)在節點一上驗證SSH等效性,如未提示輸入密碼則SSH等效性建立成功:
[grid@xzrac01 ~]$ ssh xzrac01 date
[grid@xzrac01 ~]$ ssh xzrac02 date
[grid@xzrac01 ~]$ ssh xzrac01-priv date
[grid@xzrac01 ~]$ ssh xzrac02-priv date
5)在節點二上驗證SSH等效性,如未提示輸入密碼則SSH等效性建立成功:
[grid@xzrac02 ~]$ ssh xzrac01 date
[grid@xzrac02 ~]$ ssh xzrac02 date
[grid@xzrac02 ~]$ ssh xzrac01-priv date
[grid@xzrac02 ~]$ ssh xzrac02-priv date
Oracle 用戶操作相同
上傳 Oracle、Grid 的軟件,加壓及權限變更
chown -R oracle.oinstall database
chown -R grid.oinstall grid
安裝 cvuqdisk 程序包
在兩個 Oracle RAC 節點上安裝操作系統程序包 cvuqdisk。如果沒有 cvuqdisk,集羣驗證實用程序就無法發現共享磁盤,當運行(手動運行或在 Oracle Grid Infrastructure 安裝結束時自動運行)集羣驗證實用程序時,您會收到這樣的錯誤消息:“Package cvuqdisk not installed”。使用適用於您的硬件體系結構(例如,x86_64 或 i386)的 cvuqdisk RPM。cvuqdisk RPM 包含在 Oracle Grid Infrastructure 安裝介質上的 rpm 目錄中。
cd /soft/grid/rpm
rpm -ivh cvuqdisk-1.0.9-1.rpm
配置裸設備
將之前的添加的 3 塊盤分區
fdisk /dev/sdb
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
最後 w 命令保存更改
重複步驟,格式化其他盤,得到如下分區
ls /dev/sd*
/dev/sda /dev/sda2 /dev/sdb1 /dev/sdc1 /dev/sdd1
/dev/sda1 /dev/sdb /dev/sdc /dev/sdd
vi /etc/udev/rules.d/60-raw.rules
添加如下內容:
ACTION=="add",KERNEL=="/dev/sdb1",RUN+='/bin/raw /dev/raw/raw1 %N"
ACTION=="add",ENV{MAJOR}=="8",ENV{MINOR}=="17",RUN+="/bin/raw /dev/raw/raw1 %M %m"
ACTION=="add",KERNEL=="/dev/sdc1",RUN+='/bin/raw /dev/raw/raw2 %N"
ACTION=="add",ENV{MAJOR}=="8",ENV{MINOR}=="33",RUN+="/bin/raw /dev/raw/raw2 %M %m"
ACTION=="add",KERNEL=="/dev/sdd1",RUN+='/bin/raw /dev/raw/raw3 %N"
ACTION=="add",ENV{MAJOR}=="8",ENV{MINOR}=="49",RUN+="/bin/raw /dev/raw/raw3 %M %m"
KERNEL=="raw[1-3]",OWNER="grid",GROUP="asmadmin",MODE="660"
[root@tyrac1 rpm]# start_udev
Starting udev: [ OK ]
[root@tyrac1 rpm]# ll /dev/raw
total 0
crw-rw---- 1 grid asmadmin 162, 1 May 25 16:24 raw1
crw-rw---- 1 grid asmadmin 162, 2 May 25 16:24 raw2
crw-rw---- 1 grid asmadmin 162, 3 May 25 16:24 raw3
crw-rw---- 1 root disk 162, 0 May 25 16:24 rawctl
手動運行cvu使用驗證程序驗證Oracle集羣件要求(所有節點都執行)
grid 用戶
cd /soft/grid/
./runcluvfy.sh stage -pre crsinst -n tyrac1,tyrac2 -fixup -verbose
最後出現
Pre-check for cluster services setup was successful.
及成功
安裝Grid Infrastructure
cd /soft/grid
./runInstall
這裏默認就 1 個節點,自己手動添加第 2 個節點
如果密碼比較簡單會提示警告,點 ok 即可
這個理有個警告,忽略即可
在 節點 1 和 2 的 root 用戶下分別執行以上腳本
[root@tyrac1 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@tyrac2 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@tyrac1 ~]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'tyrac1'
CRS-2676: Start of 'ora.mdnsd' on 'tyrac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'tyrac1'
CRS-2676: Start of 'ora.gpnpd' on 'tyrac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'tyrac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'tyrac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'tyrac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'tyrac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'tyrac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'tyrac1'
CRS-2676: Start of 'ora.diskmon' on 'tyrac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'tyrac1' succeeded
ASM created and started successfully.
Disk Group OCR created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 854d0a813dd14f34bf091b34629b30c7.
Successfully replaced voting disk group with +OCR.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 854d0a813dd14f34bf091b34629b30c7 (/dev/raw/raw1) [OCR]
Located 1 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'tyrac1'
CRS-2676: Start of 'ora.asm' on 'tyrac1' succeeded
CRS-2672: Attempting to start 'ora.OCR.dg' on 'tyrac1'
CRS-2676: Start of 'ora.OCR.dg' on 'tyrac1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@tyrac2 ~]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node tyrac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
是因爲沒有配置resolve.conf,可以忽略
檢查集羣狀態
[grid@tyrac1 grid]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[grid@tyrac1 grid]$ crsctl status res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE tyrac1
ONLINE ONLINE tyrac2
ora.OCR.dg
ONLINE ONLINE tyrac1
ONLINE ONLINE tyrac2
ora.asm
ONLINE ONLINE tyrac1 Started
ONLINE ONLINE tyrac2 Started
ora.gsd
OFFLINE OFFLINE tyrac1
OFFLINE OFFLINE tyrac2
ora.net1.network
ONLINE ONLINE tyrac1
ONLINE ONLINE tyrac2
ora.ons
ONLINE ONLINE tyrac1
ONLINE ONLINE tyrac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE tyrac1
ora.cvu
1 ONLINE ONLINE tyrac1
ora.oc4j
1 ONLINE ONLINE tyrac1
ora.scan1.vip
1 ONLINE ONLINE tyrac1
ora.tyrac1.vip
1 ONLINE ONLINE tyrac1
ora.tyrac2.vip
1 ONLINE ONLINE tyrac2
爲數據和快速恢復去創建ASM磁盤組
su - grid
asmca
點create,創建 DATA、FRA 磁盤組
安裝Oracle database軟件(RAC)
su - oracle
cd /soft/database
./runInstall
這個錯誤可忽略
以 root 用戶 分別執行該腳本後點 OK 即可
創建集羣數據庫
su - oracle
dbca
這裏要選中全部節點
等待完成即可