【讓學習oracle前進一步】
一、實驗環境
1.1、IP地址規劃
網卡名 | 節點名 | IP地址 | 網卡類型 | 別名 | |
Eth0 | ctpdb1 | 172.17.80.11 | private | ctpdb1-priv | vm2 |
Eth1 | ctpdb1 | 192.168.1.11 | public | ctpdb1 | Host-only |
無 | ctpdb1 | 192.168.1.111 | virtual | ctpdb1-vip | |
無 | ctpdb1 | 192.168.1.10 | 無 | rac-cluster | |
Eth0 | ctpdb2 | 172.17.80.12 | private | ctpdb2-priv | vm2 |
Eth1 | ctpdb2 | 192.168.1.12 | public | ctpdb2 | Host-only |
無 | ctpdb2 | 192.168.1.112 | virtual | ctpdb2-vip | |
#------------------------------------------------
1.2、主機信息
節點1：/etc/hosts
192.168.1.11ctpdb1
192.168.1.111 ctpdb1-vip
172.17.80.11 ctpdb1-priv
192.168.1.12ctpdb2
192.168.1.112 ctpdb2-vip
172.17.80.12 ctpdb2-priv
192.168.1.10 ctpdb-cluster
#------------------------------------------------
節點2：/etc/hosts
192.168.1.11ctpdb1
192.168.1.111 ctpdb1-vip
172.17.80.11 ctpdb1-priv
192.168.1.12ctpdb2
192.168.1.112 ctpdb2-vip
172.17.80.12 ctpdb2-priv
192.168.1.10 ctpdb-cluster
#------------------------------------------------
1.3、基礎信息
systemOS：rhel 5.8x64kernel:2.6.18-308
media: p10404530_112030_Linux-x86-64_1of7.zip(oracle)
p10404530_112030_Linux-x86-64_2of7.zip(oracle)
p10404530_112030_Linux-x86-64_3of7.zip(grid)
version:Oracle 11.2.0.3
asm:oracleasm-2.6.18-308.el5-2.0.5-1.el5.x86_64.rpm
oracleasmlib-2.0.4-1.el5.x86_64.rpm
oracleasm-support-2.1.7-1.el5.x86_64.rpm
VM:VMware Workstation9.0
1.4、分區規劃
1.4.1 節點1複製到節點2，來實現操作系統一致性。
1.4.2 共享磁盤：通過手動修改ctpdb1.vmx來實現。
1.4.3 磁盤信息
RAC1 hostname ctpdb1 共40G,mem 2G,SWAP4G, boot 100M ,/剩餘
RAC2 hostname ctpdb2 共40G,mem 2G,SWAP4G ,boot 100M, /剩餘
-----------------------------------------------------
二、安裝虛擬機操作系統、創建共享磁盤
2.1 使用wmwareworkstation 9
2.2選擇安裝linux
節點1、節點2分別存放F盤下vm_ctpdb1/vm_ctpdb2
2.3磁盤格式，選擇SCSI
2.4添加虛擬機共享硬盤，分區如下
2.4.1添加1/3OCR磁盤，大小爲：1G
2.4.2修改磁盤節點名稱爲：ocr1.vmdk，節點爲：SCSI1:0
2.5注：以上述方式分別添加和修改如下磁盤：
2.5.1添加2/3OCRGROUP,大小爲:1G,名稱爲:ocr2.vmdk, 節點爲:SCSI1:1;
2.5.2添加3/3 OCRGROUP,大小爲:1G,名稱爲:ocr3.vmdk, 節點爲:SCSI1:2;
2.5.3添加1/3DATAGROUP,大小爲:10G, 名稱爲:data1.vmdk, 節點爲:SCSI1:3;
2.5.4添加2/3DATAGROUP,大小爲:10G, 名稱爲:data2.vmdk,節點爲:SCSI1:4;
2.5.4添加3/3DATAGROUP,大小爲:10G, 名稱爲:data3.vmdk,節點爲:SCSI1:5;
2.5.5添加1/2ARCHGROUP,大小爲:5G, 名稱爲:arch1.vmdk,節點爲:SCSI1:6;
2.5.6添加2/2ARCHGROUP,大小爲:5G, 名稱爲:arch2.vmdk,節點爲:SCSI1:8;
注：SCSI1:7已經被佔用;
------------------------------------------------
2.6RMAN和DUMP備份數據存於/u02下EXT3文件系統下。
---------------------------------------------------------------------
2.7磁盤組介紹
CRSGROUP （集羣軟件）
VOTE1 1G scsi1:0
VOTE2 1G scsi1:1
VOTE3 1G scsi1:2
----------------------------
DATAGROUP （數據文件）
DATA1 10G scsi1:3
DATA2 10G scsi1:4
DATA3 10G scsi1:5
----------------------------
ARCHGROUP （歸檔日誌）
ARCH1 5G scsi1:6
ARCH2 5G scsi1:8 //1:7被佔用
------------------------------------------
三、修改虛擬機操作系統基本參數
3.1在節點1安裝linux 操作系統，安裝完成後複製成節點2即可。
注：3.1.1操作系統怎麼裝，不在介紹。
3.1.2iptables和selinux 設置爲關閉狀態。
3.2在節點1刷掉不用的服務和無效的賬號，從而提高系統的性能和安全。
以root用戶執行以下命令：
chkconfig anacron off
chkconfig atd off
chkconfig autofs off
chkconfig cups off
chkconfig cups-config-daemon off
chkconfig firstboot off
chkconfig haldaemon off
chkconfig isdn off
chkconfig kudzu off
chkconfig netfs off
chkconfig nfslock off
chkconfig pcmcia off
chkconfig portmap off
chkconfig rpcgssd off
chkconfig rpcidmapd off
chkconfig sendmail off
chkconfig smartd off
chkconfig snmpd off
chkconfig avahi-daemon off
#--------------------------------------
userdel adm
userdel lp
userdel sync
userdel shutdown
userdel halt
userdel news
userdel uucp
userdel operator
userdel gopher
#--------------------------------------
3.3關閉NTP服務，使用oracle的ctssd服務器進行時間同步。
service ntpd stop
chkconfig ntpd off
mv /etc/ntp.conf /etc/ntp.conf.org
rm /var/run/ntpd.pid
--------------------------------------------
3.4開始在ctpdb1節點配置環境,以root用戶
3.4.1修改hosts文件
192.168.1.11ctpdb1
192.168.1.111 ctpdb1-vip
172.17.80.11 ctpdb1-priv
192.168.1.12ctpdb2
192.168.1.112 ctpdb2-vip
172.17.80.12 ctpdb2-priv
192.168.1.10 ctpdb-cluster
--------------------------------------------
3.4.2 vi /etc/security/limits.conf
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
------------------------------------------------------------
3.4.5 vi /etc/sysctl.conf
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 4294967295
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
sysctl -p 使其立即生效
------------------------------------------------------------
3.4.6 vi/etc/pam.d/login
session required /lib/security/pam_limits.so
5.5 vi /etc/profile
if [ $USER = "oracle" ]||[ $USER = "grid" ];then
if [ $SHELL = "/bin/ksh" ];then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
###################
---------------------------------------------------------------------四、創建用戶和用戶環境變量
4.1創建用戶
groupadd oinstall
groupadd dba
groupadd oper
groupadd asmadmin
groupadd asmdba
groupadd asmoper
useradd -g oinstall -G dba,asmdba,oper -d/home/oracle oracle
useradd -g oinstall -Gdba,asmadmin,asmdba,asmoper -d /home/grid grid
------------------------------------------------------------
4.2創建oracle使用到的目錄結構,並賦予相應的屬性。
mkdir -p /u01/app/11.2.0/grid
mkdir -p /u01/app/grid
mkdir -p/u01/app/oracle/product/11.2.0/db_1
mkdir -p /u02/rman
chown -R grid:oinstall /u01
chown -R oracle:oinstall /u01/app/oracle
chown -R oracle:oinstall /u02
chmod -R 775 /u01
chmod -R 775 /u02
------------------------------------------------------------
4.3創建oracle用戶和grid用的環境變量,等節點2複製完後，需要做相應更改。
注：SID環境
oracle RAC SID：
ctpdb1
ctpdb2
集羣實例名：
ctpdb
-------------------------------------------------------------
4.4 root用戶執行
su - oracle
vi .bash_profile
export ORACLE_HOSTNAME= ctpdb1
ORACLE_SID= ctpdb1;export ORACLE_SID
ORACLE_BASE=/u01/app/oracle;exportORACLE_BASE
ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1;exportORACLE_HOME
ORACLE_TERM=xterm;export ORACLE_TERM
NLS_DATA_FORMAT="DD-MON-YYYYHH24:MI:SS";export NLS_DATA_FORMAT
TNS_ADMIN=$ORACLE_HOME/network/admin;exportTNS_ADMIN
ORA_NLS11=$ORACLE_HOME/nls/data;exportORA_NLS11
PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:usr/local/bin
PATH=${PATH}:/u01/app/common/oracle/bin
export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH
THREADS_FLAG=native;export THREADS_FLAG
export TEMP=/tmp
export TMPDIR=/tmp
umask 022
--------------------------------------------------------------------
4.5 root用戶登錄
su - grid
vi .bash_profile
export ORACLE_HOSTNAME= ctpdb1
ORACLE_SID=+ASM1;export ORACLE_SID
ORACLE_BASE=/u01/app/grid;exportORACLE_BASE
ORACLE_HOME=/u01/app/11.2.0/grid;exportORACLE_HOME
ORACLE_TERM=xterm;export ORACLE_TERM
NLS_DATA_FORMAT="DD-MON-YYYYHH24:MI:SS";export NLS_DATA_FORMAT
PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:usr/local/bin
PATH=${PATH}:/u01/app/common/oracle/bin
export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH
export CLASSPATH
THREADS_FLAG=native;export THREADS_FLAG
export TEMP=/tmp
export TMPDIR=/tmp
umask 022
---------------------------------------------------------------------
4.6root用戶檢查rpm包是否符合要求（可通過get_rpm.sh腳本掃描,見附件）。
注：若不符合，則由系統安裝iso文件，找到該rpm包，
執行：rpm -ivh 安裝即可。如：
mkdir /cdrom
mount /dev/cdrom /cdrom
cd /cdrom/Server
rpm -ivh libaio-devel-0.3.106-5.*
rpm -ivh libstdc++-devel-4.1.2-52.el5.*
rpm -ivh sysstat-7.0.2-11.el5.x86_64.rpm
---------------------------------------------------------------------
五、複製虛擬機，通過修改虛擬機文件是共享磁盤生效
注：【重點】如果此項修改失敗或有問題，將導致整個環境無法完成。
5.1複製節點1到新的節點2。
注：複製後，將節點1臨時重命名，避免複製的節點2會讀取節點1數據。
5.2文本編輯節點1下的ctpdb1.vmx文件改共享磁盤組
shutdown -h now 關掉節點1
新增：
注：修改前備份ctpdb1.vmx文件。
disk.locking = "false"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize ="0"
diskLib.DataCacheMinReadAheadSize ="0"
diskLib.dataCachePageSize ="4096"
diskLib.maxUnsyncedWrites = "0"
scsi1.sharedBus = "virtual"
scsi2.sharedBus = "virtual"
scsi0:0.mode ="independent-persistent"
----------------------------------------------------------------
5.3在複製後的 ctpdb2目錄內，將所有ctpdb1的文件名改爲 ctpdb2,並將目錄改爲vm_ctpdb2,
注：否則ctpdb 2會找ctpdb 1的文件運行虛擬機。
5.4修改節點2的ctpdb2.vmx虛擬機配置文件。
5.4.1將ctpdb2.vmx內，所有ctpdb1字符串改爲ctpdb2。
5.5編輯網卡參數。（具體參考附件）
5.5.1將:ethernet0.generatedAddress= "00:0c:29:32:7b:c9"
改爲:ethernet0.generatedAddress= "00:0c:29:32:b7:c9"
5.5.2將:ethernet1.generatedAddress= "00:0C:29:32:7B:D3"
改爲:ethernet1.generatedAddress= "00:0c:29:32:b7:d3"
5.5.3將:uuid.location= "56 4d f0 6c 1a 13 d8 90-9c dd ca 11 59 32 7b c9"
改爲:uuid.location ="56 4d 81 1b 7f cb f2 b1-0c 00 23 1f f2 bd b7 ff"
5.5.4將:uuid.bios= "56 4d f0 6c 1a 13 d8 90-9c dd ca 11 59 32 7b c9"
改爲:uuid.bios = "56 4df0 6c 1a 13 d8 90-9c dd ca 11 59 32 b7 c9"
---------------------------------------------------------------------
5.6啓動節點2，並修改虛擬機網卡配置文件的MAC地址
以root用戶執行
vi/etc/sysconfig/network-scripts/ifcfg-eth0/eth1
5.6.1將:HWADDR=00:0C:29:32:7B:C9
改爲:HWADDR=00:0C:29:32:B7:C9
5.6.2將:HWADDR=00:0C:29:32:7B:D3
改爲:HWADDR=00:0C:29:32:B7:D3
5.7在節點2修改虛擬機網卡的IP地址
---------------------------------------------------------------------
5.7.1以root用戶執行
vi /etc/sysconfig/network-scripts/ifcfg-eth0/eth1
將:172.17.80.11
改爲:172.17.80.12
將：192.168.1.11
改爲:192.168.1.12
--------------------------------------------------------------------
5.8在節點2修改主機名
以root用戶執行
5.8.1vi /etc/sysconfig/network
HOSTNAME=ctpdb2
---------------------------------------------------------------------
六、設置ssh信任關係
6.1分別給節點1、2的oracle和grid創建密碼。
passwd oracle
passwd grid
--------------------------------------------------------------------
6.2調通節點1、節點2的通信關係。
[root@ctpdb1 ~]# ping ctpdb1
64 bytes from ctpdb1 (192.168.1.11):icmp_seq=1 ttl=64 time=0.027 ms
[root@ctpdb1 ~]# ping ctpdb2
64 bytes from ctpdb2 (192.168.1.12):icmp_seq=1 ttl=64 time=1.73 ms
[root@ctpdb2 ~]# ping ctpdb1
64 bytes from ctpdb1 (192.168.1.11):icmp_seq=1 ttl=64 time=0.305 ms
[root@ctpdb2 ~]# ping ctpdb2
64 bytes from ctpdb2 (192.168.1.12):icmp_seq=1 ttl=64 time=0.026 ms
---------------------------------------------------------------------
6.3建SSH信任關係，oracle和grid用戶
6.3.1節點1、2的oracle和grid用戶都要執行
ssh-keygen -t rsa
---------------------------------------------------------------------
6.3.2節點1
cp id_rsa.pub authorized_keys
scp authorized_keys oracle@ctpdb2:~/.ssh
rm authorized_keys
6.3.3節點2
cat id_rsa.pub >> authorized_keys
scp authorized_keys oracle@ctpdb1:~/.ssh
---------------------------------------------------------------------
6.3.4在節點1、2分別在oracle和grid用戶都要執行
ssh ctpdb1 date
ssh ctpdb1-priv date
ssh ctpdb2 date
ssh ctpdb2-priv date
---------------------------------------------------------------------
七、建立分區
注：對物理磁盤進行分區，但不做格式化
7.1以root用戶，節點1
[root@ctpdb1 ~]# fdisk -l
Disk /dev/sda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 535 4192965 82 Linux swap / Solaris
/dev/sda3 536 5221 37640295 83 Linux
Disk /dev/sdb: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280bytes
Device Boot Start End Blocks Id System
Disk /dev/sdc: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280bytes
Device Boot Start End Blocks Id System
Disk /dev/sdd: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280bytes
Device Boot Start End Blocks Id System
Disk /dev/sde: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280bytes
Device Boot Start End Blocks Id System
Disk /dev/sdf: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280bytes
Device Boot Start End Blocks Id System
Disk /dev/sdg: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280bytes
Device Boot Start End Blocks Id System
Disk /dev/sdh: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280bytes
Device Boot Start End Blocks Id System
Disk /dev/sdi: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280bytes
Device Boot Start End Blocks Id System
---------------------------------------------------------------------
7.2root用戶執行，只需要在節點1上執行。
fdisk -l|grep dev
Disk /dev/sdb: 1073 MB, 1073741824 bytes
Disk /dev/sdc: 1073 MB, 1073741824 bytes
Disk /dev/sdd: 1073 MB, 1073741824 bytes
Disk /dev/sde: 10.7 GB, 10737418240 bytes
Disk /dev/sdf: 10.7 GB, 10737418240 bytes
Disk /dev/sdg: 10.7 GB, 10737418240 bytes
Disk /dev/sdh: 5368 MB, 5368709120 bytes
Disk /dev/sdi: 5368 MB, 5368709120 bytes
---------------------------------------------------------------------
7.3root用戶分別對以上磁盤執行：
[root@ctpdb1 ~]# fdisk /dev/sdb
Command (m for help): p
Disk /dev/sdb: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280bytes
Device Boot Start End Blocks Id System
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-130, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK(1-130, default 130):
Using default value 130
Command (m for help): p
Disk /dev/sdb: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 130 1044193+ 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition tablefailed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the nextreboot.
Syncing disks.
---------------------------------------------------------------------
7.4root用戶分別在節點1、節點2執行分區檢查。
[root@ctpdb1 ~]# fdisk -l >fdisk1.log
[root@ctpdb2 ~]# fdisk -l >fdisk2.log
[root@ctpdb1 ~]#scp fdisk1.log root@ctpdb2:~/
---------------------------------------------------------------------
7.5root用戶在節點2上對比，確保無差異，
[root@ctpdb2 ~]# diff fdisk1.log fdisk2.log
即：虛擬機環境下的RAC環境基礎已經完成大半。
---------------------------------------------------------------------
八、配置ASMLib和創建ASM磁盤
8.1 root用戶分別在節點1、節點2執行安裝oracleasm
[root@ctpdb1 oracleasm]# uname -rm
2.6.18-308.el5 x86_64
[root@ ctpdb1 oracleasm]# ls
oracleasm-2.6.18-308.el5-2.0.5-1.el5.x86_64.rpm
注：版本務必與kernel版本一致。
oracleasm-support-2.1.7-1.el5.x86_64.rpm
oracleasmlib-2.0.4-1.el5.x86_64.rpm
[root@ ctpdb1 oracleasm]# rpm -ivh *.rpm
---------------------------------------------------------------------
8.2 root用戶分別在節點1、節點2執行
/etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot propertiesof the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it willhave. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface[grid]: grid
Default group to own the driver interface[asmadmin]:asmadmin
Start Oracle ASM library driver on boot(y/n) [y]: y
Scan for Oracle ASM disks on boot (y/n)[y]:y
Writing Oracle ASM library driverconfiguration: done
Initializing the Oracle ASMLib driver:[ OK ]
Scanning the system for Oracle ASMLibdisks: [ OK ]
---------------------------------------------------------------------
8.3建立ASM磁盤
8.3.1檢查系統磁盤
[root@ctpdb1 ~]# fdisk -l |grep dev |grepLinux |grep -v sda
/dev/sdb1 1 130 1044193+ 83 Linux
/dev/sdc1 1 130 1044193+ 83 Linux
/dev/sdd1 1 130 1044193+ 83 Linux
/dev/sde1 1 1305 10482381 83 Linux
/dev/sdf1 1 1305 10482381 83 Linux
/dev/sdg1 1 1305 10482381 83 Linux
/dev/sdh1 1 652 5237158+ 83 Linux
/dev/sdi1 1 652 5237158+ 83 Linux
---------------------------------------------------------------------
8.3.2只需在節點1執行
[root@ctpdb1 ~]# oracleasm createdisk VOTE1/dev/sdb1
Writing disk header: done
Instantiating disk: done
[root@ctpdb1 ~]# oracleasm createdisk VOTE2/dev/sdc1
Writing disk header: done
Instantiating disk: done
[root@ctpdb1 ~]# oracleasm createdisk VOTE3/dev/sdd1
Writing disk header: done
Instantiating disk: done
[root@ctpdb1 ~]# oracleasm createdisk DATA1/dev/sde1
Writing disk header: done
Instantiating disk: done
[root@ctpdb1 ~]# oracleasm createdisk DATA2/dev/sdf1
Writing disk header: done
Instantiating disk: done
[root@ctpdb1 ~]# oracleasm createdisk DATA3/dev/sdg1
Writing disk header: done
Instantiating disk: done
[root@ctpdb1 ~]# oracleasm createdisk ARCH1/dev/sdh1
Writing disk header: done
Instantiating disk: done
[root@ctpdb1 ~]# oracleasm createdisk ARCH2/dev/sdi1
Writing disk header: done
Instantiating disk: done
[root@ctpdb1 ~]#
---------------------------------------------------------------------
8.3.3root用戶分別在節點1、節點2查看和掃描磁盤
oracleasm listdisks
oracleasm scandisks
注：節點2會在磁盤掃描後顯示和節點1相同的磁盤信息，務必確保正確、一致。
[root@ctpdb2 oracleasm]# oracleasmlistdisks
[root@ctpdb2 oracleasm]# oracleasmscandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "VOTE1"
Instantiating disk "VOTE2"
Instantiating disk "VOTE3"
Instantiating disk "DATA1"
Instantiating disk "DATA2"
Instantiating disk "DATA3"
Instantiating disk "ARCH1"
Instantiating disk "ARCH2"
[root@ctpdb2 oracleasm]# oracleasmlistdisks
ARCH1
ARCH2
DATA1
DATA2
DATA3
VOTE1
VOTE2
VOTE3
[root@ctpdb2 oracleasm]#
--------------------------------------------------------
九、在節點1上執行安裝grid
注、只需要節點1，節點2有oracle通過ssh自動同步。
9.1使用oracle用戶上傳以下介質到/u02下。
p10404530_112030_Linux-x86-64_1of7.zip
p10404530_112030_Linux-x86-64_2of7.zip
p10404530_112030_Linux-x86-64_3of7.zip
9.2配置Xmanager 圖形界面連接ctpdb1的oracle用戶和grid用戶
關於：配置Xmanager 圖形界面參考：
http://linuxxx.blog.51cto.com/1824808/1040165
9.3以grid用戶登錄節點1，cd /u02,執行
unzip p10404530_112030_Linux-x86-64_3of7.zip
cd grid
執行：./runInstaller啓動圖形化安裝嚮導界面
9.4選擇Skipsoftware updates
9.5選擇for aCluster
9.6選擇TypicalInstallation 典型安裝
9.7點擊add添加節點2的配置信息（圖片有個cptdb2-vip錯誤,正確爲ctpdb2-vip）
9.8擊Identifynetwork interfaces 指定網絡接口
注：配置Private 和Public地址，其他地址可設置爲Do Not Use.
9.9點擊SSHConnectivity 測試節點1到節點2的SSH功能。
輸入grid的密碼，點擊test即可。
9.10選擇ClusterRegistry Storage Type 類型爲:Oracle Automatic StorageManagement,並設置sysasm password爲：grid
注：無視此問題，點擊yes（因爲密碼太簡單的原因）
9.11設置Disk groupname爲：CRSGROUP，Redundancy 爲：Extrnal模式。
9.12選擇InventoryDirectory爲：/u01/app/oraInventory（一般默認即可）
9.13節點1安裝環境信息檢查。
9.13.1cvuqdisk-1.0.9-1,此rpm包在/u02/grid/rpm,使用root 用戶
rpm -ivh cvuqdisk-1.0.9-1.rpm
9.13.2scp rpm -ivh cvuqdisk-1.0.9-1.rpm root@ctpdb2:~/
9.13.3root用戶在節點2上，執行cvuqdisk-1.0.9-1.rpm即可。
注：忽略resolv.conf文件問題。選擇Ignore all,點擊Next, 點擊yes跳過，點擊install安裝。
9.14以root用戶在節點1、2分別執行
root@ctpdb1~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of/u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions forworld.
Changing groupname of /u01/app/oraInventoryto oinstall.
The execution of the script is complete.
[root@ctpdb1 ~]#/u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle11g
The following environment variables are setas:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bindirectory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratabfile as needed by
Database Configuration Assistant when adatabase is created
Finished running generic part of rootscript.
Now product-specific root actions will beperformed.
Using configuration parameter file:/u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites duringinstallation
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pawallet
peer wallet keys
pawallet keys
peer cert request
pacert request
peer cert
pacert
peer root cert TP
profile reader root cert TP
paroot cert TP
peer pa cert TP
pa peercert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pauser cert
Adding Clusterware entries to inittab
CRS-2672: Attempting to start 'ora.mdnsd'on 'ctpdb1'
CRS-2676: Start of 'ora.mdnsd' on 'ctpdb1'succeeded
CRS-2672: Attempting to start 'ora.gpnpd'on 'ctpdb1'
CRS-2676: Start of 'ora.gpnpd' on 'ctpdb1'succeeded
CRS-2672: Attempting to start'ora.cssdmonitor' on 'ctpdb1'
CRS-2672: Attempting to start 'ora.gipcd'on 'ctpdb1'
CRS-2676: Start of 'ora.gipcd' on 'ctpdb1' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on'ctpdb1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on'ctpdb1'
CRS-2672: Attempting to start 'ora.diskmon'on 'ctpdb1'
CRS-2676: Start of 'ora.diskmon' on'ctpdb1' succeeded
CRS-2676: Start of 'ora.cssd' on 'ctpdb1'succeeded
ASM created and started successfully.
Disk Group CRSGROUP created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCRkeys.
Creating OCR keys for user 'root', privgrp'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk0b0cea08e4f14fc7bf5c53d50eba4303.
Successfully replaced voting disk groupwith +CRSGROUP.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfullyreplaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1.ONLINE 0b0cea08e4f14fc7bf5c53d50eba4303(ORCL:VOTE1) [CRSGROUP]
Located 1 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on'ctpdb1'
CRS-2676: Start of 'ora.asm' on 'ctpdb1'succeeded
CRS-2672: Attempting to start'ora.CRSGROUP.dg' on 'ctpdb1'
CRS-2676: Start of 'ora.CRSGROUP.dg' on'ctpdb1' succeeded
CRS-2672: Attempting to start'ora.registry.acfs' on 'ctpdb1'
CRS-2676: Start of 'ora.registry.acfs' on'ctpdb1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@ctpdb1 ~]#
務必確保節點1、2都能出現
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
9.15在節點1和節點2分別進行ping通ctpdb-cluster，如果可以ping通，則此錯誤可以忽略。點擊ok,再點擊close，至此grid安裝完成。
[root@ctpdb1 ~]# ping ctpdb-cluster
PING ctpdb-cluster (192.168.1.10) 56(84)bytes of data.
64 bytes from ctpdb-cluster (192.168.1.10):icmp_seq=1 ttl=64 time=0.027 ms
[root@ctpdb2 ~]# ping ctpdb-cluster
PING ctpdb-cluster (192.168.1.10) 56(84)bytes of data.
64 bytes from ctpdb-cluster (192.168.1.10):icmp_seq=1 ttl=64 time=1.16 ms
---------------------------------------------------------------------
9.16圖形界面
使用grid用戶在節點1，執行asmca，創建DATAGROUP和ARCHGROUP磁盤組
9.17.1在Disk Groups下點擊 Create，
Disk Groups Name 爲：DATAGROUP，External(none)模式，並勾選DATA1/DATA2/DATA3，點擊ok繼續，再點擊ok確定完成。
9.17.2在Disk Groups下點擊 Create，
Disk Groups Name 爲：ARCHGROUP，External(none)模式，並勾選ARCH1/ARCH2，點擊ok繼續,再點擊ok確定完成。
最後點擊Exit退出即可。
---------------------------------------------------------------------
十、圖形界面以oracle用戶登錄,進入/u02/ 目錄
10.1 unzip p10404530_112030_Linux-x86-64_1of7.zip
unzip p10404530_112030_Linux-x86-64_2of7.zip
10.2 cd database
執行./ ./runInstaller 啓動圖形化安裝嚮導界面，取消選擇
I wish to receive security updates via MyOracle Support.
next 繼續
10.3 選擇Skipsoftware updates ,next繼續
10.4 選擇Installdatabase software only,next 繼續
10.5 選擇OracleReal Application Cluster database installation,一定勾選所有nodename,爲了順利期間，點擊SSH Connectivity ,輸入oracle密碼，點擊test,做SSH信任連接測試。next繼續
10.6選擇oracle運行語言爲English,next 繼續。
10.7選擇EnterpriceEdition (4.5G)企業版，next繼續
10.8查看oracle安裝路徑,Oracle Baes爲 /u01/app/oracle；
Software Location爲/u01/app/oracle/product/11.2.0/db_1；next繼續
10.9OSDBA爲dba,OSOPER爲oper；next繼續
10.10各項檢查通過後，next繼續
此時爲報：resolv.conf失敗和SCAN警告，忽略即可。（可點擊如下：more details查看更多信息），選擇Ignore All,next繼續，點擊yes。
10.11點擊install
10.12開始安裝oracle軟件，此過程時間較長，因爲oracle本身需要將節點1數據信息通過ssh同步至節點2。
使用root用戶，在連個節點1、節點2分別執行:
/u01/app/oracle/product/11.2.0/db_1/root.sh
點擊ok, Close 關閉即可。
十一、DBCA 建庫
11.1在節點1上，使用oracle用戶，執行dbca
選擇Oracle Real ApplicationCluster(RAC)database,next繼續；
11.2選擇Create aDatabase 創建一個數據庫, next繼續。
11.3選擇CustomDatabase 自定義安裝, next繼續。
11.4Global Database Name ctpdb,SID Prefixctpdb,Select All 所有節點。next繼續。
11.5在EnterpriseManager取消Configure Enterprise Manger 和AutomaticMaintenance Tasks 取消Enable sutomatic maintenance tasks。next繼續。
11.6設置統一密碼，點擊yes,next繼續。
11.7選擇StorageType 爲：ASM方式，Database Area爲：+DATAGROUP，next繼續。
11.8 取消flashrecovery area和Enable Archiving，next繼續。
11.9全部取消，next繼續。
11.10Memory使用默認大小，40%。Sizing BlockSize爲8192，Process爲300。
Character Sets爲：如圖所示。Connection Mode爲Dedicated Server Mode獨享模式。next繼續，再次 next繼續。
11.11點擊Flinish,點擊ok。開始創建數據庫。
十一、數據庫測試
11.1使用grid 用戶檢查數據庫運行狀態。
[grid@ctpdb1 ~]$ crs_stat -t
11.2使用grid 用戶停止oracle，啓動oracle。
[grid@ctpdb1 ~]$ srvctl status database -dctpdb
Instance ctpdb1 is running on node ctpdb1
Instance ctpdb2 is running on node ctpdb2
[grid@ctpdb1 ~]$
[grid@ctpdb1 ~]$ srvctl stop database -dctpdb
[grid@ctpdb1 ~]$
[grid@ctpdb1 ~]$ srvctl status database -dctpdb
Instance ctpdb1 is not running on nodectpdb1
Instance ctpdb2 is not running on nodectpdb2
[grid@ctpdb1 ~]$
[grid@ctpdb1 ~]$ srvctl start database -dctpdb
[grid@ctpdb1 ~]$ srvctl status database -dctpdb
Instance ctpdb1 is running on node ctpdb1
Instance ctpdb2 is running on node ctpdb2
[grid@ctpdb1 ~]$
srvctl命令：
[grid@ctpdb1 ~]$ srvctl
Usage: srvctl <command> <object>[<options>]
commands:enable|disable|start|stop|relocate|status|add|remove|modify|getenv|setenv|unsetenv|config|convert|upgrade
objects:database|instance|service|nodeapps|vip|network|asm|diskgroup|listener|srvpool|server|scan|scan_listener|oc4j|home|filesystem|gns|cvu
11.3使用grid 用戶啓動監聽，做客戶端連接測試。
[grid@ctpdb1 ~]$ lsnrctl status 檢查監聽狀態
[grid@ctpdb1 ~]$ lsnrctl stop 停掉監聽，
[grid@ctpdb1 ~]$ lsnrctl start 啓動監聽，OK 監聽正常啓動。
11.4
RAC版tnsnames.ora配置,含
# tnsnames.ora Network Configuration File: D:\oracle\product\10.2.0\db_1\network\admin\tnsnames.ora
# Generated by Oracle configuration tools.
CTPDB1=
(DESCRIPTION=
(ADDRESS=
(PROTOCOL=TCP)
(HOST=192.168.1.111)
(PORT=1521)
)
(CONNECT_DATA=
(FAILOVER_MODE=
(BACKUP=ctpdb2)
(TYPE=select)
(METHOD=preconnect)
)
(INSTANCE_NAME=ctpdb1)
(SERVICE_NAME=ctpdb)
)
)
CTPDB2=
(DESCRIPTION=
(ADDRESS=
(PROTOCOL=TCP)
(HOST=192.168.1.112)
(PORT=1521)
)
(CONNECT_DATA=
(FAILOVER_MODE=
(BACKUP=ctpdb1)
(TYPE=select)
(METHOD=preconnect)
)
(INSTANCE_NAME=ctpdb2)
(SERVICE_NAME=ctpdb)
)
)
11.5停止數據庫節點1，查看數據庫運行狀態，做節點冗餘切換測試。
[oracle@ctpdb1 ~]$ sqlplus sys/oracle assysdba
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>
根據圖片能看到oracle已經無縫的從節點1切換到節點2。
SQL> startup; 啓動節點1，客戶端仍然正常的接到節點2上。
11.6停止數據庫節點1/2，重啓數據庫服務器。//以下不再做演練操作
11.7啓動數據庫節點1/2做正常操作。
-----------------------------------------------------------------------------------------
最近寒楓不是一般的忙，隨着系統逐漸擴大，公司安排核心系統最近上線，不過切換前後的平穩的過度，能在閒時也整理整理文檔，也是一種愜意的享受。畢竟要上夜班了~
只是無論學習Oracle還是Linux等其他技術也好，均要徐徐漸進；十年磨一劍，此劍必須要能拿的出，用得上，打的贏 ；如果半途而非，最終將竹籃打水一場空，除非你的竹籃密密麻麻。
接下來的時間，等系統也穩定了，沒有這麼多瑣事了，還需定心學習Oracle、Linux，同時也喜歡在業餘時間和大家討論技術，包括Oracle、Linux以及其他關於系統交易、程序化交易、金融市場，寒楓均有很大興趣。
本文也沒有統一的編寫格式還需文學者指導。
本文主題爲：【讓學習oracle前進一步】
如果你對上文不感興趣，那麼請點擊http://umiwi.diandian.com看看這個也到消遣以下。
-----------------------------------------------------------------------------------------
轉載請註明出處:http://linuxxx.blog.51cto.com/1824808/1274890#576555
-----------------------------------------------------------------------------------------
參考文檔：
1、CTP交易系統官方安裝文檔oracle 11.2.0.0 +rhel 5.5版
2、http://wangchunhai.blog.51cto.com/225186/42184/
3、http://www.itpub.net/thread-1009235-1-1.html
4、http://linuxxx.blog.51cto.com/1824808/1040165
附件：
1、get_rpm.sh 和get_rpm 用來獲取rpm是否安裝。
2、RAC版的tnsnames.ora
3、虛擬機修改的文件
-----------------------------------------------------------------------------------------