Version 변경일자 변경자(작성자) 주요내용
1 2010.04.01 정철우 최초 작성
Author 정철우 Creation Date 2010-04-01
Last Updated Version 1.0
Copyright(C) 2004 Goodus Inc.
All Rights Reserved
11gR2 RAC
Installation Manual
(with ASM+Rawdevice)
11gR2 RAC Installation Manual
Contents
1. 개요 ... 4
2. Hardware Requirements ... 4
1.1 Shared Storage Option ... 4
1.2 Disk Space ... 5
1.3 Memory : At least 2.5GB... 5
3. Network Requirements(IP address requirements) ... 6
4. Tuning AIX System Environment ... 8
4.1. Checking Asynchronous Input Output Processes for AIX 5L ... 8
4.2. Tuning Virtual Memory Manager (VMM) ... 9
4.3. Increasing System Block Size Allocation ... 9
4.4. Configuring Shell Limits ... 9
4.5. Configuring User Process Parameters... 10
4.6. Configuring Network Tuning Parameters ... 10
5. Software Requirements ... 11
5.1. Operating system ... 11
5.2. Packages & Patches ... 11
5.2.1. AIX 6.1 ... 11
5.2.2. AIX 5.3 ... 11
6. 참고문서 ... 12
7. System Overview ... 13
7.1. System Architecture. ... 13
7.2. H/W and S/W Requirement ... 13
7.2.1. Hardware ... 13
7.2.2. Software ... 13
7.3. IP 구성정보 ... 14
7.4. HACMP Status ... 14
8. Installation Steps... 15
8.1. Preparing the system ... 15
8.1.1. Host file Configuration ... 15
8.1.2. ORACLE User and DBA Group Creation ... 15
8.1.3. ASM Disk Check ... 15
8.1.4. SSH Configuration ... 16
11gR2 RAC Installation Manual
8.1.5. NPT Service Configuration ... 16
8.1.6. User OS enviropment Configuration ... 16
8.2. Grid Infrastructure Software Installation ... 17
8.2.1. Cluvfy.sh 수행 ... 17
8.2.2. Rootpre.sh(root 유저로 반드시 양 노드 모두 수행) ... 23
8.2.3. GRID Infrastructure Software Installation ... 25
8.3. Database Software Installation ... 41
8.4. Check Public/Inter-connect Configuration ... 48
8.5. OCR & VOTE Disk Migration ASM To Rawdevice ... 49
9. Create Database ... 50
9.1. DBCA... 50
9.2. Manual DB Creation... 62
9.3. DATABASE Status ... 66
9.4. ASM Resource 제거... 66
9.5. Network Setting ... 66
9.5.1. SCAN_LISTENER 제거 ... 66
9.5.2. Remote Listener 제거 ... 66
9.5.3. LISTENER.ORA ... 66
9.6. Archive Log Mode 변경 ... 67
9.7. CRS Resource Status ... 68
9.8. DB Parameter ... 68
10. 마치며 ... 73
11gR2 RAC Installation Manual
1. 개요
10g 또는 11gR1 환경에서 기존에 RAC 를 운영하던, DBA 들의 11gR2 RAC 에 대한 가지고 있는 불만 중 하나는, 기존에 Storage 영역으로 가장 많이 사용하고 있는 RAWDEVICE 에 대한 미 지원이라고 할 수 있습니다.
(정확히 이야기 하면, Software 상에서는 지원하나, OUI or DBCA 등의 GUI 에선 미지원.)
물론 기존에 운영 중인던, RAC 시스템을 업그레이드 할때는 RAWDEVICE 를 지원하지만, 신규 시스템 구 축 일 경우 GUI 상에서 RAWDEVICE 를 미지원 한다는건 불편한 사항일 수 밖에 없습니다.
본 문서는 RAWDEVICE 를 사용하여 11gR2 RAC 를 구축하는 것을 주제로 다루고 있습니다.
본 문서는 AIX5L 기반의 장비에서 작성 되었으며, 다른 unix or linux 플랫폼에도 동일한 방법으로 적용 가능 합니다.
2. Hardware Requirements
1.1 Shared Storage Option
RAC 구성시 사전에 결정해야 될 가장 중요한 부분은 Data Storage 입니다.
RAC 에서 구성 가능한 (Shared Access) 방법은 ASM, Shared File System(CFS, NFS), RAWDEVICE 입니다.
그러나 앞서 이야기 한 것처럼 11gR2 부터는 rawdevice 를 GUI 메뉴 상에서 지원하지 않으며, 특히 Grid Install 시에는, 반드시 ASM 또는 Shared File System 으로만 OCR 과 Voting Disk 생성이 가능합니다.
11gR1 까지는 ASM Software 는 Database Software 를 공유하여 사용하였으나, 11gR2 부터는 Oracle clusterware 와 ASM 이 통합되었습니다.
이는 기존 RAC 환경에서 ASM 을 Storage 영역으로 사용하더라도, CRS 구성 후 ASM Instance 를 구성해 야 했기 때문에 CRS 구성시 필요한 OCR 과 Voting Disk 는 ASM 영역을 사용할 수 없어, RAWDEVICE 나 Shared File System(CFS or NFS)를 사용해야 했으나, 11gR2 부터는 Grid Infrastructer(이하 Grid) 구성시에 ASM Diskgroup 를 생성할 수 있기 때문에 OCR 과 Voting Disk 도 ASM 영역에 생성할 수 있도록 변경 되었습니다.
즉 Oracle 에서 11gR2 RAC 구성시 권고하는 대로 ASM 을 사용하거나 Shared File System (CFS or NFS)을 사용하게 되면, 더 이상 OS Vendor Cluster 없이 RAC 구성이 가능하다는 의미 이기도 합니다.
그러나 본 문서에서는 RAWDEVICE 를 Data Storage 영역으로 사용할 것이므로, Shraed Access 가능한
11gR2 RAC Installation Manual
RAWDEVICE 생성을 위해 IBM 에서 지원하는 OS Vendor clusterware(HACMP)를 사용할 것 입니다.
*10g R2 부터는 Cluster Ready Service(CRS)가 Oracle Clusterware 로 명칭이 변경되었으나, 본 문서에서는 편의상 CRS 로 명칭
l ASM(Automatic Storage Management) - strongly recommended l CFS(cluster file system) - GPFS on IBM platforms
l NFS(Network file system) - Not supported on AIX Based Systems l Raw Devices - Upgrades only (OUI, DBCA Not Supported)
Storage Option
OCR and Voting Disks
Oracle Clusterware binaries
Oracle RAC binaries
Oracle Database Files
Oracle Recovery Files
ASM Yes No No Yes Yes
GPFS Yes Yes Yes Yes Yes
Local file
system No Yes Yes No No
[Supported Storage Options for Oracle Clusterware and Oracle RAC]
l Note :
w You cannot place ASM files on GPFS.
w Oracle does not recommend the use of GPFS for voting disks if HACMP is used.
w Disk lock속성을 reserve_policy=no_reserve로 설정해야 양 노드에서 Access 가능. w AIX에서 ASM Disk는 /dev/rhdiskN을 Default Path로 찾음
(The response is either a reserve_lock setting, or a reserve_policy setting. If the attribute is reserve_lock, then ensure that the setting is reserve_lock = no. If the attribute is reserve_policy, then ensure that the setting is reserve_policy = no_reserve.)
1.2 Disk Space
l Oracle Clusterware Home and log files : 2GB l Oracle Database(Oracle base) : 7.5GB l Oracle Clusterware Files : 10GB
è 2 OCR Disk(at least 280MB), 3 Vote Disk(at least 280MB) l /tmp 4GB
1.3 Memory : At least 2.5GB
RAM Swap Space
Between 1.5GB and 2GB 1.5 times the size of RAM Between 2GB and 8GB Equal to the size of RAM
More than 8GB 0.75 imes the size of RAM
11gR2 RAC Installation Manual
3. Network Requirements(IP address requirements)
Cache Fusion 에 의한 부하를 최소화시키기 위해 InterConnect 용 NIC 에 최소한 Gigabit Line 을 사용할 것을 권고합니다.
또한 11gR2 RAC 에서는 Interconnect Line 을 Cross-Cable 을 이용하여 양 서버를 Direct 로 연결 할 수 없으며, 반드시 양 노드 중간에 Switch 를 사용하여 Interconnect 를 구성해야 합니다.
l For each Node w public IP address TCP/IP 을 지원 해야 함.
DNS 및 /etc/hosts 에 등록
l private IP address (interconnect) w UDP 지원 해야 함.
w Public interface 와 다른 Subnet 을 사용 l Virtural IP address (failover) w DNS 및 /etc/hosts 에 등록
w 이미 사용되지 않은 IP 와 Hostname 을 사용 w Public interface 와 동일한 Subnet 을 사용
l Interface Names : 모든 노드의 public interface name과 private interface name는 반드시 양 노드가 동일하게 설정한다.
l NIC Bonding : Network interface를 다중화 하여 network interface (private and public) Failover 하도록 권장. (Heartbeat Fatal에 의한 Split Brain 방지)
l GNS(Grid Naming Service) : Cluster 내에서 Grid Control를 사용하기 위한 Naming Service로서 설치 전 DNS에 등록하여 사용한다.
11gR2 RAC 에서는 Network 관련 2 가지의 새로운 개념이 등장 하였습니다.
첫번째는 GNS(Gird Naming Service) 로서, Grid 설치시 사용 여부를 선택 할 수 있습니다.
GNS 를 사용하게 되면, 다음에 나올 SCAN VIP 가 DHCP 로 동적으로 할당되게 됩니다.
GNS 를 사용하기 위해서는 아래와 같이 DNS 에 GNS NAME 과 GNS VIP 를 등록해 주어야 합니다.
GNS 는 기존의 2node RAC 환경을 위한 개념이라기 보다는, Grid 환경에서 다수의 Node 를 RAC Cluster 상에서 add 하거나 Remove 하는 과정을 간단하게 하는 Grid plug and play (GPnP)를 위한 개념이라고 볼 수 있으며, GNS 를 통해 추가된 node 에 IP 할당을 동적으로 수행할 수 있습니다.
참고로 11gR2 RAC 는 최대 100 대의 노드를 Cluster 환경으로 구축할 수 있습니다.
11gR2 RAC Installation Manual
본 문서에서는 2Node RAC 를 구축할 것 이므로, GNS 사용을 설정하지 않았습니다.
w Virtual IP address 사용 w DHCP 사용
w GPnP(Grid Plug and Play) 사용시에 구성
l SCAN(SINGLE CLIENT ACCESS NAME) :
11gR2 RAC에서 새로 나온 두번째 network 관련 개념으로서 Cluster 로 구성되어 운영중인 Oracle Database 에 접근하는 모든 Client 에서 SCAN 으로 접속이 가능한 신 기능이다.
SCAN으로 접속시 자동으로 Server Side Load Balancing 을 수행한다.
(Default로 Remote Listener 에 SCAN 이 등록됨)
SCAN은 Grid 설치 시 반드시 구성해야 하며, GNS 을 사용하지 않는다면, SCAN NAME 을 아래와 같이 DNS 상에 등록해줘야 합니다.
기본적으로 3개의 Public IP 를 동일한 이름으로 등록해줘야 하며, Public 와 SCAN Vip 의 subset 은 동일하여야 합니다. (최소 1개의 IP 등록)
SCAN에 할당한 IP 수 만큼, SCAN_LISTENER 와 SCAN VIP 가 Clusterware Resource 에 등록되게 됩니다.
SCAN은 Grid 설치 후에 제거는 가능하지만, 11gR2 RAC 에서는 EM 구성시 Scan Listener 를 이용해서 접속하기 때문에 EM 을 사용하기 위해서는 SCAN 을 제거하면 안됩니다.
(SCAN 이 없이 EM 을 구성하는 방법은 현재로서는 알려져 있지 않습니다.)
SCAN 사용시 Client 에서는 Tnsnames.ora 에 다음과 같이 설정하여 접속을 수행하게 됩니다.
GOODUS =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = cluster01-scan)(PORT = 1521)) (CONNECT_DATA =
(SERVER = DEDICATED) (SERVICE_NAME = GOODUS) )
)
SCAN을 제거하게 되면, Client 에서 접속시 기존 RAC 와 동일하게 VIP 를 사용해서 접속하면 됩니다.
11gR2 RAC Installation Manual
본 문서에서는 DNS 서버 없이 /etc/hosts 파일에 1 개의 scan ip 만을 지정해서 구성하였습니다.
이는 RAC 구성이 완료된 후 SCAN 을 제거하여 기존 10g RAC 환경과 동일하게 사용하기 위함 입니다.
Srvctl 명령어를 통해 SCAN 및 SCAN Listener 를 추가, 삭제할 수 있습니다.
(1개만 사용할 경우, client에서 scan-ip가 자동 할당된 노드로만 TNS를 이용하여 접속이 가능합니다.
만약 EM를 사용할 예정이라면 반드시 권고사항 대로 3개의 SCAN-IP를 할당하셔야 합니다.)
4. Tuning AIX System Environment
4.1. Checking Asynchronous Input Output Processes for AIX 5L
l AIX 6.1
11gR2 RAC Installation Manual
w ioo -o aio_maxreqs aio_maxreqs = 65536 l AIX 5.3
w lsattr -El aio0 -a maxreqs
maxreqs 65536 Maximum number of REQUESTS True
l Async I/O setup
# smitty aio
"change/show characteristics of asynchronous I/O"
"State to be configured at system restart"
Defined ---> available 로 setting 후 반드시 재부팅(init 6)
4.2. Tuning Virtual Memory Manager (VMM)
l Recommended Values for Virtual Memory Manager Parameter Value
minperm% 3(default is 20) maxperm% 90(default is 80) maxclient% = 90(default is 80) lru_file_repage 0(default is 1) strict_maxclient 1(default is 1) strict_maxperm 0(default is 0)
l Example
vmo -p -o minperm%=3 vmo -p -o maxperm%=90 vmo -p -o maxclient%=90 vmo -p -o lru_file_repage=0 vmo -p -o strict_maxclient=1 vmo -p -o strict_maxperm=0 You must restart the system for these changes to take effect.
4.3. Increasing System Block Size Allocation
chdev -l sys0 -a ncargs='128'
4.4. Configuring Shell Limits
Software의 성능 개선을 위해 Oracle S/W 의 설치 user(oracle, crs)의 shell limit를 조절
Shell Limit Item in
limits.conf Hard limit Maximum number of open file descriptors nofile -1 (unlimited)
11gR2 RAC Installation Manual
Maximum number of processes available to a single user maxuproc 16384
Maximum size of the stack segment of the process stack -1
(unlimited)
l /etc/security/limits(모든 노드에 추가) default:
w fsize = -1 w core = 2097151 w cpu = -1 w data = -1 w rss = -1 w stack = -1 w nofiles = -1
l Maximum number of processes available(Oracle S/W user) (모든 노드에 추가) w /etc/chdev -l sys0 -a maxuproc = 16384
4.5. Configuring User Process Parameters
각 user에 최대 허락된 maximum number of processes = 2048 or 그 이상으로 조정
# smit chgsys
Maximum number of PROCESSES allowed for each user = 2048 or 그 이상으로 조정 press F10 to exit
4.6. Configuring Network Tuning Parameters
Network Tuning Parameter Recommended Value
ipqmaxlen 512
rfc1323 1
sb_max 2*65536
tcp_recvspace 65536
tcp_sendspace 65536
udp_recvspace 655360
udp_sendspace 65536
l /etc/rc.net
if [ -f /usr/sbin/no ] ; then /usr/sbin/no -o udp_sendspace=65536 /usr/sbin/no -o udp_recvspace=655360 /usr/sbin/no -o tcp_sendspace=65536 /usr/sbin/no -o tcp_recvspace=65536 /usr/sbin/no -o rfc1323=1
11gR2 RAC Installation Manual
/usr/sbin/no -o sb_max=2*655360 /usr/sbin/no -o ipqmaxlen=512
fi
5. Software Requirements
5.1. Operating system
l AIX 5L V5.3 TL 09 SP1 ("5300-09-01") 64 bit kernel (Part Number E10854-01) l AIX 6.1 TL 02 SP5 ("6100-02-05") 64-bit kernel
5.2. Packages & Patches
5.2.1. AIX 6.1 5.2.2. AIX 5.3
packages (or later versions) bos.adt.base
bos.adt.lib bos.adt.libm bos.perf.libperfstat bos.perf.perfstat bos.perf.proctools rsct.basic.rte
rsct.compat.clients.rte
xlC.aix61.rte 10.1.0.0 (or later) patches (or later versions) IZ41855
IZ51456 IZ52319
packages (or later versions) bos.adt.base
bos.adt.lib bos.adt.libm bos.perf.libperfstat bos.perf.perfstat bos.perf.proctools rsct.basic.rte
rsct.compat.clients.rte
xlC.aix50.rte 10.1.0.0 (or later) patches (or later versions) IZ42940
IZ49516 IZ52331 Obtaining C/C++ Compilers : xLC.aix.rte 8.0.0.8 and xlC.rte 8.0.0.8.
Oracle RAC : xLC.aix.rte 8.0.0.8 and xlC.rte 8.0.0.8.
HACMP 5.4.1
HACMP filesets (cluster.es.*) rsct.hacmp.rte
rsct.compat.basic.hacmp.rte rsct.compat.clients.hacmp.rte
GPFS
AIX 6L: gpfs.base 3.2.1.8 or later.
AIX 5L: gpfs.base 3.2.1.8 or later
ADA : OC Systems PowerAda 5.4d
11gR2 RAC Installation Manual
JDK : Use one of the following Java versions:
Java 6 64-bit 6.0.0.50 IZ30726 (SR2) Java 5 64-bit 5.0.0.250 IZ55274 (SR8a)
Pro*FORTRAN : IBM XL Fortran v. 11.1 April 2008 PTF for AIX
Pro*C/C++, Oracle Call Interface, Oracle C++ Call Interface, Oracle XML Developer’s Kit (XDK), GNU Compiler Collection (GCC)
February 2007 XL C/C++ Enterprise Edition V8.0 for AIX PTF 8.0 gcc 3.45
Pro*COBOL
IBM COBOL for AIX version 3.1 Micro Focus Server Express 5.1
Oracle Messaging Gateway IBM WebSphere MQ V6.0.2.0, client and server:
mqm.Client.Bnd mqm.Server.Bnd
Oracle JDBC/OCI Drivers(AIX 5L v5.3) APAR required for JDK 1.4.2 (64-bit):
IY63533: JDK 1.4.2 64-bit SR1 caix64142-20040917 APARs required for JDK 1.3.1.16 (32-bit):
IY58350: SDK 1.3.1 32-Bit SR7P: CA131IFX-20040721A IY65305: JAVA142 32-bit PTF: CA142IFX-20041203
6. 참고문서
# E10814-01 : Oracle Grid Infrastructure Installation Guide, 11gR2 for IBM AIX on POWER Systems (64-Bit)
# Note ID 169706.1 : Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
11gR2 RAC Installation Manual
7. System Overview
7.1. System Architecture.
7.2. H/W and S/W Requirement 7.2.1. Hardware
NAME Value Remark
CPU 5Ghz * 4
Physical Memory 30GB
7.2.2. Software
NAME Value Remark
OS Kernel AIX 5.3 TL11
SWAP 30G
/tmp size 4GB
HACMP 5.4
IBM P590
2Nodes RAC * 1Set
11gRAC
11gR2 RAC Installation Manual
7.3. IP 구성정보
Host Name Type Value
Goodus1 Public IP 20.20.20.1
Goodus2 Public IP 20.20.20.2
Goodus1-ipc Private IP (Interconnect) 10.10.10.1
Goodus2-ipc Private IP (Interconnect) 10.10.10.2
Goodus1-vip VIP (Virtual IP) 20.20.20.3
Goodus2-vip VIP (Virtual IP) 20.20.20.4
Goodus-scan SCAN-IP 20.20.20.5
7.4. HACMP Status
# /usr/es/sbin/cluster/utilities/cllsif
Adapter Type Network Net Type Attribute Node IP Address Hardware Address Interface Name Global Name Netmask Alias for HB
goodus1 boot net_ether_01 ether public goodus1 20.20.20.1 en14 255.255.255.0
goodus1-ipc service net_ether_02 ether private goodus1 10.10.10.1 en15 255.255.255.0
goodus2 boot net_ether_01 ether public goodus2 20.20.20.2 en14 255.255.255.0
goodus2-ipc service net_ether_02 ether private goodus2 10.10.10.2 en15 255.255.255.0
# /usr/es/sbin/cluster/clstat
clstat - HACMP Cluster Status Monitor
--- Cluster: k2rac (1268804794)
Wed Mar 31 14:59:50 2010
State: UP Nodes: 2 SubState: STABLE
Node: goodus1 State: UP
Interface: goodus1 (0) Address: 20.20.20.1 State: UP
Interface: goodus1-ipc (1) Address: 10.10.10.1 State: UP
Resource Group: goodus1_rg State: On line Resource Group: goodus_reg State: On line Node: goodus2 State: UP
Interface: goodus2 (0) Address: 20.20.20.2 State: UP
Interface: goodus2-ipc (1) Address: 10.10.10.2 State: UP
Resource Group: goodus2_rg State: On line Resource Group: goodus_reg State: On line
11gR2 RAC Installation Manual
8. Installation Steps
8.1. Preparing the system 8.1.1. Host file Configuration
/etc/hosts 파일에 각노드의 Pulic IP 주소 및 VIP, Interconnect 주소를 등록합니다.
Scan vip 의 경우 설치 후 제거를 할 예정으로, DNS 가 아닌 hosts 파일에 등록 (양 노드 동일한 ip 설정) (root 사용자로 양 노드 수행합니다.)
[oracle@goodus1:/oracle]$ cat /etc/hosts
127.0.0.1 loopback localhost # loopback (lo0) name/address 20.20.20.1 goodus1
20.20.20.2 goodus2
# IP For Service
20.20.20.3 goodus1-vip 20.20.20.4 goodus2-vip
# SCAN IP
20.20.20.5 goodus-scan
# Interconnect IP
10.10.10.1 goodus1-ipc 10.10.10.2 goodus2-ipc
8.1.2. ORACLE User and DBA Group Creation
Oracle 사용자가 사용할 dba 그룹과 Oracle 사용자를 생성합니다.
11g 에선 GRID 유저와 DB 유저를 별도로 생성할 것을 권고
(단 별도로 유저 생성시 Permission 관련 문제가 많이 발생함으로, 동일 유저로 구성이 간편함)
(root 사용자로 양 노드 모두 수행합니다.) [root@ ~]# mkdir -p /oracle/db
[root@ ~]# mkdir -p /oracle/grid [root@ ~]# groupadd -g 300 dba [root@ ~]# cat /etc/passwd | grep ora
oracle:!:300:300:Oracle DBA:/oracle/db:/usr/bin/ksh
oragrid:!:301:300:Oracle CRS Admin:/oracle/grid:/usr/bin/ksh
[root@ ~]# cat /etc/group | grep dba dba:!:300:oracle,oragrid
[root@ ~]# chown –R oracle :dba /oracle/db [root@ ~]# chown –R oragrid :dba /oracle/grid [root@ ~]# chmod –R 775 /oracle/db
[root@ ~]# /usr/bin/chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE oragrid [root@ ~]# /usr/bin/chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE oracle
8.1.3. ASM Disk Check
설치 중 OCR, VOTE 생성용으로 사용할 ASM Disk 확인
goodus1:/root># chown oragrid:dba /dev/rhdisk154 goodus1:/root># chmod 660 /dev/rhdisk154
goodus1:/root># ls –l /dev/rhdisk154
crw-rw---- 1 oragrid dba 15,154 Mar 25 15:07 /dev/rhdisk154
해당 디스크의
reserve_policy 는 no_reserve 로 설정되어 있어야 함.
11gR2 RAC Installation Manual
8.1.4. SSH Configuration
설치간 양 노드에 oracle 및 grid 유저로 ssh 접속이 가능해야 함.
(grid 설치 단계에서 자동 설정 가능하나, cluster Verify 수행을 위해 사전 구성)
/usr/bin/ssh-keygen -t rsa /usr/bin/ssh-keygen -t dsa
ssh goodus1 cat /oracle/grid/.ssh/id_rsa.pub >> authorized_keys ssh goodus1 cat /oracle/grid/.ssh/id_dsa.pub >> authorized_keys ssh goodus2 cat /oracle/grid/.ssh/id_rsa.pub >> authorized_keys ssh goodus2 cat /oracle/grid/.ssh/id_dsa.pub >> authorized_keys
scp authorized_keys goodus2:/oracle/db/.ssh/
8.1.5. NPT Service Configuration
양 노드간의 Time Sync 를 위해 Network Protocol Daemon 이 반드시 x 옵션을 사용하여, start 되어야만 함.
Checking NTP daemon's boot time configuration, in file "/etc/rc.tcpip", for slewing option "-x"
[oracle@goodus1:/oracle/db]$ ps -ef | grep ntp
root 152084 180318 0 Mar 18 - 0:23 /usr/sbin/xntpd –x
8.1.6. User OS enviropment Configuration
Oracle 사용자의 Cshell 환경을 설정 합니다. /oracle/db/.profile 을 아래와 같이 생성합니다.
(oracle 사용자로 아래와 같이 양 노드 설정합니다.) GOODUS1-> 2 번 노드는 GOODUS2 (노란색 음영부분)
l 11gR2 부터는 TNS_ADMIN, ORA_NLS10, ORA_CRS_HOME 을 user profile 에서 unset 해야 함.
(셋팅시 grid 설치 중 roo.sh 수행시 GPnp Profile Create Fail 에러 발생 함) Failed to create a peer profile for Oracle Cluster GPnP. gpnptool rc=256 l Grid Home 은 Oralce Base 아래에 위치할 수 없음
[oracle@goodus1:/oracle/db]$ cat .profile umask 022
if [ -s "$MAIL" ] # This is at Shell startup. In normal then echo "$MAILMSG" # operation, the Shell checks fi # periodically.
EDITOR=vi; export EDITOR HOST=`hostname`
USER=`logname`
PS1='['`logname`'@'`hostname`':$PWD]$ '
set -o vi
stty -istrip erase
alias ls='ls -aCF' alias ll='ls -alF' alias rm='rm -i' alias vi='/usr/bin/vi'
LANG=C; export LANG
export ORACLE_BASE=/oracle/db export GI_HOME=/oracle/grid
export ORACLE_HOME=/oracle/db/11g export ORACLE_SID=GOODUS1
11gR2 RAC Installation Manual
export NLS_LANG=AMERICAN_AMERICA.UTF8 export AIXTHREAD_SCOPE=S
PATH=$PATH:$ORACLE_HOME/bin:/oracle/dba:$GI_HOME/bin:$ORACLE_HOME/OPatch
8.2. Grid Infrastructure Software Installation 8.2.1. Cluvfy.sh 수행
(oracle 사용자로 goodus1 에서 수행합니다.)
runInstaller 수행전 아래와 같이 cluvfy.sh 를 수행하여 설치과정에 miss 나 fail 부분을 미리 check 한다.
[oracle@goodus1:/oracle/INST/grid]$ ./runcluvfy.sh stage -pre crsinst -n goodus1,goodus2 -verbose
Performing pre-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "goodus1"
Destination Node Reachable?
--- --- goodus2 yes goodus1 yes Result: Node reachability check passed from node "goodus1"
Checking user equivalence...
Check: User equivalence for user "oragrid"
Node Name Comment --- --- goodus1 passed goodus2 passed Result: User equivalence check passed for user "oragrid"
Checking node connectivity...
Checking hosts config file...
Node Name Status Comment --- --- --- goodus1 passed goodus2 passed
Verification of the hosts config file successful
Interface information for node "goodus1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU --- --- --- --- --- --- --- en14 20.20.20.1 20.20.20.0 20.20.20.1 20.20.20.1 00:14:5E:78:E9:5C 1500 en15 10.10.10.1 10.10.10.0 10.10.10.1 20.20.20.1 00:14:5E:78:E9:46 1500
Interface information for node "goodus2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU --- --- --- --- --- --- --- en14 20.20.20.2 20.20.20.0 20.20.20.2 20.20.20.1 00:14:5E:78:E9:6C 1500 en15 10.10.10.2 10.10.10.0 10.10.10.2 20.20.20.1 00:14:5E:78:E9:88 1500
Check: Node connectivity of subnet "20.20.20.0"
11gR2 RAC Installation Manual
Source Destination Connected?
--- --- --- goodus1:en14 goodus2:en14 yes Result: Node connectivity passed for subnet "20.20.20.0" with node(s) goodus1,goodus2
Check: TCP connectivity of subnet "20.20.20.0"
Source Destination Connected?
--- --- --- goodus1:20.20.20.1 goodus2:20.20.20.2 passed Result: TCP connectivity check passed for subnet "20.20.20.0"
Check: Node connectivity of subnet "10.10.10.0"
Source Destination Connected?
--- --- --- goodus1:en15 goodus2:en15 yes Result: Node connectivity passed for subnet "10.10.10.0" with node(s) goodus1,goodus2
Check: TCP connectivity of subnet "10.10.10.0"
Source Destination Connected?
--- --- --- goodus1:10.10.10.1 goodus2:10.10.10.2 passed Result: TCP connectivity check passed for subnet "10.10.10.0"
Interfaces found on subnet "20.20.20.0" that are likely candidates for VIP are:
goodus1 en14:20.20.20.1 goodus2 en14:20.20.20.2
Interfaces found on subnet "10.10.10.0" that are likely candidates for a private interconnect are:
goodus1 en15:10.10.10.1 goodus2 en15:10.10.10.2
Result: Node connectivity check passed
Check: Total memory
Node Name Available Required Comment --- --- --- --- goodus1 30.25GB (3.1719424E7KB) 1.5GB (1572864.0KB) passed goodus2 29.875GB (3.1326208E7KB) 1.5GB (1572864.0KB) passed Result: Total memory check passed
Check: Available memory
Node Name Available Required Comment --- --- --- --- goodus1 26.4483GB (2.7733012E7KB) 50MB (51200.0KB) passed goodus2 25.9371GB (2.7197072E7KB) 50MB (51200.0KB) passed Result: Available memory check passed
Check: Swap space
Node Name Available Required Comment --- --- --- --- goodus1 512MB (524288.0KB) 10GB (1.048576E7KB) failed Goodus2 29.75Gb (3.1195136E7Kb) 10Gb (1.048576E7Kb) Passed Result: Swap space check failed
Check: Free disk space for "goodus1:/tmp/"
11gR2 RAC Installation Manual
Path Node Name Mount point Available Required Comment --- --- --- --- --- --- /tmp/ goodus1 /tmp 2.5644GB 1GB passed Result: Free disk space check passed for "goodus1:/tmp/"
Check: Free disk space for "goodus2:/tmp/"
Path Node Name Mount point Available Required Comment --- --- --- --- --- --- /tmp/ goodus2 /tmp 2.8295GB 1GB passed Result: Free disk space check passed for "goodus2:/tmp/"
Check: User existence for "oragrid"
Node Name Status Comment --- --- --- goodus1 exists(301) passed goodus2 exists(301) passed
Checking for multiple users with UID value 301
Result: Check for multiple users with UID value 301 passed Result: User existence check passed for "oragrid"
Check: Group existence for "oinstall"
Node Name Status Comment --- --- --- goodus1 does not exist failed goodus2 does not exist failed Result: Group existence check failed for "oinstall"
Check: Group existence for "dba"
Node Name Status Comment --- --- --- goodus1 exists passed goodus2 exists passed Result: Group existence check passed for "dba"
Check: Membership of user "oragrid" in group "oinstall" [as Primary]
Node Name User Exists Group Exists User in Group Primary Comment --- --- --- --- --- --- goodus1 yes yes no N/A failed goodus2 yes yes no N/A failed Result: Membership check for user "oragrid" in group "oinstall" [as Primary] failed
Check: Membership of user "oragrid" in group "dba"
Node Name User Exists Group Exists User in Group Comment --- --- --- --- --- goodus1 yes yes yes passed goodus2 yes yes yes passed Result: Membership check for user "oragrid" in group "dba" passed
Check: Run level
Node Name run level Required Comment --- --- --- --- goodus1 2 2 passed goodus2 2 2 passed Result: Run level check passed
Check: Hard limits for "maximum open file descriptors"
Node Name Type Available Required Comment --- --- --- --- ---
goodus1 hard 9223372036854775807 65536 passed
11gR2 RAC Installation Manual
goodus2 hard 9223372036854775807 65536 passed Result: Hard limits check passed for "maximum open file descriptors"
Check: Soft limits for "maximum open file descriptors"
Node Name Type Available Required Comment --- --- --- --- ---
goodus1 soft 9223372036854775807 1024 passed goodus2 soft 9223372036854775807 1024 passed Result: Soft limits check passed for "maximum open file descriptors"
Check: Hard limits for "maximum user processes"
Node Name Type Available Required Comment --- --- --- --- --- goodus1 hard 16384 16384 passed goodus2 hard 16384 16384 passed Result: Hard limits check passed for "maximum user processes"
Check: Soft limits for "maximum user processes"
Node Name Type Available Required Comment --- --- --- --- --- goodus1 soft 16384 2047 passed goodus2 soft 16384 2047 passed Result: Soft limits check passed for "maximum user processes"
Check: System architecture
Node Name Available Required Comment --- --- --- --- goodus1 powerpc powerpc passed goodus2 powerpc powerpc passed Result: System architecture check passed
Check: Kernel version
Node Name Available Required Comment --- --- --- --- goodus1 5.3-5300.11.02.1007 5.3-5300.09.01 passed goodus2 5.3-5300.11.02.1007 5.3-5300.09.01 passed Result: Kernel version check passed
Check: Kernel parameter for "SEM_NSEMS_MAX"
Node Name Configured Required Comment --- --- --- --- goodus1 32768 256 passed goodus2 32768 256 passed Result: Kernel parameter check passed for "SEM_NSEMS_MAX"
Check: Kernel parameter for "SEM_VALUE_MAX"
Node Name Configured Required Comment --- --- --- --- goodus1 32767 100 passed goodus2 32767 100 passed Result: Kernel parameter check passed for "SEM_VALUE_MAX"
Check: Kernel parameter for "ncargs"
Node Name Configured Required Comment --- --- --- --- goodus1 128 128 passed goodus2 128 128 passed Result: Kernel parameter check passed for "ncargs"
Check: Package existence for "cluster.license-5.4.1"
11gR2 RAC Installation Manual
Node Name Available Required Comment --- --- --- --- goodus1 cluster.license-5.4.1.1-0 cluster.license-5.4.1 passed goodus2 cluster.license-5.4.1.1-0 cluster.license-5.4.1 passed Result: Package existence check passed for "cluster.license-5.4.1"
Check: Package existence for "bos.adt.base-..."
Node Name Available Required Comment --- --- --- --- goodus1 bos.adt.base-5.3.11.0-0 bos.adt.base-... passed goodus2 bos.adt.base-5.3.11.0-0 bos.adt.base-... passed Result: Package existence check passed for "bos.adt.base-..."
Check: Package existence for "bos.adt.lib-..."
Node Name Available Required Comment --- --- --- --- goodus1 bos.adt.lib-5.3.9.0-0 bos.adt.lib-... passed goodus2 bos.adt.lib-5.3.9.0-0 bos.adt.lib-... passed Result: Package existence check passed for "bos.adt.lib-..."
Check: Package existence for "bos.adt.libm-..."
Node Name Available Required Comment --- --- --- --- goodus1 bos.adt.libm-5.3.11.0-0 bos.adt.libm-... passed goodus2 bos.adt.libm-5.3.11.0-0 bos.adt.libm-... passed Result: Package existence check passed for "bos.adt.libm-..."
Check: Package existence for "bos.perf.libperfstat-5.3.9.0"
Node Name Available Required Comment --- --- --- ---
goodus1 bos.perf.libperfstat-5.3.11.2-0 bos.perf.libperfstat-5.3.9.0 passed goodus2 bos.perf.libperfstat-5.3.11.2-0 bos.perf.libperfstat-5.3.9.0 passed Result: Package existence check passed for "bos.perf.libperfstat-5.3.9.0"
Check: Package existence for "bos.perf.perfstat-..."
Node Name Available Required Comment --- --- --- --- goodus1 bos.perf.perfstat-5.3.11.0-0 bos.perf.perfstat-... passed goodus2 bos.perf.perfstat-5.3.11.0-0 bos.perf.perfstat-... passed Result: Package existence check passed for "bos.perf.perfstat-..."
Check: Package existence for "bos.perf.proctools-..."
Node Name Available Required Comment --- --- --- --- goodus1 bos.perf.proctools-5.3.11.1-0 bos.perf.proctools-... passed goodus2 bos.perf.proctools-5.3.11.1-0 bos.perf.proctools-... passed Result: Package existence check passed for "bos.perf.proctools-..."
Check: Package existence for "rsct.basic.rte-..."
Node Name Available Required Comment --- --- --- --- goodus1 rsct.basic.rte-2.4.12.0-0 rsct.basic.rte-... passed goodus2 rsct.basic.rte-2.4.12.0-0 rsct.basic.rte-... passed Result: Package existence check passed for "rsct.basic.rte-..."
Check: Package existence for "rsct.compat.clients.rte-..."
Node Name Available Required Comment --- --- --- ---
goodus1 rsct.compat.clients.rte-2.4.12.0-0 rsct.compat.clients.rte-... passed goodus2 rsct.compat.clients.rte-2.4.12.0-0 rsct.compat.clients.rte-... passed
11gR2 RAC Installation Manual
Result: Package existence check passed for "rsct.compat.clients.rte-..."
Check: Package existence for "xlC.aix50.rte-10.1.0.0"
Node Name Available Required Comment --- --- --- --- goodus1 xlC.aix50.rte-10.1.0.3-0 xlC.aix50.rte-10.1.0.0 passed goodus2 xlC.aix50.rte-10.1.0.3-0 xlC.aix50.rte-10.1.0.0 passed Result: Package existence check passed for "xlC.aix50.rte-10.1.0.0"
Check: Operating system patch for "Patch IZ42940"
Node Name Applied Required Comment --- --- --- --- goodus1 missing Patch IZ42940 failed goodus2 missing Patch IZ42940 failed Result: Operating system patch check failed for "Patch IZ42940"
Check: Operating system patch for "Patch IZ49516"
Node Name Applied Required Comment --- --- --- --- goodus1 missing Patch IZ49516 failed goodus2 missing Patch IZ49516 failed Result: Operating system patch check failed for "Patch IZ49516"
Check: Operating system patch for "Patch IZ52331"
Node Name Applied Required Comment --- --- --- --- goodus1 missing Patch IZ52331 failed goodus2 missing Patch IZ52331 failed Result: Operating system patch check failed for "Patch IZ52331"
Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed
Check: Current group ID
Result: Current group ID check passed
Checking Core file name pattern consistency...
Core file name pattern consistency check passed.
Checking to make sure user "oragrid" is not in "root" group Node Name Status Comment --- --- --- goodus1 does not exist passed goodus2 does not exist passed Result: User "oragrid" is not part of "root" group. Check passed
Check default user file creation mask
Node Name Available Required Comment --- --- --- --- goodus1 022 0022 passed goodus2 022 0022 passed Result: Default user file creation mask check passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes NTP Configuration file check passed
Checking daemon liveness...
11gR2 RAC Installation Manual
Check: Liveness for "xntpd"
Node Name Running?
--- --- goodus1 yes goodus2 yes Result: Liveness check passed for "xntpd"
Checking NTP daemon command line for slewing option "-x"
Check: NTP daemon command line
Node Name Slewing Option Set?
--- --- goodus1 yes goodus2 yes Result:
NTP daemon slewing option check passed
Checking NTP daemon's boot time configuration, in file "/etc/rc.tcpip", for slewing option "-x"
Check: NTP daemon's boot time configuration
Node Name Slewing Option Set?
--- --- goodus1 yes goodus2 yes Result:
NTP daemon's boot time configuration check for slewing option passed
Result: Clock synchronization check using Network Time Protocol(NTP) passed Result: User ID < 65535 check passed
Result: Kernel 64-bit mode check passed.
8.2.2. Rootpre.sh(root 유저로 반드시 양 노드 모두 수행 )
[root@goodus1:/oracle/INST/grid]$ umask
022 -> root 유저의 umask 는 반드시 022 로 설정 (아닐 경우 설치시 오라클 유저가 클러스터 환경을 인식 못함) [root@goodus1:/oracle/INST/grid]$ sh rootpre.sh
rootpre.sh output will be logged in /tmp/rootpre.out_10-03-18.11:11:54 check_prereq[11]: test: argument expected
Saving the original files in /etc/ora_save_10-03-18.11:11:54....
Copying new kernel extension to /etc....
rootpre.sh[348]: test: argument expected Loading the kernel extension from /etc
Oracle Kernel Extension Loader for AIX
Copyright (c) 1998,1999 Oracle Corporation
Kernel Extension /etc/pw-syscall.64bit_kernel already loaded, unloading it Unconfigured the kernel extension successfully
Unloaded the kernel extension successfully
Successfully loaded /etc/pw-syscall.64bit_kernel with kmid: 0x4717e00 Successfully configured /etc/pw-syscall.64bit_kernel with kmid: 0x4717e00 The kernel extension was successfuly loaded.
rootpre.sh[413]: test: argument expected
Checking if group services should be configured....
Group "hagsuser" does not exist.
Creating required group for group services: hagsuser Please add your Oracle userid to the group: hagsuser
11gR2 RAC Installation Manual
Configuring HACMP group services socket for possible use by Oracle.
[root@goodus1:/oracle/INST/grid]$ vi /etc/group
hagsuser:!:205:oragrid:oracle -> oracle, oragrid 유저 추가
rootpre 재수행
[root@goodus1:/oracle/INST/grid]$ sh rootpre.sh
rootpre.sh output will be logged in /tmp/rootpre.out_10-03-18.15:53:42 check_prereq[11]: test: argument expected
Saving the original files in /etc/ora_save_10-03-18.15:53:42....
Copying new kernel extension to /etc....
rootpre.sh[348]: test: argument expected Loading the kernel extension from /etc
Oracle Kernel Extension Loader for AIX
Copyright (c) 1998,1999 Oracle Corporation
Kernel Extension /etc/pw-syscall.64bit_kernel already loaded, unloading it Unconfigured the kernel extension successfully
Unloaded the kernel extension successfully
Successfully loaded /etc/pw-syscall.64bit_kernel with kmid: 0x3fba400 Successfully configured /etc/pw-syscall.64bit_kernel with kmid: 0x3fba400 The kernel extension was successfuly loaded.
rootpre.sh[413]: test: argument expected
Checking if group services should be configured....
Please confirm your Oracle userid is a member of the group: hagsuser Configuring HACMP group services socket for possible use by Oracle.
2 번 노드도 동일한 방법으로 수행 ( rootpre 는 1 번노드의 rootpre dir 을 tar 로 묶어서 전송) 정상적으로 수행되면, /opt/ORCLcluster 디렉토리가 생성되며 , 수행시
Checking if group services should be configured....
Nothing to configure. 으로 종료되면, Install 시 node information 단계에서 양 노드가 안보이는 문제 발생함.
HACMP 와 ORACLE CLUSTERWARE 를 연동해주는 Library 생성됨 [oracle@goodus1:/oracle/db]$ ls -l /opt/ORCLcluster/lib
lrwxrwxrwx 1 root system 33 Mar 18 15:53 libskgxn2.so@ -> /opt/ORCLcluster/lib/libskgxnr.so*
-rwxr-xr-x 1 root system 208646 Mar 18 15:53 libskgxnr.so*
11gR2 RAC Installation Manual
8.2.3. GRID Infrastructure Software Installation
11gR2 부터는 Clusterware 가 Grid infrastructure 내에 포함되어 있다.
(oragrid 사용자로 goodus1 에서 수행합니다) . [oracle@goodus1:/oracle/INST/grid/install]$ ./lsnodes -n goodus1 1
goodus2 2
[oragrid@goodus1:/oracle/INST/grid]$ export DISPLAY=10.25.235.120:1 [oragird@goodus1:/oracle/INST/grid]$ chmod +x run*
[oragrid@goodus1:/oracle/INST/grid]$ chmod +x ./install/*
[oragrid@goodus1:/oracle/INST/grid]$ chmod +x ./install/.oui [oragrid@goodus1:/oracle/INST/grid]$ ./runInstaller
********************************************************************************
Your platform requires the root user to perform certain pre-installation OS preparation. The root user should run the shell script 'rootpre.sh' before you proceed with Oracle installation. rootpre.sh can be found at the top level of the CD or the stage area.
Answer 'y' if root has run 'rootpre.sh' so you can proceed with Oracle installation.
Answer 'n' to abort installation and then ask root to run 'rootpre.sh'.
********************************************************************************
Has 'rootpre.sh' been run by root? [y/n] (n) y
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 190 MB. Actual 3778 MB Passed Checking swap space: must be greater than 150 MB. Actual 30720 MB Passed Checking monitor: must be configured to display at least 256 colors
주) 설치화면에서 일부 서버 정보는 삭제하였음을 양해 바랍니다.
11gR2 RAC Installation Manual
11gR2 RAC Installation Manual
SCAN_NAME 에 지정된 이름이 DNS Server or /etc/hosts 파일에 반드시 존재해야함
SSH Connectivity->Setup 을 클릭하면 ssh 환경을 자동으로 구성해 줌.
ssh goodus1, goodus2 수행시 password 인증 없이 접속 가능해야 에러 발생하지 않음(local 에도 접속 가능해야함)
11gR2 RAC Installation Manual
반드시 양 노드의 NIC Interface 명은 용도별로 동일해야 함.
11gR2 RAC Installation Manual
OCR 과 VOTE 를 구성하기 위한 Diskgroup 생성 (Disk 가 1 개뿐이므로, External Redundancy 선택)
ASM Instance Password 는 오라클에서 요구하는 복잡성을 만족할 것을 권고 (8 자리, 특수문자, 숫자 포함)
11gR2 RAC Installation Manual
Password 는 오라클에서 요구하는 복잡성을 만족할 것을 권고 (8 자리, 특수문자, 숫자 포함)
11gR2 RAC Installation Manual
ASM 을 관리하기 위한 별도의 OS Group 을 생성할 것을 권고하나, 설치 후 RAWDEVICE 로 전환 예정임으로 Skip
11gR2 RAC Installation Manual
Grid 는 $ORACLE_BASE 의 하위 dir 에 설치할 수 없음
이 단계에서 [INS-20702] Unexpected internal driver error가 발생할 경우 설치를 종료하고 runInstaller -ignoreInternalDriverError 옵션으로 설치 재 수행
11gR2 RAC Installation Manual
해당 OS Package 는 하위 버전 Package 이므로, ignore all 에 체크하여 진행.
만약 OS Parameter 설정에 문제가 있을 경우 Fix&Check Again 을 클릭하면, OS 수정 스크립트 자동 생성됨.
실제로는 Cluster name, SCAN Name 등이 모두 정상적으로 표시됨.
11gR2 RAC Installation Manual
반드시 순차적으로 root 유저로 goodus1ègoodus2 순서로
/oracle/oraInventory/orainstRoot.sh -> /oracle/grid/root.sh 를 수행 완료 후에 OK 버튼을 Click 합니다.
11gR2 RAC Installation Manual
[root@goodus1 ~]# /oracle/oraInventory/orainstRoot.sh Changing permissions of /oracle/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /oracle/oraInventory to dba.
The execution of the script is complete.
[root@goodus2 ~]# /oracle/oraInventory/orainstRoot.sh Changing permissions of /oracle/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /oracle/oraInventory to dba.
The execution of the script is complete.
goodus1:/root># /oracle/grid/root.sh Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oragrid ORACLE_HOME= /oracle/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-03-24 17:14:59: Parsing the host name
2010-03-24 17:14:59: Checking for super user privileges 2010-03-24 17:14:59: User has super user privileges
Using configuration parameter file: /oracle/grid/crs/install/crsconfig_params Creating trace directory
stty: tcgetattr: Not a typewriter
User oragrid has the required capabilities to run CSSD in realtime mode LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'system'..
Operation successful.
root wallet root wallet cert root cert export peer wallet
profile reader wallet pa wallet
peer wallet keys pa wallet keys peer cert request pa cert request peer cert pa cert
peer root cert TP
profile reader root cert TP pa root cert TP
peer pa cert TP pa peer cert TP
profile reader pa cert TP profile reader peer cert TP peer user cert
pa user cert
Adding daemon to inittab
11gR2 RAC Installation Manual
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start 'ora.gipcd' on 'goodus1' CRS-2672: Attempting to start 'ora.mdnsd' on 'goodus1' CRS-2676: Start of 'ora.gipcd' on 'goodus1' succeeded CRS-2676: Start of 'ora.mdnsd' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'goodus1' CRS-2676: Start of 'ora.gpnpd' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'goodus1' CRS-2676: Start of 'ora.cssdmonitor' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'goodus1' CRS-2672: Attempting to start 'ora.diskmon' on 'goodus1' CRS-2676: Start of 'ora.diskmon' on 'goodus1' succeeded CRS-2676: Start of 'ora.cssd' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'goodus1' CRS-2676: Start of 'ora.ctssd' on 'goodus1' succeeded ASM created and started successfully.
DiskGroup DGDATA created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'system'..
Operation successful.
CRS-2672: Attempting to start 'ora.crsd' on 'goodus1' CRS-2676: Start of 'ora.crsd' on 'goodus1' succeeded CRS-4256: Updating the profile
Successful addition of voting disk a7b39f2bdee54f5ebf32bd638daa1426.
Successfully replaced voting disk group with +DGDATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group -- --- --- --- ---
1. ONLINE a7b39f2bdee54f5ebf32bd638daa1426 (/dev/rhdisk154) [DGDATA]
Located 1 voting disk(s).
CRS-2673: Attempting to stop 'ora.crsd' on 'goodus1' CRS-2677: Stop of 'ora.crsd' on 'goodus1' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'goodus1' CRS-2677: Stop of 'ora.asm' on 'goodus1' succeeded CRS-2673: Attempting to stop 'ora.ctssd' on 'goodus1' CRS-2677: Stop of 'ora.ctssd' on 'goodus1' succeeded CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'goodus1' CRS-2677: Stop of 'ora.cssdmonitor' on 'goodus1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'goodus1' CRS-2677: Stop of 'ora.cssd' on 'goodus1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'goodus1' CRS-2677: Stop of 'ora.gpnpd' on 'goodus1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'goodus1' CRS-2677: Stop of 'ora.gipcd' on 'goodus1' succeeded CRS-2673: Attempting to stop 'ora.mdnsd' on 'goodus1' CRS-2677: Stop of 'ora.mdnsd' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.mdnsd' on 'goodus1' CRS-2676: Start of 'ora.mdnsd' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'goodus1' CRS-2676: Start of 'ora.gipcd' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'goodus1' CRS-2676: Start of 'ora.gpnpd' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'goodus1' CRS-2676: Start of 'ora.cssdmonitor' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'goodus1' CRS-2672: Attempting to start 'ora.diskmon' on 'goodus1' CRS-2676: Start of 'ora.diskmon' on 'goodus1' succeeded CRS-2676: Start of 'ora.cssd' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'goodus1' CRS-2676: Start of 'ora.ctssd' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'goodus1' CRS-2676: Start of 'ora.asm' on 'goodus1' succeeded
11gR2 RAC Installation Manual
CRS-2672: Attempting to start 'ora.crsd' on 'goodus1' CRS-2676: Start of 'ora.crsd' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.evmd' on 'goodus1' CRS-2676: Start of 'ora.evmd' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'goodus1' CRS-2676: Start of 'ora.asm' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.DGDATA.dg' on 'goodus1' CRS-2676: Start of 'ora.DGDATA.dg' on 'goodus1' succeeded
goodus1 2010/03/24 17:22:26 /oracle/grid/cdata/goodus1/backup_20100324_172226.olr Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 30464 MB Passed The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /oracle/oraInventory 'UpdateNodeList' was successful.
[root@goodus2:/oracle]$ /oracle/grid/root.sh Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oragrid ORACLE_HOME= /oracle/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-03-25 11:18:02: Parsing the host name
2010-03-25 11:18:02: Checking for super user privileges 2010-03-25 11:18:02: User has super user privileges
Using configuration parameter file: /oracle/grid/crs/install/crsconfig_params stty: tcgetattr: Not a typewriter
User oragrid has the required capabilities to run CSSD in realtime mode LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'system'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node goodus1, number 1, and is terminating
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'goodus2' CRS-2677: Stop of 'ora.cssdmonitor' on 'goodus2' succeeded
An active cluster was found during exclusive startup, restarting to join the cluster CRS-2672: Attempting to start 'ora.mdnsd' on 'goodus2'
CRS-2676: Start of 'ora.mdnsd' on 'goodus2' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'goodus2' CRS-2676: Start of 'ora.gipcd' on 'goodus2' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'goodus2' CRS-2676: Start of 'ora.gpnpd' on 'goodus2' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'goodus2' CRS-2676: Start of 'ora.cssdmonitor' on 'goodus2' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'goodus2' CRS-2672: Attempting to start 'ora.diskmon' on 'goodus2' CRS-2676: Start of 'ora.diskmon' on 'goodus2' succeeded CRS-2676: Start of 'ora.cssd' on 'goodus2' succeeded
11gR2 RAC Installation Manual
CRS-2672: Attempting to start 'ora.ctssd' on 'goodus2' CRS-2676: Start of 'ora.ctssd' on 'goodus2' succeeded CRS-2672: Attempting to start 'ora.asm' on 'goodus2' CRS-2676: Start of 'ora.asm' on 'goodus2' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'goodus2' CRS-2676: Start of 'ora.crsd' on 'goodus2' succeeded CRS-2672: Attempting to start 'ora.evmd' on 'goodus2' CRS-2676: Start of 'ora.evmd' on 'goodus2' succeeded
goodus2 2010/03/25 11:20:18 /oracle/grid/cdata/goodus2/backup_20100325_112018.olr Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 30464 MB Passed The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /oracle/oraInventory 'UpdateNodeList' was successful.
앞에서 ignore 하고 넘어간 OS Package 로 인해 오류 발생. 무시하고 진행.
11gR2 RAC Installation Manual
설치를 종료합니다.
11gR2 RAC Installation Manual
[oracle@goodus1:/oracle/grid/bin]$ ps -ef | grep d.bin
root 102490 1 6 11:26:07 - 0:00 /oracle/grid/bin/crsd.bin reboot root 168070 1 3 11:26:08 - 0:00 /oracle/grid/bin/oclskd.bin oragrid 172222 1 0 11:25:30 - 0:00 /oracle/grid/bin/diskmon.bin -d -f oragrid 188574 1 0 11:25:55 - 0:00 /oracle/grid/bin/oclskd.bin oragrid 237576 1 0 11:25:25 - 0:00 /oracle/grid/bin/mdnsd.bin
root 254100 1 0 11:25:49 - 0:00 /oracle/grid/bin/octssd.bin reboot oragrid 274678 1 2 11:25:27 - 0:00 /oracle/grid/bin/gpnpd.bin
root 282714 1 0 11:25:28 - 0:00 /oracle/grid/bin/cssdmonitor oragrid 291002 1 0 11:25:24 - 0:00 /oracle/grid/bin/gipcd.bin root 115108 1 0 11:25:30 - 0:00 /oracle/grid/bin/cssdagent oragrid 180648 127154 0 11:26:10 pts/1 0:00 grep d.bi
root 229850 1 0 11:25:30 - 0:00 /oracle/grid/bin/orarootagent.bin oragrid 368946 434436 1 11:26:10 - 0:00 /oracle/grid/bin/evmlogger.bin -o /oracle/grid/evm/log/evmlogger.info -l /oracle/grid/evm/log/evmlogger.log
oragrid 373232 409880 4 11:25:30 - 0:00 /oracle/grid/bin/ocssd.bin oragrid 397654 1 0 11:25:24 - 0:00 /oracle/grid/bin/oraagent.bin root 409880 1 0 11:25:30 - 0:00 /bin/sh /oracle/grid/bin/ocssd root 426386 1 0 11:25:21 - 0:02 /oracle/grid/bin/ohasd.bin reboot oragrid 434436 1 13 11:25:50 - 0:00 /oracle/grid/bin/evmd.bin
[oracle@goodus1:/oracle/grid/bin]$ ifconfig -a en14:
flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM _OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
inet 20.20.20.1 netmask 0xffffff00 broadcast 20.20.20.15 -> public inet 20.20.20.3 netmask 0xffffff00 broadcast 20.20.20.15 -> vip inet 20.20.20.5 netmask 0xffffff00 broadcast 20.20.20.15 -> scan-vip tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
en15:
flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM _OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
inet 10.10.10.1 netmask 0xffffff00 broadcast 10.10.10.255 -> private tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
lo0: flags=e08084b<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT>
inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255 inet6 ::1/0
tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1 goodus1:/root># /oracle/grid/bin/crs_stat -t
Name Type Target State Host --- ora.DATA.dg ora....up.type ONLINE ONLINE goodus1
ora....N1.lsnr ora....er.type ONLINE ONLINE goodus1 -> SCAN_LISTENER ora.asm ora.asm.type ONLINE ONLINE goodus1
ora.eons ora.eons.type ONLINE ONLINE goodus1 ora.gsd ora.gsd.type OFFLINE OFFLINE ora....SM1.asm application ONLINE ONLINE goodus1 ora.goodus1.gsd application OFFLINE OFFLINE ora.goodus1.ons application ONLINE ONLINE goodus1 ora.goodus1.vip ora....t1.type ONLINE ONLINE goodus1 ora....SM2.asm application ONLINE ONLINE goodus2 ora.goodus2.gsd application OFFLINE OFFLINE ora.goodus2.ons application ONLINE ONLINE goodus2 ora.goodus2.vip ora....t1.type ONLINE ONLINE goodus2 ora....network ora....rk.type ONLINE ONLINE goodus1 ora.oc4j ora.oc4j.type OFFLINE OFFLINE ora.ons ora.ons.type ONLINE ONLINE goodus1
ora.scan1.vip ora....ip.type ONLINE ONLINE goodus1->1 개의 SCAN IP 만 지정함으로 SCAN VIP 도 1 개 생성
11gR2 RAC Installation Manual
8.3. Database Software Installation
(oracle 사용자로 goodus1 에서 수행합니다.) [oracle@goodus1:/oracle/INST/database]$ id
uid=300(oracle) gid=300(dba) groups=0(system),1(staff),3(sys),4(adm),205(hagsuser) [oracle@goodus1:/oracle/INST/database]$ export DISPLAY=10.25.235.120:1 [oracle@goodus1:/oracle/INST/database]$ chmod +x run*
[oracle@goodus1:/oracle/INST/database]$ chmod +x ./install/*
[oracle@goodus1:/oracle/INST/database]$ chmod +x ./install/.oui [oracle@goodus1:/oracle/INST/database]$ ./runInstaller
********************************************************************************
Your platform requires the root user to perform certain pre-installation OS preparation. The root user should run the shell script 'rootpre.sh' before you proceed with Oracle installation. rootpre.sh can be found at the top level of the CD or the stage area.
Answer 'y' if root has run 'rootpre.sh' so you can proceed with Oracle installation.
Answer 'n' to abort installation and then ask root to run 'rootpre.sh'.
********************************************************************************
Has 'rootpre.sh' been run by root? [y/n] (n) y
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 190 MB. Actual 3465 MB Passed Checking swap space: must be greater than 150 MB. Actual 30720 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 65536 Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2010-03-18_04-36-49PM. Please wait ...
11gR2 RAC Installation Manual
Critical Patch Update 시 공지를 위한 e-mail 주소 등록 요청 – 무시하고 진행
11gR2 RAC Installation Manual
반드시 양 노드를 모두 선택
11gR2 RAC Installation Manual
Select Options 를 클릭해서 필요한 옵션을 선택
11gR2 RAC Installation Manual
$ORACLE_HOME 은 $ORACLE_BASE 하위에 지정되어야 함
이 단계에서 [INS-20702] Unexpected internal driver error가 발생할 경우 설치를 종료하고 runInstaller -ignoreInternalDriverError 옵션으로 설치 재 수행
11gR2 RAC Installation Manual
1
11gR2 RAC Installation Manual
11gR2 RAC Installation Manual
root 사용자로 goodus1 è goodus2 순서로 /oracle/db/11g/root.sh 수행 완료 후 OK 버튼을 Click 합니다.
Goodus1:/oracle/db/11g># sh root.sh Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle ORACLE_HOME= /oracle/db/11g
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
설치 완료
8.4. Check Public/Inter-connect Configuration
설치가 완료되면 현재 OCR 에 등록된 Public, Private(Interconnect) ip 구성 정보를 확인합니다.
Oifcfg getif 조회 값과 x$ksxpia 조회값이 반드시 올바른 값으로 일치해야 합니다.
만약 이 값이 없거나, 값이 다르면 Failover 시 문제가 발생하게 됨을 유의하십시요.
특히 interconnect 값이 틀리면 public ip 로 interconnect 을 수행하므로, Failover 시 문제가 발생하게 됩니다.
(oracle 사용자로 양 노드 수행합니다.) [oracle@goodus1:/oracle/db]$ oifcfg getif en14 20.20.20.0 global public
en15 10.10.10.0 global cluster_interconnect
