Oracle 12c in with ASM and iSCSI on Oracle Linux

Share Button

This Article will guide you step-by-step how to set a Oracle 12 Database with ASM and iSCSI mounted disks in Oracle linux. This in quite straight forward and the only thing that I consider to be a hazel is mounting the iSCSI devices in a good manner. First we have to install the OS, but Im not going to cover that more than with a few words and mostly screenshots, since installing this should be easy for most. The thing to keep in mind is the host name, since the oracle installation needs to be able to lookup the local hostname (from DNS which is preferable in production or in the /etc/hosts file, my choice) . In my case my Linux is running as a virtual machine on VMWare

  • Hostname:savdb01.savecore.se
  • IP:192.168.110.25
  • Netmask: 255.255.255.0
  • GW:192.168.110.1

First you have to choose language for the install Then: Just install the basic drivers, I’ll add the iSCSI support later on Also set IP by choosing Configure Network, and input your IP/Netmask and Gateway. Also choose AUTO CONNECT!! If you are using a virutal machine for this, then Use all space is your choice, if you are afraid of losing any data choose Replace Existing Linux System(s). In my case i used Create Custom Layout to set up my partiotioning as I please, because Oracle Linux uses LVM as default and I don’t feel comfortable working with that Installation done! If you are like me, and failed to set the default Gateway from the GUI you need to adding

GATEWAY=192.168.110.1

(or what ever you have) in the /etc/sysconfig/network-scripts/ifcfg-eth0 to look something like this:

DEVICE=eth0
TYPE=Ethernet
UUID=be4bfbb4-2b09-44f9-9f2c-a3e4ba674ec5
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
HWADDR=00:50:56:9E:8E:D8
IPADDR=192.168.110.25
GATEWAY=192.168.110.1
PREFIX=24
DNS1=192.168.100.14
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"

and restart the network by /etc/init.d/network restart Once we’ve installed our Oracle Linux as a BASIC SERVER w/o KDE/GNOME etc and have the hostname/IP address configured. We can installing the Database related stuff: First turn off the firewall (or configure it for all the DB exceptions) since this is for testing only!.

[root@savdb01 ~]# service iptables stop
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Unloading modules:                               [  OK  ]
[root@savdb01 ~]# chkconfig iptables off
[root@savdb01 ~]# 

Install Oracle 12c preinstall package with all packages and some settings and some users (oracle, oinstall).

yum install -y oracle-rdbms-server-12cR1-preinstall

Then we have to add the necessary groups for grid and add a few groups to the oracle user, and the the grid user is self.

 

Note that the oracle user will not be used in this article, since we only will install the grid component, but I know i will install oracle Database on here as well so.
   groupadd asmadmin
   groupadd asmdba
   groupadd asmoper
   groupadd oper
   groupadd dba
   useradd -g oinstall -G asmoper,asmadmin,asmdba,dba -m grid
   usermod -G dba,oper,asmdba -m oracle
   passwd grid
   passwd oracle

Now we have all the prerequisite for ASM and the Database. No we need to configure the iSCSI.

yum install iscsi-initiator-utils

Configuring the iSCSI Initiator to Start on Reboot
Once the iSCSI Initiator package is installed, configure the package to start on reboot by completing the following steps: Check that the iSCSI initiator will start when the system is rebooted:

[root@savdb01 ~]# chkconfig --list iscsi
iscsi      	0:off	1:off	2:off	3:on	4:on	5:on	6:off
[root@savdb01 ~]# chkconfig --list iscsid
iscsid     	0:off	1:off	2:off	3:on	4:on	5:on	6:off

This example shows that both the iscsi and iscsid services are set to off at all run levels, so the iSCSI Initiator will not start at system reboot. If the settings at run level 3, 4, and 5 are not set to on for either the iscsi or iscsid service, enable them, as shown below:

chkconfig iscsi on
chkconfig iscsid on
[root@oel ~]# chkconfig --list iscsi
iscsi 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[root@oel ~]# chkconfig --list iscsid
iscsid 0:off 1:off 2:on 3:on 4:on 5:on 6:off
service iscsi start
service iscsid start

The iscsi and iscsid services are now set to start at system reboot. Next we have to identify the Host IQN. The Host IQN will be used by(most) the SAN and was auto-generated when we installed the initiator package.

cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1988-12.com.oracle:f5cf203ccf68

Then we need to the SAN to give us some big LUNs. This is done in many ways but if you like me are using an Oracle ZFS Storage Appliance you can follow this guide from Oracle Now we can try to identify the LUNs that we published before. First make sure the iscsi is started

service iscsi start
iscsiadm -m discovery -t sendtargets -p 192.168.110.210
192.168.110.210:3260,2 iqn.1986-03.com.sun:02:c49183bc-8519-642a-c5ff-fa01e9b2bd79
iscsiadm -m node -p 192.168.110.210 -l
tail /var/log/messages
...
..
.
Apr 30 13:56:26 savdb01 kernel: sd 3:0:0:2: [sdd] Attached SCSI disk
Apr 30 13:56:26 savdb01 kernel: sd 3:0:0:5: [sdg] Attached SCSI disk
Apr 30 13:56:26 savdb01 kernel: sd 3:0:0:6: [sdh] Attached SCSI disk
Apr 30 13:56:26 savdb01 kernel: sd 3:0:0:4: [sdf] Attached SCSI disk
Apr 30 13:56:26 savdb01 kernel: sd 3:0:0:3: [sde] Attached SCSI disk
Apr 30 13:56:26 savdb01 kernel: sd 3:0:0:7: [sdi] Attached SCSI disk
Apr 30 13:56:26 savdb01 kernel: sd 3:0:0:1: [sdc] Attached SCSI disk

 

Oracle has its own resolution for generating block device files especially for ASM, called ASMLib. This indeed works well, but how Oracle did to generate the devices, and how to troubleshoot the generation of the devices is for me utterly confusing, and i have not found any documentation on this. I’ve managed to mess my lab servers a few times, which is the reason I don’t like ASMLib, and do not feel comfortable with ASMLib. More on that can be read on ORAinternals/Riyaj Shamsudeen

When we’ve verified that the disks have attached we need to prep the disks using fdisk:

fdisk /dev/sdb
fdisk /dev/sdc
fdisk /dev/sdd
fdisk /dev/sde
fdisk /dev/sdf
-->"n", "p", "1", "Return", "Return" and "w"

You should now see the raw device and the partition we created:

<strong>Next up we need to create the UDEV Rules</strong>

To configure SCSI devices as trusted add
1options=-g

to the file /etc/scsi_id.config (note! that this file is probably missing so yu have to create it)

[root@savdb01 ~]# scsi_id -g -u -d /dev/sdb;scsi_id -g -u -d /dev/sdc;scsi_id -g -u -d /dev/sdd;scsi_id -g -u -d /dev/sde;scsi_id -g -u -d /dev/sdf;
3600144f0ebda77b600005268fa770003
3600144f0ebda77b600005268fa470002
3600144f0ebda77b600005268fa8c0004
3600144f0ebda77b600005268fa9f0005
3600144f0ebda77b600005280e58d0006
[root@savdb01 ~]# 

This is the information needed to create your udev rules to map the SCSI devices. Create the file /etc/udev/rules.d/99-asmdevices.rules and add the YOUR SCSI_id by replacing the RESULT part.

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="3600144f0ebda77b600005268fa770003", NAME="asm-disk1", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="3600144f0ebda77b600005268fa470002", NAME="asm-disk2", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="3600144f0ebda77b600005268fa8c0004", NAME="asm-disk3", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="3600144f0ebda77b600005268fa9f0005", NAME="asm-disk4", OWNER="grid", GROUP="oinstall", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="3600144f0ebda77b600005280e58d0006", NAME="asm-disk5", OWNER="grid", GROUP="oinstall", MODE="0660"

Then you can either reboot or do:

partprobe /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1

Then we test, reload,start Udev and the disks should all pop up under /dev/

[root@savdb01 ~]# udevadm test /block/sdb/sdb1
[root@savdb01 ~]# udevadm control --reload-rule
[root@savdb01 ~]# start_udev
Startar udev:                                              [     OK     ]
[root@savdb01 ~]# ls -al /dev/asm*
brw-rw----. 1 grid oinstall 8,  17 30 apr 15.34 /dev/asm-disk1
brw-rw----. 1 grid oinstall 8,  33 30 apr 15.34 /dev/asm-disk2
brw-rw----. 1 grid oinstall 8,  49 30 apr 15.34 /dev/asm-disk3
brw-rw----. 1 grid oinstall 8,  65 30 apr 15.34 /dev/asm-disk4
brw-rw----. 1 grid oinstall 8,  81 30 apr 15.34 /dev/asm-disk5

Thats the OS part, all disks are now ready to be used by ASM Install Oracle Grid Infrastructure First we need to create a folder structure, i guess you all have your favorite structure, in this case I’ll go for /oracle as $ORACLE_BASE.

[root@savdb01 ~]# mkdir -p /oracle/app/grid/gi12101
[root@savdb01 ~]# chown -R grid:oinstall /oracle/app/grid
[root@savdb01 ~]# mkdir -p /oracle/app/oracle/product/12.1.0.1/dbhome_1
[root@savdb01 ~]# chown -R oracle:oinstall /oracle/app/oracle/
[root@savdb01 ~]# chmod -R 770 /oracle/app/oracle/
[root@savdb01 ~]# chmod -R 770 /oracle/app/grid

We also have to create the grid user profile. To do that add the following to your profile (~/.bash_profile)

ORACLE_BASE=/oracle/app
ORACLE_HOME=$ORACLE_BASE/grid/gi12101
ORACLE_SID=+ASM
LD_LIBRARY_PATH=$ORACLE_HOME/lib
PATH=$PATH:/usr/local/bin:/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:$ORACLE_HOME/bin
export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH

Now we download the grid binaries and unzip them in the grid user_home. Then we can run ./runInstaller. There is however some pre req. you need to pass first, memory part should be OK but you might run into trouble with the DISPLAY. You could write a book on this but, the easiest was is to connect from a Linux or Mac (with XQuartz) and use the ssh option -X(that is capital X) to enable x forwarding like:

[andre-mac:~ macora] ssh -X grid@192.168.110.25

Now you probably forgot about the DNS or hosts entry. Add something like:

192.168.110.25 savdb01 savdb01.savecore.se savdb01.localdomain

to /etc/hosts or setup your DNS accordingly. Here we have change the discovery path to what se configured in the udev rules /dev/asm* And now w should see all (in this case) 5 disks as candidates. Select the disk and choose Normal (or extern for no ASM redundancy)  For simplicity same password everywhere If you have create the groups OK they should be like this by default: Settings accoring to ORACLE_BASE and ORACLE_HOME for grid user profile oraInventory in ORACLE_BASE Skip this unless you have suders setup accordingly. You will probably have som issues here. Memory in this case is fine its a test system and just under the limit (4GB). The other can be fixed by “FIX and CHECK again” Run the script above and then press OK This can be ignored, by Clicking Ignore and then Next Yes (ffa) Review and install!

[root@savdb01 ~]# /oracle//oraInventory/orainstRoot.sh
Changing permissions of /oracle//oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /oracle//oraInventory to oinstall.
The execution of the script is complete.
[root@savdb01 ~]# /oracle/app/grid/gi12101/root.sh
Performing root user operation for Oracle 12c 

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /oracle/app/grid/gi12101

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/app/grid/gi12101/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node savdb01 successfully pinned.
2014/05/02 01:18:38 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

savdb01     2014/05/02 01:19:05     /oracle/app/grid/gi12101/cdata/savdb01/backup_20140502_011905.olr
2014/05/02 01:19:06 CLSRSC-327: Successfully configured Oracle Grid Infrastructure for a Standalone Server

[root@savdb01 ~]# 

Run the root script when prompted. Finally, this is what we want to see :) #Done<

Share Button
Likes(1)Dislikes(0)
0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>