Oracle Linux 6.3 with Oracle 11.2.0.3 single-instance with 4k ASM

This cookbook is part of the Advanced Cookbooks series, for installing Oracle on Violin using a 4k-based ASM diskgroup. If you are unsure what that means, do not proceed any further until you have read this. Configuring ASM to use 4k sectors is interesting, but not necessary – you can achieve excellent performance by following one of the Simple Cookbooks with the added benefit that your installations will take a lot less time…

Today I need to build an Oracle 11g Release 2 database running on Oracle Linux 6 and Violin Memory flash storage, so I thought I’d capture the install process and add it as another installation cookbook.

I have already installed Oracle Linux 6 Update 3 on my IBM X Series server:

[root@half-server4 ~]# cat /etc/oracle-release
Oracle Linux Server release 6.3
[root@half-server4 ~]# uname -s -n -r -m
Linux half-server4 2.6.39-200.24.1.el6uek.x86_64 x86_64

I don’t really need the iptables firewall configured for my testing so I’ve disabled it. If you are wondering why the server is named “half-server4” then it’s because it happens to reside in a half-sized rack. I can assure you that the server itself is entirely whole!

Setup YUM and Oracle Preinstall

In order to install the Oracle Database software I first need to ensure all of the relevant OS packages are installed as well as set various kernel parameters and other configuration settings. In Oracle Linux 5 this was easy to achieve using the oracle-validated package, but in OL6 this no longer exists. Don’t worry though, it hasn’t been removed – only renamed. It’s now called the Oracle Preinstall RPM. If you are installing Oracle Linux 6 then this is the solution – however, if you are installing Red Hat 6, you have been left out in the cold because the Oracle Preinstall package does not work with RHEL6. It’s not that difficult to fix this issue though and I have a step by step guide posted here.

To use Oracle Preinstall, I first need to configure the yum package management tool. Oracle customers with Unbreakable Linux Network (ULN) support contracts can configure yum to use the ULN yum repositories. Those without ULN access can still use yum because Oracle, to its credit, provides public yum repositories (details here).

In previous cookbooks I have used the public yum servers so, for a change, on this occasion I will use the ULN repositories. The first thing to do here is register the system with ULN, which in Oracle Linux 5 we did using the up2date command:

[root@half-server4 ~]# up2date --register
-bash: up2date: command not found

Oh dear. Is this bad? No, it’s just that in Oracle Linux 6 we no longer use up2date… instead we have to use the new uln_register command:

[root@half-server4 ~]# uln_register

This brings up a quasi-graphical screen in which I can enter the details of my Oracle Single-Sign On, password and CSI. It then steps through a load of screens, for most of which I just choose the default answers, then says it’s ready to upload the system details to ULN and register the server. I give it my blessing and (after a lengthy delay) I get the message:

# Note: yum-rhn-plugin has been enabled.
#
# Please review the subscription details below:
#
# Software channel subscriptions:
#  This system will receive updates from the following Unbreakable Linux Network software channels:
#  Oracle Linux 6 Latest (x86_64)
#  Latest Unbreakable Enterprise Kernel for Oracle Linux 6 (x86_64)

This means I can now run yum against the ULN repositories. And to prove it let’s do just that:

[root@half-server4 ~]# yum list \*preinstall\*
Loaded plugins: rhnplugin, security
Available Packages
oracle-rdbms-server-11gR2-preinstall.x86_64       1.0-6.el6      ol6_x86_64_latest

This is the package I need to install, so I will go ahead and run yum with the install option:

[root@half-server4 ~]# yum install oracle-rdbms-server-11gR2-preinstall -y

This creates a huge amount of output as it decides that I need to have 21 new packages installed plus another 2 updated. I won’t bore you by spooling the whole lot here, but this is the positive outcome we want to see (the dependencies may vary but the “Complete!” line remains reassuringly constant):

Installed:
  oracle-rdbms-server-11gR2-preinstall.x86_64 0:1.0-6.el6

Dependency Installed:
  cloog-ppl.x86_64 0:0.15.7-1.2.el6        compat-libcap1.x86_64 0:1.10-1             compat-libstdc++-33.x86_64 0:3.2.3-69.el6
  cpp.x86_64 0:4.4.6-4.el6                 gcc.x86_64 0:4.4.6-4.el6                   gcc-c++.x86_64 0:4.4.6-4.el6
  glibc-devel.x86_64 0:2.12-1.80.el6_3.6   glibc-headers.x86_64 0:2.12-1.80.el6_3.6   kernel-uek-headers.x86_64 0:2.6.32-300.39.2.el6uek
  ksh.x86_64 0:20100621-16.el6             libXmu.x86_64 0:1.0.5-1.el6                libXxf86dga.x86_64 0:1.1.1-1.el6
  libXxf86misc.x86_64 0:1.0.2-1.el6        libaio-devel.x86_64 0:0.3.107-10.el6       libdmx.x86_64 0:1.1.0-1.el6
  libstdc++-devel.x86_64 0:4.4.6-4.el6     mpfr.x86_64 0:2.4.1-6.el6                  ppl.x86_64 0:0.10.2-11.el6
  xorg-x11-utils.x86_64 0:7.4-8.el6        xorg-x11-xauth.x86_64 1:1.0.2-7.1.el6

Dependency Updated:
  glibc.x86_64 0:2.12-1.80.el6_3.6                                  glibc-common.x86_64 0:2.12-1.80.el6_3.6

Complete!

The Oracle preinstallation requirements are now almost complete. I need to set the password of the newly-created oracle user (run “passwd oracle” as the root user) and then it’s time to setup the storage.

Install Device Mapper and Multipathing Daemon

Next I need to make sure the device mapper and multipathing daemon packages are installed:

[root@half-server4 ~]# yum list device-mapper device-mapper-multipath
Loaded plugins: rhnplugin, security
Installed Packages
device-mapper.x86_64              1.02.74-10.el6       @anaconda-OracleLinuxServer.x86_64/6.3
Available Packages
device-mapper.x86_64              1.02.74-10.el6_3.3   ol6_x86_64_latest
device-mapper-multipath.x86_64    0.4.9-56.el6_3.1     ol6_x86_64_latest

The device mapper multipath daemon isn’t installed, so let’s install it (and update the device mapper package while we are there):

[root@half-server4 ~]# yum install device-mapper device-mapper-multipath -y
<...snip!...>
Installed:
  device-mapper-multipath.x86_64 0:0.4.9-56.el6_3.1

Dependency Installed:
  device-mapper-multipath-libs.x86_64 0:0.4.9-56.el6_3.1

Updated:
  device-mapper.x86_64 0:1.02.74-10.el6_3.3

Dependency Updated:
  device-mapper-event.x86_64 0:1.02.74-10.el6_3.3     device-mapper-event-libs.x86_64 0:1.02.74-10.el6_3.3
  device-mapper-libs.x86_64 0:1.02.74-10.el6_3.3      kpartx.x86_64 0:0.4.9-56.el6_3.1

Complete!

Now it’s installed I need to configure it to start when the server boots:

[root@half-server4 ~]# chkconfig --list multipathd
multipathd      0:off   1:off   2:off   3:off   4:off   5:off   6:off
[root@half-server4 ~]# chkconfig multipathd on
[root@half-server4 ~]# chkconfig --list multipathd
multipathd      0:off   1:off   2:on    3:on    4:on    5:on    6:off

The next step is to configure the /etc/multipath.conf file. At the moment the file doesn’t exist, but I can get a template from /usr/share/doc/device-mapper-multipath-0.4.9 or just start a new file from scratch. I’ll go for the latter, so here is the content I’m adding:

blacklist {
        devnode "*"
}
blacklist_exceptions {
        devnode "sd*"
}
defaults {
        user_friendly_names yes
}

These first three sections deal with the blacklists (devices which should not be used by the multipath daemon) and the defaults (the parameter user_friendly_names results in devices called /dev/mapper/mpath* instead of rather-less friendly WWID-based names). Since my Violin LUNs are all presented over fibre channel they appear as SCSI devices with names in the format /dev/sd*, so this is why the blacklist exception contains an entry for sd*. In this rather basic configuration I am telling the multipath daemon that it should ignore all devices except those with names like /dev/sd*

The next section will contain a device description for Violin, which tells the multipath daemon how to interrogate Violin devices and what to do with the resulting virtual devices that will represent them. If you have other non-Violin devices also present you will probably need extra entries here, but in this setup I don’t:

# ALUA
devices {
    device {
        vendor "VIOLIN"
        product "SAN ARRAY ALUA"
        path_grouping_policy group_by_prio
        getuid_callout "/sbin/scsi_id --whitelisted --replace-whitespace --page=0x80 --device=/dev/%n"
        prio alua
        path_checker tur
        path_selector "round-robin 0"
        hardware_handler "1 alua"
        failback immediate
        rr_weight uniform
        no_path_retry fail
        rr_min_io_rq 4
    }

# non-ALUA
    device {
        vendor "VIOLIN"
        product "SAN ARRAY"
        path_grouping_policy group_by_serial
        getuid_callout "/sbin/scsi_id --whitelisted --replace-whitespace --page=0x80 --device=/dev/%n"
        path_checker tur
        path_selector "round-robin 0"
        hardware_handler "0"
        failback immediate
        rr_weight uniform
        no_path_retry fail
        rr_min_io_rq 4
    }
}

An important fact to notice here is that this is different to the entry we used for Oracle Linux 5. This is partly because some of the multipathing parameters have changed in RHEL6 / OL6 and partly because the syntax of the /sbin/scsi_id command has changed. For an explanation of the multipathing parameters a good place to start is this Guide to DM Multipath from Red Hat. The scsi_id command is used to interrogate devices in order to determine what they are and bring back information about them. Perhaps the best way to explain this is to run it manually with the additional –export option which prints all the details in a list (although in RHEL5 / OL5 the options are different):

[root@half-server4 host1]# /sbin/scsi_id --whitelisted --page=0x80 --device=/dev/sdb --export
ID_SCSI=1
ID_VENDOR=VIOLIN
ID_VENDOR_ENC=VIOLIN\x20\x20
ID_MODEL=SAN_ARRAY
ID_MODEL_ENC=SAN\x20ARRAY\x20\x20\x20\x20\x20\x20\x20
ID_REVISION=551
ID_TYPE=disk
ID_SERIAL_RAW="SVIOLIN  SAN ARRAY      8D4CC077FDB18BB5"
ID_SERIAL=SVIOLIN_SAN_ARRAY_8D4CC077FDB18BB5
ID_SERIAL_SHORT=8D4CC077FDB18BB5

The two most important bits of information here are the vendor and the serial code: the vendor tells the multipathing daemon to treat it using the parameters defined above, while the serial code (which is unique to each LUN) allows the daemon to know which of the many SCSI devices correspond to the same underlying LUN on the array. A LUN presented over multiple paths appears as all sorts of different SCSI devices with random names like /dev/sdb or /dev/sddz. But if these devices have the same serial, the multipathing daemon knows they are the same underlying LUN and can add them both to the same virtual device, e.g. /dev/dm-3.

Now that the multipath.conf file is setup I will start the multipathing daemon, check that it’s loaded in the kernel (not necessary, I’m just doing this as a demonstration) and then list the devices it can see. The latter will create an enormous amount of screen output so I will reduce this by using grep:

[root@half-server4 host1]# service multipathd start
Starting multipathd daemon:                                [  OK  ]
[root@half-server4 host1]# lsmod | grep mult
dm_multipath           18776  11 dm_round_robin
dm_mod                 76415  23 dm_multipath,dm_mirror,dm_log
[root@half-server4 host1]# multipath -ll | grep VIOLIN
mpathe (SVIOLIN_SAN_ARRAY_8D4CC0770414BA85) dm-4 VIOLIN,SAN ARRAY
mpathd (SVIOLIN_SAN_ARRAY_8D4CC077573E3AA9) dm-7 VIOLIN,SAN ARRAY
mpathc (SVIOLIN_SAN_ARRAY_8D4CC077EF0D02F7) dm-5 VIOLIN,SAN ARRAY
mpathb (SVIOLIN_SAN_ARRAY_8D4CC077FDB18BB5) dm-8 VIOLIN,SAN ARRAY
mpathk (SVIOLIN_SAN_ARRAY_8D4CC077A0617977) dm-12 VIOLIN,SAN ARRAY
mpathj (SVIOLIN_SAN_ARRAY_8D4CC077B2DDF035) dm-10 VIOLIN,SAN ARRAY
mpathi (SVIOLIN_SAN_ARRAY_8D4CC07732414DBD) dm-11 VIOLIN,SAN ARRAY
mpathh (SVIOLIN_SAN_ARRAY_8D4CC0777B6A4F7E) dm-9 VIOLIN,SAN ARRAY
mpathg (SVIOLIN_SAN_ARRAY_8D4CC077A6337433) dm-6 VIOLIN,SAN ARRAY
mpathf (SVIOLIN_SAN_ARRAY_8D4CC077C4BA60F9) dm-3 VIOLIN,SAN ARRAY

Prior to this whole exercise I created and presented 10 LUNs from the Violin array: 8 x 20GB LUNs for use in a DATA diskgroup and 2x 160GB LUNs for use in a RECO diskgroup. The sizes are relatively arbitrary, however the decision to use 8 LUNs in DATA is deliberate. Violin recommends using at least 8 LUNs for best performance – and ASM best practice advices using an even number of disks in a diskgroup. The requirement for 8+ LUNS is not actually a direct Violin issue: it’s to do with the I/O device queues in the operating system. We find that the OS can achieve better I/O performance if operations are split over 8 or more queues.

How can I tell which LUN is which from the output of multipath -ll? In this case, since the LUNs for DATA are 20GB and the LUNs for RECO are 160GB it’s relatively simple (the full multipath -ll output shows the size of each LUN). But let’s say I had lots of LUNs all of similar sizes… how could I know which is which?

The answer comes in the form of the excellent sg3utils package which contains a number of useful utilities. For me, every server should have this package installed – just the rescan-scsi-bus.sh utility alone is a good enough reason (when you present new storage to your host it’s a lot simpler to run this than to reboot!). Let’s install it:

[root@half-server4 host1]# yum install sg3_utils -y
<..snip!..>
Installed:
  sg3_utils.x86_64 0:1.28-4.el6

Dependency Installed:
  sg3_utils-libs.x86_64 0:1.28-4.el6

Complete!

The package contains many useful utilities, one of which is sq_inq. I can use this to query the Violin devices and bring back the LUN names as I created them on the array (highlighted in red):

[root@half-server4 ~]# for device in /dev/dm-*; do echo -n "$device = "; sg_inq -i $device | grep "vendor specific:" | cut -d':' -f2-; done
/dev/dm-8 =   41207F00147:SERVER4-DATA1:8D4CC077FDB18BB5
/dev/dm-5 =   41207F00147:SERVER4-DATA2:8D4CC077EF0D02F7
/dev/dm-7 =   41207F00147:SERVER4-DATA3:8D4CC077573E3AA9
/dev/dm-4 =   41207F00147:SERVER4-DATA4:8D4CC0770414BA85
/dev/dm-3 =   41207F00147:SERVER4-DATA5:8D4CC077C4BA60F9
/dev/dm-6 =   41207F00147:SERVER4-DATA6:8D4CC077A6337433
/dev/dm-9 =   41207F00147:SERVER4-DATA7:8D4CC0777B6A4F7E
/dev/dm-11 =  41207F00147:SERVER4-DATA8:8D4CC07732414DBD
/dev/dm-10 =  41207F00147:SERVER4-RECO1:8D4CC077B2DDF035
/dev/dm-12 =  41207F00147:SERVER4-RECO2:8D4CC077A0617977

Now that I have this information I’m going to combine it with the serial numbers from the multipath -ll output and use the results to add entries to the multipath.conf file to give each device a name I recognise. This is done by using the /dev/dm-* device to match the LUN name against the serial number (i.e. SVIOLIN_SAN_ARRAY_*). Here’s a sample of the new section I’ll add to my multipath.conf:

multipaths {
    multipath {
        wwid SVIOLIN_SAN_ARRAY_8D4CC077FDB18BB5
        alias violin_data1
    }
    multipath {
        wwid SVIOLIN_SAN_ARRAY_8D4CC077EF0D02F7
        alias violin_data2
    }
    multipath {
        wwid SVIOLIN_SAN_ARRAY_8D4CC077573E3AA9
        alias violin_data3
    }
...etc...
}

To make the multipathing daemon pick these changes up we need to flush the existing config and then rerun it:

[root@half-server4 ~]# multipath -F
[root@half-server4 ~]# multipath -v1
violin_data5
violin_data4
violin_data2
violin_data6
violin_data3
violin_data1
violin_data7
violin_reco1
violin_data8
violin_reco2
[root@half-server4 ~]# ls -l /dev/mapper/violin*
lrwxrwxrwx. 1 root root 7 Feb  1 12:14 /dev/mapper/violin_data1 -> ../dm-8
lrwxrwxrwx. 1 root root 7 Feb  1 12:14 /dev/mapper/violin_data2 -> ../dm-5
lrwxrwxrwx. 1 root root 7 Feb  1 12:14 /dev/mapper/violin_data3 -> ../dm-7
lrwxrwxrwx. 1 root root 7 Feb  1 12:14 /dev/mapper/violin_data4 -> ../dm-4
lrwxrwxrwx. 1 root root 7 Feb  1 12:14 /dev/mapper/violin_data5 -> ../dm-3
lrwxrwxrwx. 1 root root 7 Feb  1 12:14 /dev/mapper/violin_data6 -> ../dm-6
lrwxrwxrwx. 1 root root 7 Feb  1 12:14 /dev/mapper/violin_data7 -> ../dm-9
lrwxrwxrwx. 1 root root 8 Feb  1 12:14 /dev/mapper/violin_data8 -> ../dm-11
lrwxrwxrwx. 1 root root 8 Feb  1 12:14 /dev/mapper/violin_reco1 -> ../dm-10
lrwxrwxrwx. 1 root root 8 Feb  1 12:14 /dev/mapper/violin_reco2 -> ../dm-12

And there are my nicely-named Violin devices.

Setup UDEV Rules

The next stage on this journey is to configure an additional UDEV rule for the Violin devices. UDEV is the device manager for Linux and is responsible for creating all of the various device files in the /dev directory structure. This includes the /dev/sd* SCSI devices which represent the paths to the Violin array, as well as the /dev/dm-* multipath devices which are what we need to use with Oracle in order to have protection from path failures.

Why do we need to add a new rule? There are a few reasons, some Violin-related and others Oracle-related. The Violin-related reason is that by default all of the SCSI devices presented to the host will be using the deadline I/O scheduler. I’ve written about I/O schedulers before, but to recap they are algorithms which attempt to trade some extra CPU in order to merge or reorder I/Os in order to reduce the time spent waiting on storage (for example by reducing the number of disk head movements). This is all well and good, but with the Violin array we do not need to worry about any of this, so we prefer to set all Violin devices to use the noop scheduler (noop stands for no optimisations) so that we can just get on with the business of reading and writing. Another thing we like to do is tell the Linux kernel that the Violin devices are flash rather than disk – kernels with version 2.6.32 and above have an extra flag called rotational which is set to 0 when the device is not a spinning disk. By default the Violin SCSI devices will tell the kernel to use the correct value, but annoyingly the /dev/dm-* multipath devices based on Violin will need to be set manually. Finally we like to change the OS queue depths on Violin devices to reflect the increased capacity for I/Os. Here’s what a non-Violin device (based on a spinning disk) looks like:

[root@half-server4 ~]# cat /sys/block/sda/queue/scheduler
noop [deadline] cfq
[root@half-server4 ~]# cat /sys/block/sda/queue/rotational
1
[root@half-server4 ~]# cat /sys/block/sda/queue/nr_requests
128

For Violin we want noop, rotational = 0 and nr_requests = something like 512 or 1024. What about the Oracle reason I mentioned earlier? Well this is the issue of the device ownership. Oracle runs as the user oracle and group dba (or if you are using ASM with role separation it runs with grid and asmadmin so change the text in red below), so the multipath devices being used for the database / ASM will need to have the correct ownership and permissions in order that they can be accessed. In RHEL5 / OL5 we used to set this in the /etc/multipath.conf file but this is no longer possible in RHEL6 / OL6 so we have to set it in UDEV. It’s no big deal though. Here is the UDEV file required to do all of these things – there are two rules, one for the /dev/sd* devices and the other for the /dev/dm-* devices:

# UDEV rules for RH6 and OL6 with Violin Memory
# Set scheduler and queue depth for Violin SCSI devices
KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", SYSFS{model}=="SAN ARRAY*", RUN+="/bin/sh -c 'echo noop > /sys/$devpath/queue/scheduler && echo 1024 > /sys/$devpath/queue/nr_requests'"
# Set rotational, scheduler and queue depth for Violin multipath devices
KERNEL=="dm-[0-9]*", ENV{DM_UUID}=="mpath-SVIOLIN*", OWNER:="oracle", GROUP:="dba", MODE:="660", RUN+="/bin/sh -c 'echo 0 > /sys/$devpath/queue/rotational && echo noop > /sys/$devpath/queue/scheduler && echo 1024 > /sys/$devpath/queue/nr_requests'"

It’s important that each rule is contained on one line with no line breaks, otherwise it won’t run properly The file needs to be saved as /etc/udev/rules.d/12-violin.rules in order to run correctly. The UDEV rules directory (/etc/udev/rules.d) contains multiple files with names starting with numbers – they are run in ascending order. Each device processed by UDEV is subjected to all of these rules – unless it hits the magic command OPTIONS=”last_rule” in which case all subsequent rules will be ignored on that device. To ensure the Violin devices are picked up before the last_rule option could be hit, we call the file a name starting with 12.

To pick up the changes we have to reload the rules and then trigger UDEV. I’ll demonstrate this by showing the changes on device /dev/dm-8:

[root@half-server4 etc]# ls -l /dev/dm-8
brw-rw----. 1 root disk 252, 8 Feb  1 12:58 /dev/dm-8
[root@half-server4 etc]# cat /sys/block/dm-8/queue/scheduler
noop [deadline] cfq
[root@half-server4 etc]# cat /sys/block/dm-8/queue/rotational
1
[root@half-server4 etc]# udevadm control --reload-rules
[root@half-server4 etc]# udevadm trigger
[root@half-server4 etc]# ls -l /dev/dm-8
brw-rw----. 1 oracle dba 252, 8 Feb  1 12:58 /dev/dm-8
[root@half-server4 etc]# cat /sys/block/dm-8/queue/scheduler
[noop] deadline cfq
[root@half-server4 etc]# cat /sys/block/dm-8/queue/rotational
0

This is the final step for preparing the Violin devices, so we are now ready to begin installing Oracle.

Install Grid Infrastructure

The next step is the installation of Oracle’s Grid Infrastructure software which includes Clusterware and ASM. Unfortunately I cannot simply install this using the Oracle Universal Installer, because the OUI in 11.2.0.3 still does not understand the concept of 4k sector sizes. If you do not know what I am talking about here, it is essential that you read this because Violin, like all modern storage, is based Advanced Format 4k blocks. Linux kernels from 2.6.32 support 4k devices, as does Oracle’s database software from version 11.2. But… they still haven’t updated the OUI to allow the creation of 4k sector ASM diskgroups and database redo logs.

Important Note: You do not need to do this in order to achieve maximum performance from a 4k-based system such as Violin – see here for more details. Nothing from this point on is absolutely necessary, so if it looks like too much trouble simply make sure that the Violin LUNs are exported as 512 byte, then perform a normal Grid Infrastructure install. When finished, skip down to the bottom of this article to read the section entitled “Next Steps…”

When I created the LUNs for this installation I chose the 4096 byte block size in the Violin GUI:

I can see the result of this simply by running fdisk against one of the devices:

[root@half-server4 ~]# fdisk -l /dev/mapper/violin_data1
Note: sector size is 4096 (not 512)

Disk /dev/mapper/violin_data1: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 326 cylinders
Units = cylinders of 16065 * 4096 = 65802240 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 524288 bytes

The reason for choosing 4k is that the alternative (and default) choice is 512 byte sectors. At the physical level Violin always performs I/Os in 4k blocks, so reading and writing 512 byte blocks is just wasting time and effort. This is particularly true, for example, when Oracle’s log writer flushes the log buffer to the redo longs. If these log write I/Os are 512 byte, Violin has to read the 4k block, modify the 512 bytes being written, then write the 4k block back to flash. This the infamous read-modify-write operation – and while it still performs considerably faster than a normal disk I/O, it’s a penalty we don’t need to incur if we simply set Oracle up to use 4k sectors.

We therefore need to perform a software-only install of Grid Infrastructure:

(Actually, the first time I ran the installer it came up with a helpful box which said “half-server4: half-server4” and then exited. This, bizarrely, means that you need to look at the /etc/hosts file and make sure it’s correctly configured with the hostname. The default /etc/hosts file is almost always wrong in my experience)

After the software installs one of the post-installation tasks is to run the root.sh file, which is located in the new GI home. If you perform a software-only installation it’s important to heed the message printed when the root.sh completes. Here’s the (relevant) output from mine:

To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/roothas.pl

To configure Grid Infrastructure for a Cluster execute the following command:
/u01/app/11.2.0/grid/crs/config/config.sh
This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.

These are the commands necessary to start up the various Grid Infrastructure services and daemons, depending on whether you are installing a cluster (Cluster Ready Services) or a standalone system (High Availability Services). In my case it’s the latter, so I will need HAS:

[root@half-server4 ~]# /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/roothas.pl
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node half-server4 successfully pinned.
Adding Clusterware entries to upstart

half-server4     2013/02/01 14:41:51     /u01/app/11.2.0/grid/cdata/half-server4/backup_20130201_144151.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server

Next I need to start the css daemon:

[root@half-server4 ~]# /u01/app/11.2.0/grid/bin/crsctl start resource ora.cssd
CRS-2672: Attempting to start 'ora.cssd' on 'half-server4'
CRS-2672: Attempting to start 'ora.diskmon' on 'half-server4'
CRS-2676: Start of 'ora.diskmon' on 'half-server4' succeeded
CRS-2676: Start of 'ora.cssd' on 'half-server4' succeeded

Create ASM Instance

Because we didn’t use the OUI we will have to manually create an ASM instance. This is a good thing because it allows us to specify a 4k sector size for the ASM diskgroups (neither OUI or DBCA has this option yet). The first step is to add an ASM entry to the /etc/oratab file (note I have switched to the oracle user now):

[oracle@half-server4 ~]$ ls -l /etc/oratab
-rw-rw-r--. 1 oracle oinstall 741 Feb  1 14:35 /etc/oratab
[oracle@half-server4 ~]$ echo "+ASM:/u01/app/11.2.0/grid:N" >> /etc/oratab
[oracle@half-server4 ~]$ . oraenv
ORACLE_SID = [oracle] ? +ASM
The Oracle base has been set to /u01/app/oracle

I have also set the environment up using oraenv. Next I’ll create a parameter file for ASM:

[oracle@half-server4 ~]$ mkdir -p /u01/app/oracle/admin/+ASM/pfile
[oracle@half-server4 ~]$ vi /u01/app/oracle/admin/+ASM/pfile/init+ASM.ora

The contents of the parameter file will be:

asm_power_limit=5
diagnostic_dest='/u01/app/oracle'
instance_type='ASM'
large_pool_size=30M
memory_target=350M
remote_login_passwordfile='EXCLUSIVE'

I also need to create a symbolic link from the Oracle home’s dbs directory to the pfile:

[oracle@half-server4 ~]$ cd $ORACLE_HOME/dbs
[oracle@half-server4 dbs]$ ln -s /u01/app/oracle/admin/+ASM/pfile/init+ASM.ora

Now I’m ready to create the ASM instance. However, I can’t just start it with SQL*Plus, I need to start it through srvctl – and that means first adding it as a resource:

ASM home: /u01/app/11.2.0/grid
ASM listener was not found
PRCA-1032 : ASM listener LISTENER does not exist
Spfile:
ASM diskgroup discovery string: ++no-value-at-resource-creation--never-updated-through-ASM++

The config command shows me the resource configuration – and immediately I can see an issue (highlighted in red), which is that I have no listener. Let’s fix that straight away:

[oracle@half-server4 ~]$ srvctl add listener -l LISTENER
[oracle@half-server4 ~]$ srvctl start listener -l LISTENER
[oracle@half-server4 ~]$ srvctl add asm
[oracle@half-server4 ~]$ srvctl config asm
ASM home: /u01/app/11.2.0/grid
ASM listener: LISTENER
Spfile:
ASM diskgroup discovery string: ++no-value-at-resource-creation--never-updated-through-ASM++

The other issue we can see is that there is no spfile for ASM. Let’s fix that next:

[oracle@half-server4 ~]$ srvctl start asm
[oracle@half-server4 dbs]$ sqlplus / as sysasm

SQL> create spfile='/u01/app/oracle/admin/+ASM/pfile/spfile+ASM.ora' from pfile;

File created.

This new spfile now needs to be referenced from the pfile in the $ORACLE_HOME/dbs directory and also in the CRS resource configuration:

[oracle@half-server4 dbs]$ srvctl modify asm -p /u01/app/oracle/admin/+ASM/pfile/spfile+ASM.ora
[oracle@half-server4 dbs]$ srvctl config asm
ASM home: /u01/app/11.2.0/grid
ASM listener: LISTENER
Spfile: /u01/app/oracle/admin/+ASM/pfile/spfile+ASM.ora
ASM diskgroup discovery string: ++no-value-at-resource-creation--never-updated-through-ASM++

Since I’m in the $ORACLE_HOME/dbs directory I might as well create the password file:

[oracle@half-server4 dbs]$ orapwd file=/u01/app/11.2.0/grid/dbs/orapw+ASM password=oracle
[oracle@half-server4 dbs]$ ls -l /u01/app/11.2.0/grid/dbs/orapw+ASM
-rw-r-----. 1 oracle oinstall 1536 Feb  1 16:16 /u01/app/11.2.0/grid/dbs/orapw+ASM

I also need to manually create the ASMSNMP user with SYSDBA privileges otherwise I’ll have problems later on when I create the database.

SQL> create user asmsnmp identified by oracle;

User created.

SQL> grant sysdba to asmsnmp;

Grant succeeded.

That’s it, so I can restart ASM in order to pick up the changes and check the status of the CRS resources:

[oracle@half-server4 dbs]$ srvctl stop asm
[oracle@half-server4 dbs]$ srvctl start asm
[oracle@half-server4 dbs]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       half-server4
ora.asm
               ONLINE  ONLINE       half-server4             Started
ora.ons
               OFFLINE OFFLINE      half-server4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       half-server4
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE  ONLINE       half-server4

The next step ought to be to create the diskgroups… but we are going to discover that this is not possible without first installing ASMlib.

Install ASMLib

Not everyone is a fan of the ASMLib kernel library supplied by Oracle to manage devices in Linux. However, in the case of installing a 4k sector device on Oracle Linux 6 we have no choice. Here’s why:

SQL> alter system set asm_diskstring='/dev/mapper/violin*' scope=memory;

System altered.

SQL> select disk_number, mount_status, header_status, state, sector_size, path from v$asm_disk
  2  where path like '/dev/mapper/violin_reco%';

DISK_NUMBER MOUNT_S HEADER_STATU STATE    SECTOR_SIZE PATH
----------- ------- ------------ -------- ----------- ----------------------------------------
          0 CLOSED  UNKNOWN      NORMAL          4096 /dev/mapper/violin_reco1
          2 CLOSED  UNKNOWN      NORMAL          4096 /dev/mapper/violin_reco2

SQL> CREATE DISKGROUP RECO EXTERNAL REDUNDANCY
  2  DISK '/dev/mapper/violin_reco1', '/dev/mapper/violin_reco2';
CREATE DISKGROUP RECO EXTERNAL REDUNDANCY
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15080: synchronous I/O operation to a disk failed
ORA-15080: synchronous I/O operation to a disk failed
ORA-15080: synchronous I/O operation to a disk failed
ORA-15080: synchronous I/O operation to a disk failed

Quite why Oracle ASM should decide this device is unusable is a mystery, because with operating system utilities it is perfectly accessible. And here’s the thing, with ASMLib it also works perfectly. A scan of the Oracle bug database shows a number of other 4k storage devices exhibiting this same problem, so we know the issue is not specific to the storage system.

As it happens, ASMLib may possibly provide some benefits anyway: first of all it simplifies device naming and administration, particular in clusters. And secondly, it results in a far lower use of file descriptors. The authors even claim a reduction in context switching. Some people disagree but, since we have no choice with this release of Oracle, let’s embrace it.

ASMLib used to consist of three packages: the library, the kernel driver and the support package. In OL6 since we have the Unbreakable Enterprise Kernel the ASMLib kernel driver is no longer needed (it’s been kernelized so is already present). That just leaves the library (available from here) and the support package (available from yum):

[root@half-server4 ~]# wget http://download.oracle.com/otn_software/asmlib/oracleasmlib-2.0.4-1.el6.x86_64.rpm
Saving to: oracleasmlib-2.0.4-1.el6.x86_64.rpm

[root@half-server4 ~]# yum localinstall oracleasmlib-2.0.4-1.el6.x86_64.rpm
<...snip!...>
Installed:
  oracleasmlib.x86_64 0:2.0.4-1.el6

[root@half-server4 ~]# yum install oracleasm-support -y
<...snip!...>
Installed:
  oracleasm-support.x86_64 0:2.1.5-1.el6

ASMLib needs to be configured to run at startup and use the relevant user and group:

[root@half-server4 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

There is a configuration file located in /etc/sysconfig/oracleasm which tells ASMLib which devices to scan and which to ignore. We want the /dev/dm-* multipath devices to be scanned and the /dev/sd* SCSI devices (which do not have path failover) to be ignored:

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="dm"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sd"

Picking up the changes requires a restart of the driver:

[root@half-server4 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

Finally I need to label the Violin LUNs with ASMLib labels:

[root@half-server4 ~]# for lun in 1 2 3 4 5 6 7 8; do
> oracleasm createdisk DATA$lun /dev/mapper/violin_data$lun
> done
Writing disk header: done
Instantiating disk: done
Writing disk header: done
Instantiating disk: done
Writing disk header: done
Instantiating disk: done
Writing disk header: done
Instantiating disk: done
Writing disk header: done
Instantiating disk: done
Writing disk header: done
Instantiating disk: done
Writing disk header: done
Instantiating disk: done
Writing disk header: done
Instantiating disk: done
[root@half-server4 ~]# for lun in 1 2; do
> oracleasm createdisk RECO$lun /dev/mapper/violin_reco$lun
> done
Writing disk header: done
Instantiating disk: done
Writing disk header: done
Instantiating disk: done
[root@half-server4 ~]# oracleasm listdisks
DATA1
DATA2
DATA3
DATA4
DATA5
DATA6
DATA7
DATA8
RECO1
RECO2

I can also check that these devices are visible to ASM using kfod:

[oracle@half-server4 ~]$ kfod cluster=false asm_diskstring='ORCL:*'
--------------------------------------------------------------------------------
 Disk          Size Path                                     User     Group
================================================================================
   1:      20480 Mb ORCL:DATA1                                
   2:      20480 Mb ORCL:DATA2                                
   3:      20480 Mb ORCL:DATA3                                
   4:      20480 Mb ORCL:DATA4                                
   5:      20480 Mb ORCL:DATA5                                
   6:      20480 Mb ORCL:DATA6                                
   7:      20480 Mb ORCL:DATA7                                
   8:      20480 Mb ORCL:DATA8                                
   9:     163840 Mb ORCL:RECO1                                
  10:     163840 Mb ORCL:RECO2                                
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
================================================================================
      +ASM /u01/app/11.2.0/grid

Create ASM Diskgroups

The default value for the ASM_DISKSTRING parameter is null, which actually means the same as a value of ‘ORCL:*’ i.e. all disks presented via the ASMLib driver. I can therefore select from v$asm_disk to see what disks are available:

SQL> select disk_number, mount_status, header_status, state, sector_size, path
  2  from v$asm_disk where path like 'ORCL%';

DISK_NUMBER MOUNT_S HEADER_STATU STATE    SECTOR_SIZE PATH
----------- ------- ------------ -------- ----------- ------------
          0 CLOSED  PROVISIONED  NORMAL          4096 ORCL:DATA1
          1 CLOSED  PROVISIONED  NORMAL          4096 ORCL:DATA2
          2 CLOSED  PROVISIONED  NORMAL          4096 ORCL:DATA3
          3 CLOSED  PROVISIONED  NORMAL          4096 ORCL:DATA4
          9 CLOSED  PROVISIONED  NORMAL          4096 ORCL:RECO2
          5 CLOSED  PROVISIONED  NORMAL          4096 ORCL:DATA6
          6 CLOSED  PROVISIONED  NORMAL          4096 ORCL:DATA7
          7 CLOSED  PROVISIONED  NORMAL          4096 ORCL:DATA8
          8 CLOSED  PROVISIONED  NORMAL          4096 ORCL:RECO1
          4 CLOSED  PROVISIONED  NORMAL          4096 ORCL:DATA5

10 rows selected.

This is what I want to see, so I’l go ahead and create my diskgroups – being sure to use the 4k sector size options. First I’ll paste the script here to make it easier for cut and pasters:

CREATE DISKGROUP DATA EXTERNAL REDUNDANCY
DISK 'ORCL:DATA1', 'ORCL:DATA2',
     'ORCL:DATA3', 'ORCL:DATA4',
     'ORCL:DATA5', 'ORCL:DATA6',
     'ORCL:DATA7', 'ORCL:DATA8'
ATTRIBUTE
     'sector_size'='4096',
     'compatible.asm' = '11.2',
     'compatible.rdbms' = '11.2';

CREATE DISKGROUP RECO EXTERNAL REDUNDANCY
DISK 'ORCL:RECO1', 'ORCL:RECO2'
ATTRIBUTE
     'sector_size'='4096',
     'compatible.asm' = '11.2',
     'compatible.rdbms' = '11.2';

Now here’s the output from running it:

Diskgroup created.

Diskgroup created.

SQL> select group_number, name, sector_size, block_size, state
  2  from v$asm_diskgroup;

GROUP_NUMBER NAME       SECTOR_SIZE BLOCK_SIZE STATE
------------ ---------- ----------- ---------- -----------
           1 DATA              4096       4096 MOUNTED
           2 RECO              4096       4096 MOUNTED

Actually I have to say that I’m starting to prefer the output from the ASMCMD utility for stuff like this – the commands are so much quicker to type and don’t require any formatting of columns etc:

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N        4096   4096  1048576    163840   163774                0          163774              0             N  DATA/
MOUNTED  EXTERN  N        4096   4096  1048576    327680   327624                0          327624              0             N  RECO/

Next Steps…

We now have a working system with UDEV and multipathing correctly configured, as well as Oracle Grid Infrastructure up and running. That’s the end of the this cookbook, but probably not the end of your journey as you will most likely want to install the database software too. The installation of the database using a 4k sector size is covered in the following cookbook – enjoy!:

Installing Oracle 11.2.0.3 single-instance using 4k sector size