Oracle Linux 6 Update 5 within an Oracle VM Template

This cookbook covers the installation of Oracle 11g Release 2 within a virtual machine created using Oracle VM 3.3.1 and the Oracle Linux 6 Update 5 template available from Oracle’s eDelivery site. Oracle VM has already been configured with Violin LUNs using multipathing. A check of the kernel version tells us that we have the Unbreakable Enterprise Kernel v3 in place:

root@localhost ~]# cat /etc/oracle-release
Oracle Linux Server release 6.5
[root@localhost ~]# uname -r
3.8.13-16.2.2.el6uek.x86_64

As always, the first thing I have done before even contemplating an Oracle installation is to disable selinux and the iptables firewall:

[root@localhost ~]# sestatus
SELinux status:                 disabled
[root@localhost ~]# chkconfig --list | grep table
ip6tables      	0:off	1:off	2:off	3:off	4:off	5:off	6:off
iptables       	0:off	1:off	2:off	3:off	4:off	5:off	6:off

You might say that this isn’t necessary and that it’s a security risk. Let me tell you now, trying to work with selinux and iptables is a security risk to my mental health. Off it goes…

Packages

First of all I need to install all the packages associated with Oracle. The simplest way to do this is to use yum with the relevant Oracle Preinstall package:

[root@localhost ~]# yum list oracle\*preinstall\*
Available Packages
oracle-rdbms-server-11gR2-preinstall.x86_64                               1.0-10.el6                                       public_ol6_latest
oracle-rdbms-server-12cR1-preinstall.x86_64                               1.0-12.el6                                       public_ol6_latest

Today I need Oracle 11g Release 2 so I’ll choose that one (I’ve cut out much of the output to save space onscreen):

[root@localhost ~]# yum install oracle-rdbms-server-11gR2-preinstall -y
Setting up Install Process
Resolving Dependencies
<output truncated>
Transaction Summary
=================================================================================
Install      38 Package(s)
Upgrade       5 Package(s)
<output truncated>
Complete!

As part of that installation, an oracle user would have been created so let’s now set its password so we can use it in the future:

[root@localhost ~]# passwd oracle
Changing password for user oracle.
New password:
Retype new password:

I also want to install the sg3_utils and wget packages:

[root@localhost ~]# yum install sg3_utils -y
[root@localhost ~]# yum install wget -y

And I’m going to be using Oracle ASMLib so I’ll install the support package for that too:

[root@localhost ~]# yum install oracleasm\*
Installed:
  oracleasm-support.x86_64 0:2.1.8-1.el6

In addition to the support package I need the library package as well (I don’t need the kernel package because I have the Unbreakable Enterprise Kernel from Oracle which comes with the ASMLib module built in). The library isn’t available via yum but you can download it from the Oracle website here:

[root@localhost ~]# wget http://download.oracle.com/otn_software/asmlib/oracleasmlib-2.0.4-1.el6.x86_64.rpm
--2014-12-16 07:48:31--  http://download.oracle.com/otn_software/asmlib/oracleasmlib-2.0.4-1.el6.x86_64.rpm
Resolving download.oracle.com... 184.86.240.88, 184.86.240.82
Connecting to download.oracle.com|184.86.240.88|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13300 (13K) [application/x-redhat-package-manager]
Saving to: `oracleasmlib-2.0.4-1.el6.x86_64.rpm'

100%[=====================================================================>] 13,300      --.-K/s   in 0.004s

2014-12-16 07:48:31 (3.24 MB/s) - `oracleasmlib-2.0.4-1.el6.x86_64.rpm' saved [13300/13300]

I now have the library RPM locally so I need to install it either with the rpm command or with yum’s localinstall option:

root@localhost ~]# ls -l oracleasmlib-2.0.4-1.el6.x86_64.rpm
-rw-r--r--. 1 root root 13300 Jun 28  2012 oracleasmlib-2.0.4-1.el6.x86_64.rpm
[root@localhost ~]# yum localinstall oracleasmlib-2.0.4-1.el6.x86_64.rpm -y
Installed:
  oracleasmlib.x86_64 0:2.0.4-1.el6

To setup ASMLib to run at boot time we need to run the configure command and tell it which user and group to run as (I’m using oracle and dba):

root@localhost ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

Ok that’s all the packages taken care of, so now let’s look at the storage.

Virtual Storage Devices

Now if this was a physical (non-virtualized) installation there are certain steps we would usually take:

1. Install and configure the Linux DM Multipathing packages
2. Give the LUNs presented from Violin a user-friendly name in /dev/mapper
3. Create a UDEV rule to switch all Violin LUNs to use the noop I/O scheduler
4. Optionally configure UDEV so that the devices which will be used by ASM are owned by oracle:dba
5. Configure ASMLib to use the /dev/dm-* multipath devices and ignore the single-path /dev/sd* devices

But hang on. Usually when we present external storage to Linux (in this case over fibre channel) we expect to see a load of /dev/sd* SCSI devices. So where are they?

[root@localhost ~]# ls -l /dev/sd*
ls: cannot access /dev/sd*: No such file or directory

They don’t exist. And the reason they don’t exist is because they are being presented via the Oracle VM (Xen-based) hypervisor, which means they appear as virtual storage devices with the name /dev/xvd* (for Xen Virtual Disk):

[root@localhost ~]# ls -l /dev/xvd*
brw-rw---- 1 root disk 202,   0 Dec 16 08:00 /dev/xvda
brw-rw---- 1 root disk 202,   1 Dec 16 08:00 /dev/xvda1
brw-rw---- 1 root disk 202,   2 Dec 16 08:00 /dev/xvda2
brw-rw---- 1 root disk 202,   3 Dec 16 08:00 /dev/xvda3
brw-rw---- 1 root disk 202,  16 Dec 16 08:00 /dev/xvdb
brw-rw---- 1 root disk 202,  32 Dec 16 08:00 /dev/xvdc
brw-rw---- 1 root disk 202,  48 Dec 16 08:00 /dev/xvdd
brw-rw---- 1 root disk 202,  64 Dec 16 08:00 /dev/xvde
brw-rw---- 1 root disk 202,  80 Dec 16 08:00 /dev/xvdf
brw-rw---- 1 root disk 202,  96 Dec 16 08:00 /dev/xvdg
brw-rw---- 1 root disk 202, 112 Dec 16 08:00 /dev/xvdh
brw-rw---- 1 root disk 202, 128 Dec 16 08:00 /dev/xvdi

Here we can see my eight Violin LUNs named xvdb to xvdi, plus the xvda device which is used for the local filesystem (you can see that it has three partitions on it). Ok, so the devices have different names from what we usually expect… so what?

Just taking a step back for a second, here’s what happens when I run an sg_inq command on a normal Violin device presented to a non-virtualized OL6 system:

[root@server4 ~]# sg_inq -i /dev/sdb
VPD INQUIRY: Device Identification page
  Designation descriptor number 1, descriptor length: 45
    designator_type: T10 vendor identification,  code_set: ASCII
    associated with the addressed logical unit
      vendor id: VIOLIN
      vendor specific: 81321F00111:TEST:16968FC5C635E1E2
<output truncated>

I can see all sorts of useful information about the vendor, as well as the LUN name (“TEST”), the serial number and so on. Now let’s do the same thing to the xvd device:

[root@localhost ~]# sg_inq -i /dev/xvdb
VPD INQUIRY: Device Identification page
inquiry: pass through os error: Invalid argument
    inquiry: failed, res=-1

Not so much. Well, what about udevadm info? After all, it’s UDEV that actually creates all the devices in the /dev filesystem, so it must be able to see some device information otherwise the /dev/xvd* devices would not exist. Let’s run a query:

[root@localhost ~]# udevadm info --query=all --name=/dev/xvdb
P: /devices/vbd-51728/block/xvdb
N: xvdb
W: 53
S: block/202:16
S: disk/by-path/xen-vbd-51728
E: UDEV_LOG=3
E: DEVPATH=/devices/vbd-51728/block/xvdb
E: MAJOR=202
E: MINOR=16
E: DEVNAME=/dev/xvdb
E: DEVTYPE=disk
E: SUBSYSTEM=block
E: ID_PATH=xen-vbd-51728
E: DEVLINKS=/dev/block/202:16 /dev/disk/by-path/xen-vbd-51728

That’s surprising. It might look like we are getting useful info back here, but in reality there is hardly anything to work with. The same udevadm info command run on the non-virtualized platform brings back 34 lines including information about the vendor, model and serial number. Normally we use this information to configure multipathing and add the extra UDEV rule which implements the noop scheduler – the rules are all designed to be triggered if VENDOR=”VIOLIN”. What do we do now?

Thinking Outside of the (Virtual) Box

Let’s revisit the list I made at the start of the previous section:

1. Linux DM Multipathing: we don’t need to implement multipathing within the VM, because it is taken care of by the hypervisor
2. User Friendly Names: since multipathing won’t be installed this is no longer required and we will use ASMLib to ensure we have device names which are human-readable
3. UDEV Rule to Switch I/O Scheduler: park this one, I’ll come back to it later
4. Optionally configure UDEV to set device ownership to oracle:dba: I’ll come back to this one as well
5. Configure ASMLib to use the correct devices: let’s do this one now

So we are going to skip all of the usual multipathing config in this VM, which means we are creating ASMLib labels directly on the /dev/xvd* devices. By the way, these devices are configured in Advanced Format Emulation Mode, which means they have a 512 byte logical sector size and a 4k physical sector size (see here for more details):

[root@localhost ~]# fdisk -l /dev/xvdb | grep "Sector size"
Sector size (logical/physical): 512 bytes / 4096 bytes

At this point, on non-virtualized systems which use multipathing, we would usually configure ASMLib to have an ORACLEASM_SCANORDER of “dm” and an ORACLEASM_SCANEXCLUDE of “sd”. This tells ASMLib not to look at the single paths to storage, only the paths presented by multipathing – which makes sense because we don’t ever want ASM to use a single path if there are multipath devices available (since they offer resilience).

But in our VM we only have single path devices present, albeit in the format of xvd* instead of sd*. By default, ASMLib will query all devices it finds in the /proc/partitions table:

[root@localhost ~]# cat /proc/partitions
major minor  #blocks  name

 202        0   12582912 xvda
 202        1     514048 xvda1
 202        2    9970688 xvda2
 202        3    2097152 xvda3
 202       16   52428800 xvdb
 202       32   52428800 xvdc
 202       48   52428800 xvdd
 202       64   52428800 xvde
 202       80   52428800 xvdf
 202       96   52428800 xvdg
 202      112   52428800 xvdh
 202      128   52428800 xvdi

This means there is no need for us to set the scan order or scan exclude settings for ASMLib, although for completeness you could set the scan order to xvd:

[root@localhost ~]# oracleasm configure -o xvd
Writing Oracle ASM library driver configuration: done
[root@localhost ~]# grep ORACLEASM_SCANORDER /etc/sysconfig/oracleasm
# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="xvd"

Anyway, there are no special considerations for multipathing, so presumably we just issue the usual oracleasm createdisk command to the /dev/xvd* device, right?

[root@localhost ~]# oracleasm createdisk DATA1 /dev/xvdb
Device "/dev/xvdb" is not a partition

What’s happened here is that ASMLib has had a look at the device and determined, correctly, that it has not been partitioned. But so what? I don’t want to partition it – partitions don’t bring me any benefit.

Unfortunately, somebody in Oracle still thinks that partitions are essential for storage devices, which is why ASMLib has this check built in. However… it doesn’t usually flag up this message when you create a disk on an unpartitioned multipath device, because it’s harder for ASMLib to tell if a multipath device has been partitioned or not.

I don’t like ASMLib telling me what to do, so I’m going to overrule it:

[root@localhost ~]# /usr/sbin/asmtool -C -l /dev/oracleasm -n DATA1 -s /dev/xvdb -a force=yes
asmtool: Device "/dev/xvdb" is not a partition
asmtool: Continuing anyway
[root@localhost ~]# oracleasm listdisks
DATA1

Ha. That’ll teach it. Let’s label the other seven devices too:

[root@localhost ~]# num=2; for lun in c d e f g h i; do
> /usr/sbin/asmtool -C -l /dev/oracleasm -n DATA$num -s /dev/xvd$lun -a force=yes
> let num++
> done
asmtool: Device "/dev/xvdc" is not a partition
asmtool: Continuing anyway
asmtool: Device "/dev/xvdd" is not a partition
asmtool: Continuing anyway
asmtool: Device "/dev/xvde" is not a partition
asmtool: Continuing anyway
asmtool: Device "/dev/xvdf" is not a partition
asmtool: Continuing anyway
asmtool: Device "/dev/xvdg" is not a partition
asmtool: Continuing anyway
asmtool: Device "/dev/xvdh" is not a partition
asmtool: Continuing anyway
asmtool: Device "/dev/xvdi" is not a partition
asmtool: Continuing anyway
[root@localhost ~]# oracleasm listdisks
DATA1
DATA2
DATA3
DATA4
DATA5
DATA6
DATA7
DATA8

All present and correct. So that’s everything on the list sorted, with the exception of those UDEV rules to configure the I/O scheduler and the device ownership… Let’s look at that last.

I/O Schedulers, UDEV Rules and Device Ownership

This is where things get murky. In a non-virtualized installation we normally configure a UDEV rule called 12-violin.rules which gets dropped into the /etc/udev/rules.d directory and looks something like this:

# UDEV rules for RH6 and OL6 with Violin Memory
# Set scheduler and queue depth for Violin SCSI devices
KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", SYSFS{model}=="SAN ARRAY*", RUN+="/bin/sh -c 'echo noop > /sys/$devpath/queue/scheduler && echo 1024 > /sys/$devpath/queue/nr_requests'"
# Set rotational, scheduler and queue depth for Violin multipath devices
KERNEL=="dm-[0-9]*", ENV{DM_UUID}=="mpath-SVIOLIN*", OWNER:="oracle", GROUP:="dba", MODE:="660", RUN+="/bin/sh -c 'echo 0 > /sys/$devpath/queue/rotational && echo noop > /sys/$devpath/queue/scheduler && echo 64 > /sys/$devpath/queue/nr_requests'"

There are two rules in this file: one for the /dev/sd* single-path SCSI devices and the other for the /dev/dm-* multipath devices. They perform similar actions:

• Trigger for all devices with VENDOR=”VIOLIN” and MODEL=”SAN ARRAY” (sd devices)
• Trigger for all devices with a DM_UUID that starts with “mpath-SVIOLIN*” (dm devices)
• Set the scheduler to noop
• Set the nr_requests parameter to 1024 (sd) or 64 (dm)
• Set the rotational parameter to 0 (dm devices only)
• Set the dm devices only to be owned by oracle with a group of dba

Hopefully it’s immediately obvious that neither of these rules are going to be useful today, because in this virtual machine our devices are called /dev/xvd* so the KERNEL== clause will not match for either rule.

But even if I change that clause to be KERNEL==”xvd*[!0-9]” I have another problem: the Xen virtual devices aren’t forwarding the device vendor, model and serial properties so I cannot write a rule that only pick up Violin devices. I could have a blanket rule for all xvd* devices but that would also trigger for the xvda device which contains the root filesystem… and I definitely do not want that owned by oracle.

To be honest, changing the I/O scheduler is not a big deal anymore. The UEKv3 kernel uses the deadline I/O schedule by default, which is far superior to the old cfq scheduler. We are quite likely to get great performance from the default configuration. But what about the device ownership? The /dev/xvd* devices are owned by root:disk and therefore cannot be used with ASM – we can confirm this with the kfod tool:

[oracle@localhost ~]$ kfod cluster=false asm_diskstring='/dev/xvd*' disks=all
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
================================================================================
      +ASM /u01/app/oracle/product/11.2.0/grid

On the other hand, ASMLib allows us to create our own oracle:dba devices which work perfectly:

oracle@localhost ~]$ kfod cluster=false asm_diskstring='ORCL:*' disks=all
--------------------------------------------------------------------------------
 Disk          Size Path                                     User     Group
================================================================================
   1:      51200 Mb ORCL:DATA1                                
   2:      51200 Mb ORCL:DATA2                                
   3:      51200 Mb ORCL:DATA3                                
   4:      51200 Mb ORCL:DATA4                                
   5:      51200 Mb ORCL:DATA5                                
   6:      51200 Mb ORCL:DATA6                                
   7:      51200 Mb ORCL:DATA7                                
   8:      51200 Mb ORCL:DATA8                                
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
================================================================================
      +ASM /u01/app/oracle/product/11.2.0/grid

I haven’t got an answer here, other than to use ASMLib. I’m sure there must be a way to configure UDEV correctly, but I cannot think of it. However, ASMLib gives me exactly what I want, as we can see by querying the /dev/oracleasm/disks directory:

[root@localhost ~]# ls -l /dev/oracleasm/disks
total 0
brw-rw---- 1 oracle dba 202,  16 Dec 16 11:53 DATA1
brw-rw---- 1 oracle dba 202,  32 Dec 16 11:53 DATA2
brw-rw---- 1 oracle dba 202,  48 Dec 16 11:53 DATA3
brw-rw---- 1 oracle dba 202,  64 Dec 16 11:53 DATA4
brw-rw---- 1 oracle dba 202,  80 Dec 16 11:53 DATA5
brw-rw---- 1 oracle dba 202,  96 Dec 16 11:53 DATA6
brw-rw---- 1 oracle dba 202, 112 Dec 16 11:53 DATA7
brw-rw---- 1 oracle dba 202, 128 Dec 16 11:53 DATA8

Not everyone likes ASMlib – and many people just feel ambivalent about it – but in this case I can’t think of an alternative way of making the devices accessible to ASM.

Oracle Installation

That’s the end of the preparation, now it’s just a standard installation of Oracle 11gR2 Grid Infrastructure using the 8 LUNs as ASM disks for an external redundancy diskgroup called +DATA. Then on top of that we install Oracle 11gR2 Database and everything is in place. I’m not going to document these two procedures because they are completely standard…

Have fun!