Installing Oracle VM with Violin

Installing Violin Memory with Oracle VM is fairly straight forward; it’s basically a case of getting the multipath.conf file correctly configured and setting some UDEV rules.

In this example I have installed Oracle VM Server version 3.1.1 for x86_64, which gives me the following:

[root@ovs1 ~]# cat /etc/oracle-release
Oracle VM Server release 5.7
[root@ovs1 ~]# uname -r
2.6.39-200.1.1.el5uek

It’s important to note here that version 2 of the Oracle Unbreakable Enterprise Kernel (UEK) is being used by default. As you can see this shows a kernel version of 2.6.39 but in fact it is based on 3.0.16. Why the discrepancy? To quote the release notes:

This was done to avoid potential breakage of certain low-level utilities of the Oracle Linux distribution (also known as the “plumbing”) that potentially can’t cope with the new 3.x version scheme.

As part of the installation of Oracle VM Server (which is based on Oracle Linux 5, which in turn is based on Red Hat Enterprise Linux 5) the Linux Device Mapper Multipath drivers are already installed and configured. If you have your operating system located on a local disk and then intend to use Violin as high-performance storage for virtual machines, the device used for the root filesystem will automatically be placed in the blacklist entries like this:

[root@ovs1 ~]# cat /etc/multipath.conf.rpmorig
defaults {
        user_friendly_names no
}
blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|nbd)[0-9]*"
        devnode "^(hd|xvd|vd)[a-z]*"
        devnode "^etherd"
        devnode "^cciss!c[0-9]d[0-9]*"
        # Device sda is used by FS [ / ]
        wwid "SATA_INTEL_SSDSA2BW0CVPR150001MP080BGN_"
}

The important thing to realise here is that you need to edit this file rather than overwrite it, otherwise you will lose the line highlighted in red. I speak from experience, because I didn’t notice this on my first attempt and spent a lot of time wondering why things were then bad.

Clearly the type of internal disk you have will affect the entry here. For example, here is the same line from a different server:

# Device sda is used by FS [ / ]
wwid "SATA_WDC_WD5003ABYX-_WD-WMAYP2066497"

So how can you find out what the correct entry should be if you need to create this manually? One option is the storage_info command:

[root@ovs1 ~]# storage_info --do_listDisks
SVIOLIN_SAN_ARRAY_1867AF6CA637606C

In this example all that I can see is the LUN presented from my Violin array, so that is correct. However, if I had not correctly configured the multipathing daemon to blacklist the local disk, that would also have shown up in the list… providing me with the correct entry to add to the blacklist.

So what should the multipath.conf file look like with the Violin entries included? Here’s mine, with the newly-added Violin device info in blue:

[root@ovs1 ~]# cat /etc/multipath.conf
defaults {
        user_friendly_names no
}
blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|nbd)[0-9]*"
        devnode "^(hd|xvd|vd)[a-z]*"
        devnode "^etherd"
        devnode "^cciss!c[0-9]d[0-9]*"
        # Device sda is used by FS [ / ]
        wwid "SATA_INTEL_SSDSA2BW0CVPR150001MP080BGN_"
}
devices {
    device {
        vendor                "VIOLIN"
        product               "SAN ARRAY"
        path_grouping_policy  group_by_serial
        getuid_callout        "/sbin/scsi_id -p 0x80 -g -u -s /block/%n"
        path_checker          tur
        path_selector         "round-robin 0"
        hardware_handler      "0"
        failback              immediate
        rr_weight             uniform
        no_path_retry         fail
        rr_min_io             4
        max_fds               8192
    }
}

This is essentially the same as the standard Violin multipath.conf entries for RHEL5 or OL5 (see here), however without the extra section containing the multipath aliases. We don’t need to create aliases for the Violin devices because they will automatically show up in the OVM Manager.

Configuring UDEV

Another action that needs to be taken is to add a UDEV rule to switch the Violin device files to use the noop I/O scheduler. We do this because with ultra-low latency flash storage it makes no sense to spend CPU cycles trying to batch up I/Os into groups as we have to do when using spinning disks (to minimise head movement). Since this is the 3.0.16 kernel it is pretty advanced at detecting and correctly-handing non-mechanical storage, so it already works out that my Violin SCSI devices (i.e. the ones with names /dev/sd*) need to have the rotational flag set to zero and that the noop scheduler is the best choice:

[root@ovs2 rules.d]# cat /sys/block/sdb/queue/rotational
0 
[root@ovs2 rules.d]# cat /sys/block/sdb/queue/scheduler
[noop] deadline cfq

Unfortunately, this improved handling doesn’t stretch as far as correcting the same attributes for the dynamic multipathing devices (i.e. the ones with names /dev/dm-*). Remember that this is storage presented over multiple paths (in this case fibre-channel) and that the Linux Device Mapper is creating a single “multipath” virtual device to represent all of the available paths. If we look at the attributes for this, rotational is still set to one and the deadline scheduler is in use:

[root@ovs2 rules.d]# cat /sys/block/dm-0/queue/rotational
1
[root@ovs2 rules.d]# cat /sys/block/dm-0/queue/scheduler
noop [deadline] cfq

To fix this we have to add a UDEV rule. For completeness I am going to add a rule for the SCSI devices too, just to ensure they have the correct settings. Adding a rule is a simple case of adding a file to the /etc/udev/rules.d directory:

[root@ovs2 rules.d]# cat 12-violin.rules

# Set scheduler and queue depth for Violin SCSI devices
KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", SYSFS{model}=="SAN ARRAY*", RUN+="/bin/sh -c 'echo 0 > /sys/$devpath/queue/rotational && echo noop > /sys/$devpath/queue/scheduler && echo 512 > /sys/$devpath/queue/nr_requests'"
#
# Set rotational, scheduler and queue depth for Violin multipath devices
SUBSYSTEM=="block", KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="SVIOLIN*", RUN+="/bin/sh -c 'echo 0 > /sys/$devpath/queue/rotational && echo noop > /sys/$devpath/queue/scheduler && echo 512 > /sys/$devpath/queue/nr_requests'"

It’s important to have each rule on a single line with no line breaks. The rule file has a name starting with “12” to ensure it runs early in the UDEV processing phase (the rules are run in numerical order). The pre-existing rule file 40-multipath.rules contains a call of OPTIONS=”last_rule” which means that subsequent rules will not be applied to the device in question, so a Violin rule file starting with (for example) 60 will not get processed.

Incidentally, this UDEV file is remarkably similar to the one you would use if you were installing a database and using ASM, but with one big difference: for a database you would need to change the ownership, group and permissions of the /dev/dm-* devices to be owned by oracle:dba (or grid:asmadmin if using role separation). For Oracle VM the devices should be owned by root:root so no changes are necessary.

For completeness, in case anyone stumbles upon this looking for the database UDEV rule, here it is with the extra clauses highlighted in red. Do not use this version with OVM though…

[root@ovs2 rules.d]# cat 12-violin.rules

# Set scheduler and queue depth for Violin SCSI devices
KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", SYSFS{model}=="SAN ARRAY*", RUN+="/bin/sh -c 'echo 0 > /sys/$devpath/queue/rotational && echo noop > /sys/$devpath/queue/scheduler && echo 512 > /sys/$devpath/queue/nr_requests'"
#
# Set rotational, scheduler and queue depth for Violin multipath devices
SUBSYSTEM=="block", KERNEL=="dm-[0-9]*", OWNER:="oracle", GROUP:="dba", MODE:="660", PROGRAM="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="SVIOLIN*", RUN+="/bin/sh -c 'echo 0 > /sys/$devpath/queue/rotational && echo noop > /sys/$devpath/queue/scheduler && echo 512 > /sys/$devpath/queue/nr_requests'"

Lock and Load

Now that the multipathing and UDEV configuration files are updated, it’s time to reload the UDEV rules and trigger a refresh:

[root@ovs1 rules.d]# udevcontrol reload_rules
[root@ovs1 rules.d]# udevtrigger

Now check that the multipath device has picked up the correct settings:

[root@ovs1 rules.d]# cat /sys/block/dm-0/queue/rotational
0
[root@ovs1 rules.d]# cat /sys/block/dm-0/queue/scheduler
[noop] deadline cfq

Perfect. Next we need to restart the multipathing daemon to use the new configuration we added earlier on (this output is quite big in my demo because I have a lot of paths from array to host):

[root@ovs1 rules.d]# multipath -F
[root@ovs1 rules.d]# multipath -v2
 create: SVIOLIN_SAN_ARRAY_1867AF6CA637606C undef VIOLIN,SAN ARRAY
 size=100G features='0' hwhandler='0' wp=undef
 `-+- policy='round-robin 0' prio=1 status=undef
 |- 6:0:0:1 sdb 8:16 undef ready running
 |- 6:0:1:1 sdc 8:32 undef ready running
 |- 7:0:0:1 sdd 8:48 undef ready running
 |- 6:0:2:1 sde 8:64 undef ready running
 |- 8:0:0:1 sdf 8:80 undef ready running
 |- 7:0:1:1 sdg 8:96 undef ready running
 |- 9:0:0:1 sdh 8:112 undef ready running
 |- 6:0:3:1 sdi 8:128 undef ready running
 |- 8:0:1:1 sdk 8:160 undef ready running
 |- 7:0:2:1 sdj 8:144 undef ready running
 |- 9:0:1:1 sdl 8:176 undef ready running
 |- 10:0:0:1 sdm 8:192 undef ready running
 |- 8:0:2:1 sdo 8:224 undef ready running
 |- 7:0:3:1 sdn 8:208 undef ready running
 |- 9:0:2:1 sdp 8:240 undef ready running
 |- 8:0:3:1 sdq 65:0 undef ready running
 |- 10:0:1:1 sdr 65:16 undef ready running
 |- 9:0:3:1 sds 65:32 undef ready running
 |- 10:0:2:1 sdt 65:48 undef ready running
 |- 10:0:3:1 sdu 65:64 undef ready running
 |- 11:0:0:1 sdv 65:80 undef ready running
 |- 12:0:0:1 sdw 65:96 undef ready running
 |- 13:0:0:1 sdx 65:112 undef ready running
 |- 11:0:1:1 sdy 65:128 undef ready running
 |- 11:0:2:1 sdz 65:144 undef ready running
 |- 12:0:1:1 sdaa 65:160 undef ready running
 |- 11:0:3:1 sdab 65:176 undef ready running
 |- 12:0:2:1 sdac 65:192 undef ready running
 |- 13:0:1:1 sdad 65:208 undef ready running
 |- 12:0:3:1 sdae 65:224 undef ready running
 |- 13:0:2:1 sdaf 65:240 undef ready running
 `- 13:0:3:1 sdag 66:0 undef ready running

That’s it. All that remains is to fire up the Oracle VM Manager and discover the OVM Server…