The Ultimate Guide To Oracle with Advanced Format 4k


It’s a brave thing, calling something the “Ultimate Guide To …” as it can leave you open to criticism that it’s anything but. However, this topic – of how Oracle runs on Advanced Format storage systems and which choices have which consequences – is one I’ve been learning for two years now, so this really is everything I know. And from my desperate searching of the internet, plus discussions with people who are usually much knowledgeable than me, I’ve come to the conclusion that nobody else really understands it.

In fact, you could say that it’s a topic full of confusion – and if you browsed the support notes on My Oracle Support you’d definitely come to that conclusion. Part of that confusion is unfortunately FUD, spread by storage vendors who do not (yet) support Advanced Format and therefore benefit from scaring customers away from it. Be under no illusions, with the likes of Western DigitalHGST and Seagate all signed up to Advanced Format, plus Violin Memory and EMC’s XtremIO both using it, it’s something you should embrace rather than avoid.

However, to try and lessen the opportunity for those competitors to point and say “Look how complicated it is!”, I’ve split my previous knowledge repository into two: a high-level page and an Oracle on 4k deep dive. It’s taken me years to work all this stuff out – and days to write it all down, so I sincerely hope it saves someone else a lot of time and effort…!

Advanced Format with 4k Sectors

Advanced Format: Oracle on 4k Deep Dive

New installation cookbook for SUSE Linux Enterprise Server 11 SP3

Exactly what it says on the tin, I’ve added a new installation cookbook for SUSE 11 SP3 which creates Violin on a set of 4k devices.

I’ve started setting the add_random tunable of the noop I/O scheduler because it seems to give a boost in performance during benchmarking runs. If I can find the time, I will blog about this at some point…

For more details read this document from Red Hat.

Oracle ASMLib: Physical and Logical Blocksize

This article is about the use of Advanced Format devices on Oracle’s ASMLib kernel library for Linux. For background, read this page on 4k sector sizes first, otherwise it might all sound like nonsense. Mind you, it mind sound like nonsense anyway, I can’t guarantee anything here. By the way, a big hello to my buddy Nate who asked for this information: you rock, dude.

In more recent versions of ASMLib, Oracle introduced a new parameter into the /etc/sysconfig/oracleasm file:

[root@half-server4 mapper]# tail -5 /etc/sysconfig/oracleasm
# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size
# reported by the underlying disk instead of the physical. The default
# is 'false'

To understand what this parameter does, consider this device which I am presenting from a Violin array:

[root@half-server4 ~]# ls -l /dev/mapper/testlun
lrwxrwxrwx 1 root root 8 Feb 27 15:33 /dev/mapper/testlun -> ../dm-19
[root@half-server4 ~]# fdisk -l /dev/mapper/testlun

Disk /dev/mapper/testlun: 34.4 GB, 34359738368 bytes
255 heads, 63 sectors/track, 4177 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 524288 bytes

The important bit there is highlighted in red. This device has a 4k physical blocksize (as all Violin devices do, as well as many other modern storage systems) but has a 512 byte logical blocksize. Essentially, this LUN is pretending to be a 512 byte based.

Now that’s all well and good. Operating systems and applications that cannot support 4k block sizes (e.g. Red Hat 5 and Oracle Linux 5) will happily use this, because they believe it to be 512 byte. But later versions of ASMLib have started being too clever for their own good.

Don’t Look Behind The Curtain

Let’s create an ASMLib label on this device:

root@half-server4 ~]# oracleasm createdisk TESTLUN /dev/mapper/testlun 
Writing disk header: done
Instantiating disk: done

And now we can attempt to put an ASM diskgroup on it:

     'compatible.asm' = '11.2',
     'compatible.rdbms' = '11.2';  
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15038: disk '' mismatch on 'Sector Size' with target disk group [4096]

What happened? Well, ASMLib has looked behind the smoke and mirrors and decided that this is actually a 4k device. It’s therefore presenting this to Oracle ASM as 4k, which causes the problem (because I explicitly asked for sector size to be 512 byte on this diskgroup).

One possible solution is to change the ASM_DISKSTRING from it’s default value of NULL (meaning ‘ORCL:*’) to ‘/dev/oracleasm/disks/*’, i.e. the location where ASMLib creates its own block devices. We can test this theory with fdisk:

[oracle@half-server4 ~]$ ls -l /dev/oracleasm/disks/TESTLUN 
brw-rw---- 1 oracle dba 252, 19 Feb 27 15:38 /dev/oracleasm/disks/TESTLUN
[oracle@half-server4 ~]$ fdisk -l /dev/oracleasm/disks/TESTLUN | grep "Sector size"
Sector size (logical/physical): 512 bytes / 4096 bytes

So that would work. But it would lose many of the claimed benefits of ASMLib such as reduced file descriptors and context switching. Also, it feels like a hack.


The answer, as you probably guessed, is to set this new parameter. It defaults, wrongly in my opinion, to using the physical block size. We can either edit the value in the file to be true in order to use the logical blocksize, or preferably use the oracleasm configure command:

root@half-server4 ~]# oracleasm configure -b
Writing Oracle ASM library driver configuration: done
[root@half-server4 ~]# oracleasm configure | grep ORACLEASM_USE_LOGICAL_BLOCK_SIZE

It can be set back to using the physical blocksize with the following command:

[root@half-server4 ~]# oracleasm configure -p
Writing Oracle ASM library driver configuration: done
[root@half-server4 ~]# oracleasm configure | grep ORACLEASM_USE_LOGICAL_BLOCK_SIZE

Finally, a word of warning. If you are like me, then you are a bit stupid and can’t follow instructions properly. I set the value of the parameter to TRUE in upper case and then spent hours wondering why it didn’t work. The answer, to my embarrassment, is that it’s case sensitive. TRUE is not a valid value so it defaults to false. Doh.

New script for setting up multipath.conf entries

I’ve added a new script to the Useful Scripts page called which automates the process of creating entries for the /etc/multipath.conf file on Red Hat 6 / Oracle Linux 6.

As the name suggests, I wrote it with Violin devices in mind, but there should be overlap with other storage which will potentially make it useful elsewhere… see here for more details.

Oracle VM 3.1.1 with Violin Memory storage

Oracle-VMAs part of the Installation Cookbook series I have now posted a new entry on how to install Oracle VM with Violin Memory flash storage:

Oracle VM 3.1.1 with Violin Memory storage

Using Oracle Preinstall RPM with Red Hat 6

Recently I’ve been building Red Hat 6 systems and struggling to use the Oracle Preinstall RPM because it has a dependency on the Oracle Unbreakable Enterprise Kernel.

I’ve posted an article on this subject and my methods for getting around it:

Using Oracle Preinstall RPM with Red Hat 6

This system is not registered with ULN / RHN

One of the features of WordPress is the ability to see search terms which are taking viewers to your blog. One of the all-time highest searches bringing traffic to my site is “This system is not registered with ULN”… and sure enough if I search for that phrase on Google my site is one of the top links, taking people to one of my Violin Memory array Installation Cookbooks.

So I guess it’s only fair that I give these passers by some sort of advice on what to do if you see this message…

1. Don’t Panic

Chances are you have built a new Linux system using Red Hat Enterprise Linux or its twin sister Oracle Linux. You are probably now trying to use yum to install software packages, but every time you do so you see something similar to this:

# yum install oracle-validated -y
Loaded plugins: rhnplugin, security
This system is not registered with ULN.
ULN support will be disabled.
ol5_u7_base | 1.1 kB 00:00
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package oracle-validated.x86_64 0:1.1.0-14.el5 set to be updated

This is the Oracle Linux variant. If it was Red Hat you would see:

This system is not registered with RHN
RHN support will be disabled.

The first thing to understand is that you can quite happily build and run a system in this state – in fact I’m willing to bet there are many systems out there exhibiting this message every time somebody calls yum.

The message simply means that you have not registered this build of your system with Oracle or Red Hat. Both companies have a paid support offering which allows you to register a system and do things like get software updates. Red Hat’s is call the Red Hat Network (RHN) and Oracle’s, with their marketing department’s usual sense of humour, is called the Oracle Unbreakable Linux Network (ULN). [I’ve been critical of some Oracle products in the past but I have to say that I love Oracle Linux, even if I do find the name “Unbreakable” a bit daft…]

You don’t have to register your system with the vendor’s support network in order to be able to use it. I’m not making any statements about your support contract, if you have one – I’m just saying that it will work quite normally without it.

2. Register Your System

If you have Oracle or Red Hat support then you might as well register your system so that it can take advantage of their yum channels.

In systems prior to RHEL6 / OL6 you used a utility called up2date to register:

# up2date --register

Or if you want to use the text-mode version:

# up2date-nox --register

You can find a good tutorial explaining the process here. In RH6 / OL6 the process changed, so now you call the relevant utility.

For Red Hat you need to use the rhn_register command (actually this also became available in RHEL5). You will need your Red Hat Network login and password.

For Oracle Linux you need to use the uln_register command. You will need your Oracle ULN login, password and Customer Service Identifier (CSI).

Once your registration is complete the message “This system is not registered” should leave you alone.

3. Don’t Have Support?

Of course, not everybody has a support contract with Red Hat or Oracle. Some people have one but can’t find the details. Others can’t be bothered to set it up. If any of these applies to you then there is another alternative, which is the Oracle Public Yum Server. [Fair play to Wim and the team for making this available, because it’s been making my life easier for years now…]

Oracle’s public yum server is a freely available source of Linux OS downloads. Simply point your browser here and follow the instructions:

In essence, you use wget to download the Oracle repo file which relates to your system and then (optionally) edit it to choose the yum channel you want to subscribe to (otherwise it will use the latest publicly-available stuff). The versions of the Linux software on the public yum repositories are (I believe) not as up-to-date as those you would get if you subscribed to a support contract, but they are still very new.

And the best bit is you can also use it if you have Red Hat installed; it isn’t restricted to Oracle Linux users. Having said that, make sure you don’t do anything which invalidates your support contracts. By pointing a RHEL system at the Oracle public yum server and running an update you are effectively converting your system to become Oracle Linux.

Here’s an example of how to set it up for OL6:

# cd /etc/yum.repos.d
# wget

This gets yum working and so allows for the Oracle Validated / Oracle Preinstall RPM to be installed in order to setup the database…

Hopefully that will satisfy some of my wayward blog visitors!