SUSE Linux Enterprise Server 11 SP3 with Oracle using 4k ASM

stopThis cookbook is part of the Advanced Cookbooks series, for installing Oracle on Violin using a 4k-based ASM diskgroup. If you are unsure what that means, do not proceed any further until you have read this. Configuring ASM to use 4k sectors is interesting, but not necessary – you can achieve excellent performance by following one of the Simple Cookbooks with the added benefit that your installations will take a lot less time… In this cookbook I will be installing Oracle single-instance on SUSE Linux SP3 which has a 3.0.76-0.11 kernel version. There isn’t much deviation from the installations I am more used to on Oracle Linux and Red Hat Linux, with the exception that the 3.x kernel brings with it some changes to the multipath settings and UDEV rules. SUSE has already been installed on the server and I’ve followed the basic Oracle Installation Guide instructions to setup things like the Oracle user and create the relevant directories (e.g. /u01):

server2:~ # cat /etc/SuSE-release
SUSE Linux Enterprise Server 11 (x86_64)
server2:~ # uname -r

I’ve also, for the sake of convenience, disabled the SUSE firewall altogether:

server2:/etc # /sbin/SuSEfirewall2 off
SuSEfirewall2: batch committing...
SuSEfirewall2: Firewall rules unloaded.

I’m not going to include a complete cut and paste of all the usual Oracle pre-installation tasks, so let’s skip straight to the crucial stuff. First up is ASMLib.

Installing and Configuring Oracle ASMLib

I’ve used the truly-horrendous yast application on the SUSE media to install the relevant packages that I require, which includes the oracleasm kernel library:

server2:~ # rpm -qa | grep oracle

Next I need to download the other two parts of ASMLib, the library package and the support module:

server2:~ # wget


100%[===========================================================>] 123,671      238K/s   in 0.5s

2014-03-24 14:29:43 (238 KB/s) - `oracleasm-support-2.1.8-1.SLE11.x86_64.rpm' saved [123671/123671]

server2:~ # wget


100%[===========================================================>] 31,149      --.-K/s   in 0.1s

2014-03-24 14:30:43 (214 KB/s) - `oracleasmlib-2.0.4-1.sle11.x86_64.rpm' saved [31149/31149]

After this I can install the RPMs and then run the configure command to set ASMLib up:

server2:~ # rpm -i oracleasmlib-2.0.4-1.sle11.x86_64.rpm
server2:~ # rpm -i oracleasm-support-2.1.8-1.SLE11.x86_64.rpm
server2:~ # modprobe oracleasm
server2:~ # lsmod | grep oracleasm
oracleasm              58884  0
server2:~ # /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration:                        done
Initializing the Oracle ASMLib driver:                                  done
Scanning the system for Oracle ASMLib disks:                            done

Notice the configuration file placed in the /etc/sysconfig directory. It has some configuration settings that need to be amended:

server2:~ # ls -l /etc/sysconfig/oracleasm*
lrwxrwxrwx 1 root root  24 Mar 24 14:38 /etc/sysconfig/oracleasm -> oracleasm-_dev_oracleasm
-rw-r--r-- 1 root root 973 Mar 24 14:38 /etc/sysconfig/oracleasm-_dev_oracleasm
server2:~ # tail -11 /etc/sysconfig/oracleasm
# ORACLEASM_SCANORDER: Matching patterns to order disk scanning

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan

# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size
# reported by the underlying disk instead of the physical. The default
# is 'false'

I’m going to set the ORACLEASM_SCANORDER value to “dm” because I want Oracle to use the multipath devices. I’m also going to exclude the SCSI (i.e. single path) devices by setting ORACLEASM_SCANEXCLUDE to “sd“. Finally, I need to set ORACLEASM_USE_LOGICAL_BLOCK_SIZE to be true (lower case is required) because I want to be able to present LUNs from Violin which are using 512 byte emulation. If this value were to be left at the default of false then ASMLib would always use the physcial block size from Violin, which is always 4k. ASMLib needs to be restarted to pick up the changed configuration:

server2:~ # /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                                           done
Shutting down the Oracle ASMLib driver:                                 done
Initializing the Oracle ASMLib driver:                                  done
Scanning the system for Oracle ASMLib disks:                            done

ASMLib is now ready, so let’s turn our attention to the UDEV rules.

Configuring UDEV

Linux uses UDEV as its device mapper. The UDEV system uses a set of (some would say complicated) rules to configure and present devices as files within the /dev filesystem. The rules are located in the /dev/udev/rules.d directory and are processed according to filename: each filename has the format nn-name.rules where nn is a two-digit number. This means files which start with lower numbers are processed first. It’s also possible for one of the files to contain a rule which tells UDEV not to process any further rules for the device in question – making it really easy to lose track of what is going on with your device management. In our case, we want to override some of the default choices UDEV makes, such as which I/O scheduler is in use, whether the device is rotational (clue: no!) and so on. Anyway, enough chat. Let’s setup the rules for Violin:

server2:~ # cd /etc/udev/rules.d
server2:/etc/udev/rules.d # vi 12-violin.rules

My rules file will look like this:

KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", RUN+="/bin/sh -c 'echo noop > /sys/$devpath/queue/scheduler'"
KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", RUN+="/bin/sh -c 'echo 0 > /sys/$devpath/queue/rotational'"
KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", RUN+="/bin/sh -c 'echo 1024 > /sys/$devpath/queue/nr_requests'"
KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", RUN+="/bin/sh -c 'echo 2 > /sys/$devpath/queue/rq_affinity'"
KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", RUN+="/bin/sh -c 'echo 1 > /sys/$devpath/queue/nomerges'"
KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", RUN+="/bin/sh -c 'echo 4096 > /sys/$devpath/queue/max_sectors_kb'"
KERNEL=="sd*[!0-9]|sg*", BUS=="scsi", SYSFS{vendor}=="VIOLIN", RUN+="/bin/sh -c 'echo 0 > /sys/$devpath/queue/add_random'"
ENV{DM_UUID}=="mpath-36001b97?*", RUN+="/bin/sh -c 'echo noop > /sys/$devpath/queue/scheduler'"
ENV{DM_UUID}=="mpath-36001b97?*", RUN+="/bin/sh -c 'echo 0 > /sys/$devpath/queue/rotational'"
ENV{DM_UUID}=="mpath-36001b97?*", RUN+="/bin/sh -c 'echo 64 > /sys/$devpath/queue/nr_requests'"
ENV{DM_UUID}=="mpath-36001b97?*", RUN+="/bin/sh -c 'echo 2 > /sys/$devpath/queue/rq_affinity'"
ENV{DM_UUID}=="mpath-36001b97?*", RUN+="/bin/sh -c 'echo 1 > /sys/$devpath/queue/nomerges'"
ENV{DM_UUID}=="mpath-36001b97?*", RUN+="/bin/sh -c 'echo 4096 >  /sys/$devpath/queue/max_sectors_kb'"
ENV{DM_UUID}=="mpath-36001b97?*", RUN+="/bin/sh -c 'echo 0 >  /sys/$devpath/queue/add_random'"

Note that there are two sets of rules – the first are for the SCSI devices (i.e. the /dev/sd* devices corresponding to each single path) and the second are for the /dev/dm-* devices created by the multipathing daemon. The new rules can be picked up with the following two commands:

server2:~ # udevadm control --reload-rules
server2:~ # udevadm trigger

Configuring Multipathing

In SLES 11 SP3 there have been some changes to the various options avialable in the /etc/multipath.conf file. This includes the deprecation of the getuid_callout parameter and the change of path_selector from it’s previous default value of “round robin 0” to a new default of “service-time 0”. However, the testing we have performed at Violin shows that the round robin consistently allows better results under benchmarking, so at least for now I will be sticking with that. There are plenty of other changes from previous multipath configurations, which you can read about here. Here’s my /etc/multipath.conf:

defaults {
blacklist {
	devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
	devnode "^hd[a-z]"
	devnode "^dcssblk[0-9]*"
blacklist_exceptions {
	property "(ID_SCSI_VPD|ID_WWN)"
devices {
        device {
                vendor                  "VIOLIN"
                product                 "SAN ARRAY"
                path_grouping_policy    group_by_serial
                path_selector           "round-robin 0"
                path_checker            tur
                hardware_handler        "0"
                failback                immediate
                rr_weight               uniform
                no_path_retry           fail
        device {
        	vendor                  "VIOLIN"
        	product                 "SAN ARRAY ALUA"
                path_selector           "round-robin 0"
        	prio                    "alua"
        	path_grouping_policy    group_by_prio
        	path_checker            tur
        	hardware_handler        "0"
        	failback                immediate
        	no_path_retry           fail
		detect_prio		yes
multipaths {

There are two entries for Violin here, one for LUNs presented using ALUA and one for the those that aren’t (since the product ID changes). The settings are slightly different as ALUA LUNs should be grouped by priority rather than serial number. The new settings are picked up using the following commands:

server2:/dev/mapper # multipath -F
server2:/dev/mapper # multipath -v1

Now it’s time to create the LUNs I need and present them to the database server.

Presenting Violin Storage

In this cookbook I’m going to build a database using 4k based LUNs (see this page if you don’t know what that means). This guarantees that all I/Os will be aligned to the underlying flash and that the minimum I/O size will always be 4k, which in turn guarantees the very best performance available. I will need one single LUN with a 512 byte sector size in which to create the Oracle SPFILE (it would also be home to other Oracle files such as the OCR and Voting Disks if I were building a cluster). lun-creation-with-highlight In the above picture, which is a snapshot of the LUN creation screen on a Violin array, the LUN sector size is defined by the “Block Size” radio buttons. I’ve created 8 x 250GB LUNs for my +DATA diskgroup, 2 x 1TB LUNs for +RECO (the Fast Recovery Area) and 1 X 10GB LUN for +SYSTEMDG. All except the SYSTEMDG are using 4k sectors. I’ve used the script (found in most flavours of Linux these days) to rescan the SCSI bus and then run the multipath -F and multipath -v1 commands to pick up the new devices, which get presented as multipath aliases in the /dev/mapper directory using names that aren’t nice and friendly. Luckily I can use my script to give them all friendly names. I’ve run it and added the output to the /etc/multipath.conf file, so now I see lots of nicely-named devices in /dev/mapper:

server2:/dev/mapper # ls -l
total 0
lrwxrwxrwx 1 root root       7 Mar 24 16:27 DATA1 -> ../dm-0
lrwxrwxrwx 1 root root       7 Mar 24 16:27 DATA2 -> ../dm-5
lrwxrwxrwx 1 root root       7 Mar 24 16:27 DATA3 -> ../dm-6
lrwxrwxrwx 1 root root       7 Mar 24 16:27 DATA4 -> ../dm-7
lrwxrwxrwx 1 root root       7 Mar 24 16:27 DATA5 -> ../dm-8
lrwxrwxrwx 1 root root       7 Mar 24 16:27 DATA6 -> ../dm-9
lrwxrwxrwx 1 root root       8 Mar 24 16:27 DATA7 -> ../dm-10
lrwxrwxrwx 1 root root       8 Mar 24 16:27 DATA8 -> ../dm-11
lrwxrwxrwx 1 root root       7 Mar 24 16:27 RECO1 -> ../dm-2
lrwxrwxrwx 1 root root       7 Mar 24 16:27 RECO2 -> ../dm-3
lrwxrwxrwx 1 root root       7 Mar 24 16:27 SYSTEMDG -> ../dm-4

These devices are now ready to be labelled by Oracle ASMLib.

Labelling ASMLib Devices

To keep things simple I’m going to give each device the same ASMLib name as it’s device name. I’m also going to overwrite each device with 32MB of zeros first, so it’s nice and clean:

erver2:/dev/mapper # for lun in `ls -1 DATA* RECO* SYSTEMDG`; do dd if=/dev/zero of=$lun bs=32k count=1024; oracleasm createdisk $lun $lun; done
1024+0 records in
1024+0 records out
33554432 bytes (34 MB) copied, 0.0576828 s, 582 MB/s
Writing disk header: done
Instantiating disk: done
1024+0 records in
1024+0 records out
33554432 bytes (34 MB) copied, 0.0546716 s, 614 MB/s
Writing disk header: done
Instantiating disk: done
1024+0 records in
1024+0 records out
33554432 bytes (34 MB) copied, 0.0614032 s, 546 MB/s
Writing disk header: done
Instantiating disk: done
erver2:/dev/mapper # oracleasm listdisks

Ok so the devices are all present and correct, so let’s install Oracle.

Installing Oracle Grid Infrastructure

I don’t think there’s any need to document this in detail, other than to point out a couple of key points. I’m installing Oracle Grid Infrastructure and I’m choosing to configure for a Standalone Installation (mainly because I only have one test server available). When it comes to setting up the ASM diskgroups I’m going to configure my single 10GB LUN as a diskgroup called +SYSTEMDG (using external reduncancy): OUI-configure-SYSTEMDG Because this is SUSE and I didn’t have Oracle’s handy preinstall RPM available I have to allow OUI to create and run a fixup script to change all the various kernel parameters that I missed during my skimming of the Oracle installation guide. Naturally OUI also complains that my swap space isn’t large enough (when is it ever large enough for OUI? That’s a rhetorical question, don’t answer it…) but I tell it I don’t care. The installation completes with no issues and it’s almost time to install the database, but first we need two more ASM disk groups:

oracle@server2:~> . oraenv
ORACLE_SID = [oracle] ? +ASM
The Oracle base has been set to /u01/app/oracle
oracle@server2:~> sqlplus / as sysasm
SQL*Plus: Release Production on Mon Mar 24 17:48:57 2014
Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release - 64bit Production
With the Automatic Storage Management option

     'ORCL:DATA3', 'ORCL:DATA4',
     'ORCL:DATA5', 'ORCL:DATA6',
     'compatible.asm' = '11.2',
     'compatible.rdbms' = '11.2';

Diskgroup created.

     'compatible.asm' = '11.2',
     'compatible.rdbms' = '11.2';

Diskgroup created.

We can check that the diskgroups have the correct sector sizes with the v$asm_diskgroup view, or with my new favourite utility, ASMCMD:

oracle@server2:~> asmcmd
ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N        4096   4096  1048576   2048000  2047918                0         2047918              0             N  DATA/
MOUNTED  EXTERN  N        4096   4096  1048576   2097152  2097080                0         2097080              0             N  RECO/
MOUNTED  EXTERN  N         512   4096  1048576     10240    10181                0           10181              0             N  SYSTEMDG/

So +DATA and +RECO have 4096 sector sizes while +SYSTEMDG has 512… perfect.

Installing and Creating Oracle Database

There is very little of note for  the installation, since I am just choosing the basic single “Install database software only” option from OUI. During the database creation with DBCA however, I need to choose the option to create the database as a set of scripts, because that way I can amend the location of the SPFILE to put it in the +SYSTEMDG diskgroup: DBCA-creation-options One the scripts have been created, I can browse them in the /u01/app/oracle/admin/<ORACLE-SID>/scripts directory:

oracle@server2:~> cd /u01/app/oracle/admin/orcl/scripts/
oracle@server2:/u01/app/oracle/admin/orcl/scripts> ls -l
total 9580
-rw-r----- 1 oracle oinstall    1893 Mar 24 18:10 cloneDBCreation.sql
-rw-r----- 1 oracle oinstall     823 Mar 24 18:10 CloneRmanRestore.sql
-rw-r----- 1 oracle oinstall    2019 Mar 24 18:10 init.ora
-rw-r----- 1 oracle oinstall    2112 Mar 24 18:10 initorclTempOMF.ora
-rw-r----- 1 oracle oinstall    2055 Mar 24 18:10 initorclTemp.ora
-rw-r----- 1 oracle oinstall     507 Mar 24 18:10 lockAccount.sql
-rwxr-xr-x 1 oracle oinstall     511 Mar 24 18:10
-rwxr-xr-x 1 oracle oinstall     940 Mar 24 18:10 orcl.sql
-rw-r----- 1 oracle oinstall     854 Mar 24 18:10 postDBCreation.sql
-rw-r----- 1 oracle oinstall    1008 Mar 24 18:10 postScripts.sql
-rw-r----- 1 oracle oinstall    1457 Mar 24 18:10 rmanRestoreDatafiles.sql
-rw-r----- 1 oracle oinstall 9748480 Mar 24 18:10 tempControl.ctl
oracle@server2:/u01/app/oracle/admin/orcl/scripts> grep -il spfile *

We can see here that there are two files that need amending to change the SPFILE location, so let’s edit them. Here’s the orcl.sql file before I amended it:

set verify off
ACCEPT sysPassword CHAR PROMPT 'Enter new password for SYS: ' HIDE
ACCEPT systemPassword CHAR PROMPT 'Enter new password for SYSTEM: ' HIDE
host /u01/app/oracle/product/11.2.0/dbhome_1/bin/orapwd file=/u01/app/oracle/product/11.2.0/dbhome_1/dbs/orapworcl force=y
host /u01/app/oracle/product/11.2.0/grid/bin/setasmgidwrap o=/u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle
host /u01/app/oracle/product/11.2.0/dbhome_1/bin/srvctl add database -d orcl -o /u01/app/oracle/product/11.2.0/dbhome_1 -p +DATA/orcl/spfileorcl.ora -n orcl -a "DATA,RECO"
host echo "SPFILE='+DATA/orcl/spfileorcl.ora'" > /u01/app/oracle/product/11.2.0/dbhome_1/dbs/initorcl.ora

To change it I need to move the SPFILE to +SYSTEMDG and then also add +SYSTEMDG to the list of diskgroups required by that service in Clusterware. Here are the two modified lines:

host /u01/app/oracle/product/11.2.0/dbhome_1/bin/srvctl add database -d orcl -o /u01/app/oracle/product/11.2.0/dbhome_1 -p +SYSTEMDG/orcl/spfileorcl.ora -n orcl -a "DATA,RECO,SYSTEMDG"
host echo "SPFILE='+SYSTEMDG/orcl/spfileorcl.ora'" > /u01/app/oracle/product/11.2.0/dbhome_1/dbs/initorcl.ora

The other file was postDBCreation.sql, which looks like this:

connect "SYS"/"&&sysPassword" as SYSDBA
set echo on
spool /u01/app/oracle/admin/orcl/scripts/postDBCreation.log append
@/u01/app/oracle/product/11.2.0/dbhome_1/rdbms/admin/catbundle.sql psu apply;
select 'utl_recomp_begin: ' || to_char(sysdate, 'HH:MI:SS') from dual;
execute utl_recomp.recomp_serial();
select 'utl_recomp_end: ' || to_char(sysdate, 'HH:MI:SS') from dual;
execute dbms_swrf_internal.cleanup_database(cleanup_local => FALSE);
connect "SYS"/"&&sysPassword" as SYSDBA
set echo on
create spfile='+DATA/orcl/spfileorcl.ora' FROM pfile='/u01/app/oracle/admin/orcl/scripts/init.ora';
shutdown immediate;
host /u01/app/oracle/product/11.2.0/dbhome_1/bin/srvctl enable database -d orcl;
host /u01/app/oracle/product/11.2.0/dbhome_1/bin/srvctl start database -d orcl;
connect "SYS"/"&&sysPassword" as SYSDBA
spool off

Again I need to amend the diskgroup, but there is only one line to change in this case:

create spfile='+SYSTEMDG/orcl/spfileorcl.ora' FROM pfile='/u01/app/oracle/admin/orcl/scripts/init.ora';

All that remains now is for me to run the script which will create the database and corresponding Clusterware services. I really don’t have enough space to cut and paste all of the output from that, so let’s finish up with a shot from Clusterware showing that everything is up and running:

racle@server2:~> . oraenv
ORACLE_SID = [orcl] ? +ASM
The Oracle base remains unchanged with value /u01/app/oracle
oracle@server2:~> crsctl stat res -t
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
Local Resources
               ONLINE  ONLINE       server2
               ONLINE  ONLINE       server2
               ONLINE  ONLINE       server2
               ONLINE  ONLINE       server2
               ONLINE  ONLINE       server2                  Started
               OFFLINE OFFLINE      server2
Cluster Resources
      1        ONLINE  ONLINE       server2
      1        OFFLINE OFFLINE
      1        ONLINE  ONLINE       server2
      1        ONLINE  ONLINE       server2                  Open

Plus, just for good measure, a quick look at the redo logs to make sure they are sized with a 4k block size:

QL> select * from v$log;

---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ ---------
	 1	    1	       1   52428800	  4096		2 NO  INACTIVE		      995548 24-MAR-14	    1000690 24-MAR-14
	 2	    1	       2   52428800	  4096		2 NO  INACTIVE		     1000690 24-MAR-14	    1008924 24-MAR-14
	 3	    1	       3   52428800	  4096		2 NO  CURRENT		     1008924 24-MAR-14	 2.8147E+14

Great. To be honest those logs aren’t big enough (and there aren’t enough of them) for my purposes going forward, but other than that everything is good to go. Time to close the cookbook and start thinking about subjecting this database to some Oracle SLOB tests…


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: