Oracle 12.1.0.2 ASM Filter Driver: First Impressions
July 25, 2014 21 Comments
This is a very quick post, because I’m about to log off and take an extended summer holiday (or vacation as my crazy American friends call it… but then they call football “soccer” too). Before I go, I wanted to document my initial findings with the new ASM Filter Driver feature introduced in this week’s 12.1.0.2 patchset. [For a more recent post on this topic, read here]
Currently a Linux-only feature, the ASM Filter Driver (or AFD) is a replacement for ASMLib and is described by Oracle as follows:
Oracle ASM Filter Driver (Oracle ASMFD) is a kernel module that resides in the I/O path of the Oracle ASM disks. Oracle ASM uses the filter driver to validate write I/O requests to Oracle ASM disks.
The Oracle ASMFD simplifies the configuration and management of disk devices by eliminating the need to rebind disk devices used with Oracle ASM each time the system is restarted.
The Oracle ASM Filter Driver rejects any I/O requests that are invalid. This action eliminates accidental overwrites of Oracle ASM disks that would cause corruption in the disks and files within the disk group. For example, the Oracle ASM Filter Driver filters out all non-Oracle I/Os which could cause accidental overwrites.
Interesting, eh? So let’s find out how that works.
Installation
I found this a real pain as you need to have 12.1.0.2 installed before the AFD is available to label your disks, yet the default OUI mode wants to create an ASM diskgroup… and you cannot do that without any labelled disks.
The only solution I could come up with was to perform a software-only install, which in itself is a pain. I’ll skip the numerous screenshots of that part though and just skip straight to the bit where I have 12.1.0.2 Grid Infrastructure installed.
I’m following these instructions because I am using a single-instance Oracle Restart system rather than a true cluster.
First of all we need to do this:
[oracle@server3 ~]$ $ORACLE_HOME/bin/asmcmd dsset 'AFD:*' [oracle@server3 ~]$ $ORACLE_HOME/bin/asmcmd dsget parameter:AFD:* profile:AFD:* [oracle@server3 ~]$ srvctl config asm ASM home: Password file: ASM listener: LISTENER Spfile: /u01/app/oracle/admin/+ASM/pfile/spfile+ASM.ora ASM diskgroup discovery string: AFD:*
Then we need to stop HAS and run the AFD_CONFIGURE command:
[root@server3 ~]# $ORACLE_HOME/bin/crsctl stop has -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'server3'
CRS-2673: Attempting to stop 'ora.asm' on 'server3'
CRS-2673: Attempting to stop 'ora.evmd' on 'server3'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'server3'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'server3' succeeded
CRS-2677: Stop of 'ora.evmd' on 'server3' succeeded
CRS-2677: Stop of 'ora.asm' on 'server3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'server3'
CRS-2677: Stop of 'ora.cssd' on 'server3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'server3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@server3 ~]# $ORACLE_HOME/bin/asmcmd afd_configure
Connected to an idle instance.
AFD-627: AFD distribution files found.
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.
ASMCMD-9524: AFD configuration failed 'ERROR: OHASD start failed'
Er… that’s not really what I had in mind. But hey, let’s carry on regardless:
[root@server3 oracleafd]# $ORACLE_HOME/bin/asmcmd afd_state Connected to an idle instance. ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'DEFAULT' on host 'server3.local' [root@server3 oracleafd]# $ORACLE_HOME/bin/crsctl start has CRS-4123: Oracle High Availability Services has been started.
Ok it seems to be working. I wonder what it’s done?
Investigation
The first thing I notice is some Oracle kernel modules have been loaded:
[root@server3 ~]# lsmod | grep ora oracleafd 208499 1 oracleacfs 3307969 0 oracleadvm 506254 0 oracleoks 505749 2 oracleacfs,oracleadvm
I also see that, just like ASMLib, a driver has been plonked into the /opt/oracle/extapi directory:
[root@server3 1]# find /opt/oracle/extapi -ls 2752765 4 drwxr-xr-x 3 root root 4096 Jul 25 15:15 /opt/oracle/extapi 2752766 4 drwxr-xr-x 3 root root 4096 Jul 25 15:15 /opt/oracle/extapi/64 2753508 4 drwxr-xr-x 3 root root 4096 Jul 25 15:15 /opt/oracle/extapi/64/asm 2756532 4 drwxr-xr-x 3 root root 4096 Jul 25 15:15 /opt/oracle/extapi/64/asm/orcl 2756562 4 drwxr-xr-x 2 root root 4096 Jul 25 15:15 /opt/oracle/extapi/64/asm/orcl/1 2756578 268 -rwxr-xr-x 1 oracle dba 272513 Jul 25 15:15 /opt/oracle/extapi/64/asm/orcl/1/libafd12.so
And again, just like ASMLib, there is a new directory under /dev called /dev/oracleafd (whereas for ASMLib it’s called /dev/oracleasm):
[root@server3 ~]# ls -la /dev/oracleafd/ total 0 drwxrwx--- 3 oracle dba 80 Jul 25 15:15 . drwxr-xr-x 21 root root 15820 Jul 25 15:15 .. brwxrwx--- 1 oracle dba 249, 0 Jul 25 15:15 admin drwxrwx--- 2 oracle dba 40 Jul 25 15:15 disks
The disks directory is currently empty. Maybe I should create some AFD devices and see what happens?
Labelling
So let’s look at my Violin devices and see if I can label them:
root@server3 mapper]# ls -l /dev/mapper total 0 crw-rw---- 1 root root 10, 236 Jul 11 16:52 control lrwxrwxrwx 1 root root 7 Jul 25 15:49 data1 -> ../dm-3 lrwxrwxrwx 1 root root 7 Jul 25 15:49 data2 -> ../dm-4 lrwxrwxrwx 1 root root 7 Jul 25 15:49 data3 -> ../dm-5 lrwxrwxrwx 1 root root 7 Jul 25 15:49 data4 -> ../dm-6 lrwxrwxrwx 1 root root 7 Jul 25 15:49 data5 -> ../dm-7 lrwxrwxrwx 1 root root 7 Jul 25 15:49 data6 -> ../dm-8 lrwxrwxrwx 1 root root 7 Jul 25 15:49 data7 -> ../dm-9 lrwxrwxrwx 1 root root 8 Jul 25 15:49 data8 -> ../dm-10 lrwxrwxrwx 1 root root 7 Jul 11 16:53 VolGroup-lv_home -> ../dm-2 lrwxrwxrwx 1 root root 7 Jul 11 16:53 VolGroup-lv_root -> ../dm-0 lrwxrwxrwx 1 root root 7 Jul 11 16:52 VolGroup-lv_swap -> ../dm-1
The documentation appears to be incorrect here, when it says to use the command $ORACLE_HOME/bin/afd_label. It’s actually $ORACLE_HOME/bin/asmcmd with the first parameter afd_label. I’m going to label the devices called /dev/mapper/data*:
[root@server3 mapper]# for lun in 1 2 3 4 5 6 7 8; do > asmcmd afd_label DATA$lun /dev/mapper/data$lun > done Connected to an idle instance. Connected to an idle instance. Connected to an idle instance. Connected to an idle instance. Connected to an idle instance. Connected to an idle instance. Connected to an idle instance. Connected to an idle instance. root@server3 mapper]# asmcmd afd_lsdsk Connected to an idle instance. -------------------------------------------------------------------------------- Label Filtering Path ================================================================================ DATA1 ENABLED /dev/mapper/data1 DATA2 ENABLED /dev/mapper/data2 DATA3 ENABLED /dev/mapper/data3 DATA4 ENABLED /dev/mapper/data4 DATA5 ENABLED /dev/mapper/data5 DATA6 ENABLED /dev/mapper/data6 DATA7 ENABLED /dev/mapper/data7 DATA8 ENABLED /dev/mapper/data8
That seemed to work ok. So what’s going on in the /dev/oracleafd/disks directory now?
[root@server3 ~]# ls -l /dev/oracleafd/disks/ total 32 -rw-r--r-- 1 root root 26 Jul 25 15:52 DATA1 -rw-r--r-- 1 root root 26 Jul 25 15:49 DATA2 -rw-r--r-- 1 root root 26 Jul 25 15:49 DATA3 -rw-r--r-- 1 root root 26 Jul 25 15:49 DATA4 -rw-r--r-- 1 root root 26 Jul 25 15:49 DATA5 -rw-r--r-- 1 root root 26 Jul 25 15:49 DATA6 -rw-r--r-- 1 root root 26 Jul 25 15:49 DATA7 -rw-r--r-- 1 root root 26 Jul 25 15:49 DATA8
There they are, just like with ASMLib. But look at the permissions, they are all owned by root with read-only privs for other users. In an ASMLib environment these devices are owned by oracle:dba, which means non-Oracle processes can write to them and corrupt them in some situations. Is this how Oracle claims the AFD protects devices?
I haven’t had time to investigate further but I assume that the database will access the devices via this mysterious block device:
[oracle@server3 oracleafd]$ ls -l /dev/oracleafd/admin brwxrwx--- 1 oracle dba 249, 0 Jul 25 16:25 /dev/oracleafd/admin
It will be interesting to find out.
Distruction
Of course, if you are logged in as root you aren’t going to be protected from any crazy behaviour:
[root@server3 ~]# cd /dev/oracleafd/disks [root@server3 disks]# ls -l total 496 -rw-r--r-- 1 root root 475877 Jul 25 16:40 DATA1 -rw-r--r-- 1 root root 26 Jul 25 15:49 DATA2 -rw-r--r-- 1 root root 26 Jul 25 15:49 DATA3 -rw-r--r-- 1 root root 26 Jul 25 15:49 DATA4 -rw-r--r-- 1 root root 26 Jul 25 15:49 DATA5 -rw-r--r-- 1 root root 26 Jul 25 15:49 DATA6 -rw-r--r-- 1 root root 26 Jul 25 15:49 DATA7 -rw-r--r-- 1 root root 26 Jul 25 15:49 DATA8 [root@server3 disks]# od -c -N 256 DATA8 0000000 / d e v / m a p p e r / d a t a 0000020 8 \n 0000032 [root@server3 disks]# dmesg >> DATA8 [root@server3 disks]# od -c -N 256 DATA8 0000000 / d e v / m a p p e r / d a t a 0000020 8 \n z r d b t e 2 l I n i t i a 0000040 l i z i n g c g r o u p s u 0000060 b s y s c p u s e t \n I n i t 0000100 i a l i z i n g c g r o u p 0000120 s u b s y s c p u \n L i n u x 0000140 v e r s i o n 3 . 8 . 1 3 - 0000160 2 6 . 2 . 3 . e l 6 u e k . x 8 0000200 6 _ 6 4 ( m o c k b u i l d @ 0000220 c a - b u i l d 4 4 . u s . o r 0000240 a c l e . c o m ) ( g c c v 0000260 e r s i o n 4 . 4 . 7 2 0 1 0000300 2 0 3 1 3 ( R e d H a t 4 0000320 . 4 . 7 - 3 ) ( G C C ) ) 0000340 # 2 S M P W e d A p r 1 0000360 6 0 2 : 5 1 : 1 0 P D T 2 0000400
Proof, if ever you need it, that root access is still the fastest and easiest route to total disaster…
[Update July 2015: Ok, so look. I was wrong in this post – these /dev/oracleafd/disks devices are simply pointers to devices in /dev/dm-* and thus I was only overwriting the pointer. To read a more accurate post on the subject, please read here]
flashdba 
Pingback: Oracle
What about the Fresh installation, do I have to instal ASM Lib then migrate? or what is the procedure for fresh installation?
For a fresh installation there is no need to install ASMLib, just follow the instructions in the Oracle documentation.
Pingback: Oracle
Pingback: Oracle
Actually you ARE Protected from root users doing anything crazy …. check your /var/log/messages if you try and dd the underlying device … its blocked…. the permissions issues will be resolved in updates but the driver still works regardless of that anomoly
Thanks Jamie, that’s interesting as I’ve definitely overwritten an ASM Filter Driver device using dd as the root user. I discussed that with the ASM development team and they told me my findings were expected. I will revisit and retest with the latest version.
Hmm. Looking at the comment from Kamus I think maybe there was a miscommunication with the ASM development team. I told them that when I ran dd as root against the device I could no longer see the data with an os tool like od, i.e. I thought it had been trashed. They told me this was expected behaviour, but looking at Kamus’s blog post perhaps they meant it’s expected that you cannot see anything until you restart ASM…
In my test, even you can see massive data by “od”, or can’t see anything (by using dd if=/dev/zero of=/dev/mapper/data8 to erase all data in that disk), you can still mount the ASM diskgroup again which contains that disk, you can create tablespace and insert data into that disk, everything is OK. You can check my post, almost in Chinese, but the test process is in English. http://www.dbform.com/html/2014/2283.html
Thank you, that is really interesting. I really need to find some time (and lab gear) to test this again more thoroughly…
Thank you for the post. Can you please clarify the following:
> ASMFD will work on RHEL (6.5) and does not require an OEL?
> With reference to the following statement in the ASM Admin Guide:
“You can also set udev rules, in addition to or instead of Oracle ASMFD and Oracle ASMLIB, for device persistence.”
With ASMFD, is udev still needed? If not then what would be better b/w ASMFD and udev to manage disk bindings?
Thank you
I can’t really speak for Oracle here, but my interpretation of this is that – just as was the case with ASMLib – you can use either/or ASMFD and UDEV for device persistence. So no, UDEV is not still needed – but I would always configure it anyway, because I think it’s good practice to keep things tidy.
Hello,
All the disk labels you are seeing under /dev/oracleafd/admin/ are just text files. So there is no point in trying to corrupt it using “dmesg >> DATA8”. These text files contains the names of the actual devices. Here is the output from my system:
[root@ltc-rac01 ~]# $ORACLE_HOME/bin/asmcmd afd_lsdsk
Connected to an idle instance.
——————————
Label Filtering Path
==============================
OCRVOTE12C1 ENABLED /dev/dm-4
DATA1 ENABLED /dev/dm-6
[root@ltc-rac01 ~]#
[root@ltc-rac01 ~]# ls -l /dev/oracleafd/
total 0
brwxrwx— 1 oracle dba 252, 0 Dec 29 18:38 admin
drwxrwx— 2 oracle dba 120 Dec 29 18:38 disks
[root@ltc-rac01 ~]# file /dev/oracleafd/admin
/dev/oracleafd/admin: block special (252/0)
[root@ltc-rac01 ~]# ls -lrt /dev/oracleafd/disks/
total 16
-rw-r–r– 1 root root 10 Dec 29 18:38 OCRVOTE12C1
-rw-r–r– 1 root root 10 Dec 29 18:38 DATA1
[root@ltc-rac01 ~]# file /dev/oracleafd/disks/*
/dev/oracleafd/disks/DATA1: ASCII text
/dev/oracleafd/disks/OCRVOTE12C1: ASCII text
[root@ltc-rac01 ~]#
[root@ltc-rac01 ~]# cat /dev/oracleafd/disks/OCRVOTE12C1
/dev/dm-4
[root@ltc-rac01 ~]# cat /dev/oracleafd/disks/DATA1
/dev/dm-6
[root@ltc-rac01 ~]#
Cheers,
Rijesh
Pingback: Oracle Automatic Storage Management Filter Driver (Oracle ASMFD) – All Things Oracle
Thanks, good article.
Have a question. I have 3 disks that I used to build a Grid Infrastructure (Oracle Restart) install. Is there a way I can label those disks once they are in use? For example,
{code}
asmcmd afd_label ‘grid_disk1’ ‘/dev/raw/raw1’
ORA-15227: could not perform label set/clear operation
ORA-15031: disk specification ‘/dev/raw/raw1’ matches no disks (DBD ERROR: OCIStmtExecute)
ASMCMD-9513: ASM disk label set operation failed.
{code}
Or as I suspect is this chicken and egg scenario?
Hi Will, I’m a little confused by the “matches no disks” error message that you are receiving here. I wouldn’t expect that error message to result from the disks being in use… more that the block device /dev/raw/raw1 either doesn’t exist or isn’t accessible?
Thanks for the prompt response, my knowledge is basic, I installed GI as standalone (not software only – been reading your other post), I was required to spec 3 raw devices which my unix sysadmin gave me:
grid@xxxxxxx:/home/grid $ ls -l /dev/raw/*
crw-rw—- 1 grid dba 162, 1 Apr 22 11:17 /dev/raw/raw1
crw-rw—- 1 grid dba 162, 2 Apr 5 13:52 /dev/raw/raw2
crw-rw—- 1 grid dba 162, 3 Apr 5 13:52 /dev/raw/raw3
crw-rw—- 1 grid dba 162, 0 Mar 16 13:02 /dev/raw/rawctl
These are in used as +GRID disk group in the GI ASM instance +ASM.
I followed the ASMFD installation guide….by the letter…hit Bug 22717225, but by all intents I I have ASMFD installed.
grid@xxxxxxx:/home/grid $ $ORACLE_HOME/bin/asmcmd afd_state
ASMCMD-9526: The AFD state is ‘LOADED’ and filtering is ‘ENABLED’ on host ‘xxxxxxxx’
I’m working an SR but it’s errrm slow to say the least, so am doing my own research…
Do those devices have ASM header information on them already? You can tell by reading the first 256 bytes using od (I’m assuming this is Linux):
od -c -N 256 /dev/raw/raw1
You may also be able to use kfod to see if the devices are visible, but I’m afraid I don’t have a 12.1 installation available right now to work out the syntax.
Thanks once again. Running od…
grid@xxxxxxxx:/home/grid $ od -c -N 256 /dev/raw/raw1
od: /dev/raw/raw1: read error: Invalid argument
0000000
Thanks for your insight – I’m not expecting you to support me but am grateful for your replies. Unless there is something obvious I am missing here, I will prompt Oracle for another update on my SR.
I’m not convinced your raw devices are set up properly. You should be able to read from them without getting errors… To be honest I’ve never been a fan of creating raw devices in the way that your sysadmin has.
When I provision storage for ASM on Linux I tend to do it completely differently:
Thanks, and thanks for the time and sharing the link, I will try and have a chat with our sysadmin……..