Exadata Software

Disclaimer: This post was written in 2012 and covers the Exadata X2 model. I have no plans to update it to cover the later models, so please be aware that the information below is no longer be up to date.

Oracle Exadata runs a complete stack of software from Oracle. In the case of patching, all patches are provided by Oracle in a distinct set of downloadable patch bundles.

Operating Systems

For the database servers customers have a choice of two x86 (64bit) based operating systems:

• Oracle Linux 5
• Solaris 11

The vast majority of customers choose Oracle Linux, which is a clone of Red Hat Enterprise Linux.

As of February 2012 (Exadata release 11.2.2.4.0) the X2-2 database servers run OL5 Update 5 (2.6.18-238) with the standard Red Hat kernel. However the X2-8 database servers run OL5 Update 5 (2.6.32-100) and use Oracle’s own Unbreakable Enterprise Kernel (UEK).

Oracle claims that this kernel, whilst based on the Red Hat kernel, contains many fixes and performance improvements leading to the following performance increases:

More than 75 percent performance gain demonstrated in OLTP performance tests over a Red Hat Compatible Kernel; 200 percent speedup of Infiniband messaging; 137 percent faster solid state disk access

The ULK was first introduced on to the X2-8 variant of Exadata to give optimisations when addressing the large amount of physical memory on each X2-8 database server (1TB). It is likely that the X2-2 database servers will transition to using this kernel at some point.

There is no choice of operating system for the storage servers, which always run Oracle Linux. As of February 2012 (Exadata release 11.2.2.4.0) the version in use is OL5 Update 5 (2.6.18-238) with the standard Red Hat kernel.

Exadata storage servers also run the Exadata Storage Software which has a five digit release number (e.g. 11.2.2.4.0) similar, but not identical, to that of the database software.

An unusual but supported scenario is the use of a customer’s own “gold build” Linux distribution on the Exadata database servers. Although these servers are supplied pre-installed with the Exadata database server Linux image, in theory a customer is allowed to replace this with another build of their own creation – providing that the usual support guidelines for Oracle database and grid infrastructure software are supported. The main caveat to this scenario is that the Infiniband drivers must be those supplied for the relevant Exadata software release. Oracle attempts to dissuade customers from choosing this configuration method in order to maintain the supportability of these servers.

Until March 2012 the version of Solaris available for Exadata was Solaris 11 Express. However on 13th March 2012 a support notice was published by Oracle (Note 1429907.1) announcing that Solaris 11 Express was to be desupported after six months on 6th September 2012  and that “Systems running Oracle Solaris 11 Express 2010.11 must be upgraded to Oracle Solaris 11 2011.11 before then”.

Database and Grid Infrastructure Software

Exadata database servers run Oracle Grid Infrastructure 11g Release 2 (which contains both the Automatic Storage Manager [ASM] and Oracle Clusterware [CRS] products) and Oracle Database 11g Release 2.

In Oracle terminology each installation of a product on a server is called a “home”. In a clustered environment each node of a cluster would have a corresponding home for each installed product.

In Exadata, each database server has one Grid Infrastructure (GI) home and one or more database (RDBMS) homes. RDBMS homes can be of varying versions and patchset levels providing the GI home is at the highest release level.

Databases running on Exadata run out of a specific set of homes, one per node. A typical environment would see each database using Real Application Clusters (RAC) to run in a clustered manner, with an instance on each database server. However, it is possible for customers to choose only a subset of nodes on which to run database instances. Equally it would be possible for a customer to avoid the cost of RAC licenses altogether by running on single-instance databases on a single Exadata database server – however this would introduce a single point of failure and so would be an unlikely configuration.

Patching

Exadata systems are patched on a proactive basis at three levels:

• Database and Grid Infrastructure (Exadata bundle patches)
• Exadata Storage Software
• Infiniband Switch firmware

In addition to these levels there are infrequent patches made available for the PDU and KVM units, but Oracle does not ask customers to patch these components proactively.

Exadata Bundle Patches

Exadata Bundle Patches (BPs) are released on a monthly to quarterly basis for application to both the grid infrastructure and database homes. BPs contain fixes from Oracle’s standard quarterly Patch Set Updates (PSUs) which may address security issues or bugs in functionality, as well as Exadata-specific fixes. In this sense Exadata BPs are supersets of the traditional PSUs known to non-Exadata customers.

Exadata bundle patches can be installed in a rolling manner, patching each node in turn to avoid taking a full service outage (assuming Real Application Clusters is in use).

The patching of the Grid Infrastructure home can have manageability implications, particularly on a consolidation environment, because all database homes will be affected.

Exadata Storage Software

The Exadata storage software can be patched on a rolling basis by taking each storage server down in turn – however, unless ASM high redundancy (triple mirroring) is in use this will leave customers exposed with no resilience during the patching cycle. A standard Exadata patch can take approximately two hours, resulting in a 28 hour patching window when using the rolling method on a full rack.

The storage software patches can update not only the storage software on each cell but also the operating system, kernel version, firmware for components such as the LSI disk controller, Integrated Lights Out Management (ILOM) card and Infiniband drivers.

An optional addition which used to be contained in the storage server patch download was the “minimal pack” (previously named by Oracle as the “convenience pack” but renamed in 2011). This was a similar type of patch which could be used to update the database servers – again it updates the operating system, kernel, firmware and Infiniband drivers, but did not affect any of the database or grid infrastructure software. From Exadata Storage Server software version 11.2.3.1 (released in March 2012) the minimal pack was deprecated in favour of a new yum channel using the Oracle Unbreakable Linux Network.

Exadata storage software patches are released approximately every quarter.

Infiniband Patches

The Sun Datacenter Infiniband switches have firmware which is patched using a separate download. New firmware is released on a bi-annual to annual basis. Patching can take place online by patching the switches in a rolling manner.