All About Patching

I spent a number of years working in Oracle Advanced Customer Services – and the world of patching was always one fraught with pain. Here is my attempt to demystify some of the numerous terms and phrases used in the world of Oracle Support.

(Huge respect and thanks to the living legend that is Jesper from Oracle BDE – as I asked for the eighth time what the difference was between a label and a bug it occurred to me that the Danes must have superhuman reserves of patience…)

Oracle’s Support Policies

These are explained here in detail:

https://www.oracle.com/support/lifetime-support/

However, there are three levels of support:

• Premier Support provides maintenance and support for five years from the general availability date of the product.
• Extended Support provides an extra three years of support for specific Oracle releases for an additional fee.
• Sustaining Support provides technical support for the lifetime of the systems.

All three levels of support allow customers access to existing patches, but only Premier and Extended Support allow for the creation of new patches (either to fix a new bug or to backport an existing fix to a different platform or version of Oracle). If a customer only has Sustaining Support and they need a patch that is not available on their platform or patchset, they are not eligible for it unless they have a specific arrangement with Development (an “exception”).

Grace Periods

The customer also needs to be on the correct patchset release in order to be entitled to bugfixes and backports. When a new patchset is released, customers on the previous patchset have a “grace period” where this patchset is still fully supported – the idea being that this period is long enough for the customer to upgrade to the next (i.e. latest) patchset release.

The grace period for the Oracle Database is typically one year from the initial base release of the new patch set, which is always on Linux (because Linux is the environment we develop on). Since new patchsets are released at different times for different platforms, it may be that some of the less common platforms do not have the new patchset released until some considerable time into this grace period. In this case, if the new patchset is released more than nine months after the base release, the grace period will usually be extended to three months from the time of the release.

Using 10.2 as an example, 10.2.0.5 was released on Linux in May 2010. This meant that for most platforms the grace period, i.e. the time during which customers could still get new patches created for 10.2.0.4, ran until May 2011. However, if you were a customer running on IBM z/OS and this didn’t come out until April 2011 (for example), you would have three months from this time (so until July 2011) to upgrade to 10.2.0.5…

The general availability date for 10.2 was July 2005 which meant that Premier Support ended in July 2010, just two months after the release of the terminal patchset 10.2.0.5. Because of this, fees for the first year of Extended Support were being waived. Note however that although customers got a free year of extended support they still only had another three years before extended support finished.

Terminology Used by Oracle

Label – the name for a set of changes (e.g. the fixing of a bug) in the source code; the output of a label is a set of C files (*.c, *.h etc)

PSE – a Patch Set Exception is created when Oracle ports a label to a specific platform, e.g. Linux x86_64, HPUX-IA etc; the output of a PSE is compiled C files (such as *.o files, or *.exe on Windows)

Interim Patch – this is the official name for a one-off patch; to create an interim patch a label and a PSE is required, i.e. a fix or fixes in the source code and then the output of this when it is compiled.

Conflict – two or more bugfixes which modify the same source code files; if this is the case then only one can be applied, otherwise the code for the fixes will need to be “merged” to produce a new label

Merge – a combination of labels; if for example you have two bugfixes which touch the same files then you would need to have those fixes merged together, producing one new patch with a new label

MLR – a Merge Label Request; a merge of two or more fixes requires two steps: a label for the new set of merged code (an MLR) followed by a PSE (for the platform it is required on)

Backport – also known as an OOB or One-Off Backport, this is a change or fix in a later piece of code which is then retrospectively applied as a patch to an earlier level of the code e.g. a fix in 11.1.0.7 might then be backported on top of 10.2.0.5 and made available as an interim patch

BLR – a Backport Label Request; as with merges, a backport requires two phases: a new label (created by the BLR) and then a PSE

Bundle – a set of patches which are released in what is effectively one large merge. Whereas merges are often created for a particular customer’s set of circumstances, a bundle is normally created by Development and Support to include a set of critical or recommended fixes. Bundles are much more thoroughly tested than merges and interim patches. They are also much more likely to cause conflicts because of their size.

CPU – a Critical Patch Update; in essence a CPU is a bundle which is released quarterly and only includes security-related fixes. CPUs are usually cumulative, so a later CPU will contain all of the fixes from an earlier CPU for the same patchset. CPUs are “molecular” which means groups of related fixes (“molecules”) can be excluded if they conflict with existing patches.

PSU – a Patch Set Update; another bundle released every quarter, the PSU contains critical bug fixes for its intended patchset. Initially they are cumulative, so they contain all fixes from previous PSUs for the same patchset – they also include the CPU released on the same date. So for example PSUApr2010 includes CPUApr2010. There are very specific rules about which fixes can be included in a PSU; nothing that affects optimizer query plans can be included, nor anything that requires a configuration change. Port-specific bugs also cannot be included. Application of a PSU will increment the last digit of the release number, e.g. PSUApr2010 for 10.2.0.4 results in the version becoming 10.2.0.4.4

Overlay Patch – when a one-off interim patch conflicts with a PSU, an overlay patch is required. This is basically a merge of the patch you want and the PSU.

Overlay PSU – During the lifecycle of a patchset, PSUs will normally be released every quarter. As the patchset reaches maturity it would be expected that there are less critical bugs which can be considered as candidates for inclusion in PSUs. At some point the number of bug fixes included in each PSU will reach a low enough number that it is no longer considered worthwhile to make the PSU cumulative. At this point the previous PSU becomes a baseline and all future PSUs are released as “overlay PSUs”. An example of this is 10.2.0.4.4 which is the last base PSU for the 10.2.0.4 patchset. From 10.2.0.4.5 all PSUs are overlay PSUs, which means that in order to install, for example, 10.2.0.4.10, the user must first install 10.2.0.4.4 and then install 10.2.0.4.10 on top. The overlay PSU is cumulative down to the base PSU, which means that by installing 10.2.0.4.4 the user automatically gets all of the fixes in 10.2.0.4.3 and below, then by installing 10.2.0.4.10 all of the fixes in 10.2.0.4.9 down to 10.2.0.4.5 are also included. If a conflict were to occur between an interim patch and the overlay PSU then a merge of the overlay PSU would be required. For example, if patch 999999 conflicted with 10.2.0.4.10 then the user would install 10.2.0.4.4 as usual and then require a merge overlay patch of 10.2.0.4.10 merged with 999999.

Examples

You have a totally new bug 123456, which you hit in 10.2.0.4 on AIX. In the SR, a bug is raised with Development who identify the issue and then fix it in the latest version they are working on (right now this would probably be 12.1 and 11.2.0.4). In your SR the analyst would then log a One-Off Backport request (an OOB) which would instruct Development or Sustaining Engineering to create two things: a new label (a BLR) for your bug on the release you need it on, i.e. 10.2.0.4; and then a Patch Set Exception (PSE) to have this compiled on to AIX for your customer. Both the BLR and the PSE would have new bug numbers but ultimately the patch number the customer would need to apply would still be 123456.

You have a bug 654321 which is fixed in 11.1 and you want a backport for an HPUX-Itanium system running 10.2.0.4 – but you discover that this patch would conflict with something you already have applied (666666). You create an SR and the analyst logs an OOB for you. Once this has been approved by BDE and Development, the process would begin: First they would create a label for 10.2.0.4 (logged as a BLR). Once this was complete, they would create a merge label (logged as a Merge Label Request) of the new BLR and the conflicting bug 666666. The result of this MLR would be a totally new patch number for the merge – this number would be the one that you would need to download. Finally however, you would need a PSE to have this merge compiled for your target environment, HPUX-IA.

Confusing isn’t it?