Cell Offload Processing

Disclaimer: This post was written in 2012 and covers the Exadata X2 model. I have no plans to update it to cover the later models, so please be aware that the information below is no longer be up to date.

Exadata has the ability to offload processing from the database nodes to the storage servers, or “cells” – this is known as “Cell Offload Processing”. There are many possible situations where processing can be offloaded to the cells, however by far the most important (both from a marketing and performance perspective) is the ability to offload SQL processing. This feature is called “Smart Scan” and will be discussed in the next section.

Non-Exadata systems are unable to offload processing to storage because on a traditional database system (for example servers connected to a storage array via fibre channel) the storage has no intelligence. Oracle claims that the Exadata storage cells are an “Intelligent Storage Grid”. The basis of this claim is that each storage cell is actually a server with CPU and an operating system, running Exadata cell software. The database servers are able to communicate with the cells using a new, proprietary protocol from Oracle called the “Intelligent Database protocol (iDB)”. Instead of receiving block-level IO commands in the manner of a traditional storage array, iDB allows cells to understand the type of workload being processed and – potentially – alter their behaviour accordingly.

Examples of operations that can be offloaded to storage include:

• Smart Scan
• File Creation (cells can create datafiles in the background so that foreground sessions do not have to wait)
• Uncompression of certain types of compressed data during smart scans
• Unencryption of encrypted data during smart scans (using hardware assistance from the Intel Westmere CPUs on the storage servers)
• Incremental backup filtering

In addition to this, iDB allows for many additional details about each IO call to be passed to the storage layer. This enables the cells to prioritize IO calls based on configuration of the “I/O resource manager” as well as to enable Quality of Service rules to be applied. Again this is impossible on a traditional database implementation because the storage layer is unable to understand or act on this information.

Kevin Closson, ex-Performance Architect for Oracle Exadata (now working at EMC), criticised Oracle’s implementation of Cell Offload Processing in that the sharing of processing power is asymmetrical. There are 168 CPU cores in the storage servers on an X2-2 full rack in addition to the 96 cores on the database servers. However, if the database servers are running at capacity on CPU any spare capacity on the storage servers cannot be used except within the strict bounds of the cell offload processing features mentioned above. Additionally individual storage cells cannot pool resources, so whilst one cell may be at 100% CPU another may be running idle – again there is no method of sharing these resources. From this perspective Exadata storage can be viewed as a shared-nothing MPP system.

If you are interested in learning the (dramatic) implications of this last statement, take some time to visit this page and watch the videos. I guarantee you will learn something useful.