Patching with Confidence!

I was reading this recent article regarding the latest security patches released by Oracle, and it had me thinking about how to best manage patching cycle.

At the outset it addresses a frightening reality:
“In many cases, companies, especially large ones with many databases, are reluctant to bring down production databases to implement new patches. Many are also wary about deploying untested patches in live environments or need to wait for their packaged application vendors to test and certify the patches before they can be deployed"

“As a result, there usually is a considerable lag time between when a patch becomes available from Oracle and when it gets deployed. In some cases, the lag can be months. Other users simply skip entire patch cycles and choose to deploy the patches on a yearly or twice-yearly basis.”

Furthermore, the article said: “…of 305 Oracle database administrators from 14 Oracle user groups between August 2007 and January 2008 and found that two-thirds of Oracle DBAs apparently are not installing Oracle's security patches at all, no matter how critical the vulnerabilities are.”

This drew the author to the make the following stark but obvious conclusion: “…such practices can leave companies dangerously exposed to attacks directed against database vulnerabilities.”

Patch Management is no that difficult. So why are organisations still struggling? Many DBA’s and Operations Managers I’ve spoke to mention three recurring themes.

1. Application Regression Testing- often, because of 3rd party application certification dependencies, there is a requirement to wait for the next application refresh cycle so that appropriate regression testing can be completed prior to upgrading.
2. Confidence is often low because - regardless of the fact that an update has gone through certification - many DBA's have a story to tell about when a tested patch in dev worked differently in production as a result of environmental differences.
3. The timing and time required to apply an update. DBA's need to allocate time to apply and test the update in a test environment, then they need to find a slot in the upgrade schedule to apply it in production. All of this needs to multiplied by the number of environments that need updating and the number of updates to be applied. Clearly, mechanisms are required to cut down cycle times and automate recurring tasks.

Patch Management requires planning and change management to ensure that each updateis priorotised and scheduled accordingly. The DBA should review each vulnerability patch and make a determination on the criticality and impact on their specific environment. Often not all patches need to be applied based on the broader security setup and criticality of the system.

As it stands today, Oracle provides a robust set of tools under our System and Application Management (SAM) portfolio.Real Application Testing, an innovation in Oracle Database 11g, allows database workloads to be efficiently captured in production and replayed for testing purposes. The facility faithfully replays transactions with order and timing preserved. It then reports on the performance of the test and highlights areas of change. Using Oracle Testing capabilities, confidence is increased are tests are based on real production workloads rather than synthetic test scenarios. Testing cycles are therefore shortened, not only because the test is largely automated, but also because the increased confidence associated with using real production workloads.

Furthermore, Oracle Enterprise Manager Provisioning Pack provides automated provisioning and patching facilities that make it easier to locate and apply patches. It also drastically reduces the effort required to patch multiple systems, which is great news if you’ve got scores of databases to manage.

Finally, when it comes to applying updates to production environments, 11g “Hot patching” means that it will be possible to apply an update to the Oracle binary while the database continues to run avoiding small downtime windows.

The article goes on to say that of DBAs and IT Managers at 300 organisations surveyed by the Independent Oracle Users Group, “…20 percent said they expected their databases to be breached in the coming year”.

Fortunately, these and other innovations in 11g, if utilised will take the pain out for DBA's, Application and System administrators which in turn will make their life easier.

Marc Caltabiano
Director, Enterprise Architecture