What keeps you and me awake at night? When I was a consultant for a Fortune 500 company in Southern California, we were getting beaten up to prove that databases were being backed up correctly and that disaster recovery would work. The truth of the matter was that we didn’t even have an accurate list of all the databases and databases would go down for days without anybody noticing which means that they were not even being monitored correctly. In fact, the yet unsolved NoCOUG murder mystery is based on my time there. Every time we tested disaster recovery there were frustrating glitches.
In my book on database administration, I related three horror stories from personal experience.
A deployment of a new version of a business application was unsuccessful, and the database administrator was asked to undo the changes made to the database by recovering the database from the previous night’s backups and the redo logs. He discovered that the backup script that was scheduled to run every night had been failing for three months; its defect was that it did not send an alert to the database administrator when it failed in its task. Luckily the redo logs were being copied to a file server and none of the archived redo logs had been deleted. The database administrator restored the database from the last good backup and started applying the redo logs to recover the missing transactions. Even the tiniest damage to any of the archived redo logs would have abruptly terminated the recovery operation but, luckily, none of the redo logs were damaged.
A new database administrator was creating a new database and was told by a system administrator that the system administration team handled all aspects of backup. The database administrator took him at his word but unfortunately, this particular server had been built using a new standard; the database administration team was responsible for backups, not the system administration team. This mission-critical database was used for a whole year before and then decommissioned. It had never been backed up.
A database administrator had disabled an automated backup script one night because it would conflict with a deployment of a new version of a business application. The deployment failed and the database in question was successfully restored to its previous state using the previous night’s backup and redo logs. However, the database administrator forgot to re-enable the automated backup script. One month later, there was another application deployment, which also failed. This time, the database could not be restored and the data had to be manually recreated.
Ask yourself this: How much confidence do you have in your backups? And when was the last time you tested your backups?
Don’t let me get started on the subject of disaster recovery.
What else keeps database administrators up at night? Why, the work of course. The darkest hours of the night is when all the hardest work happens—database provisioning, patching, upgrades, and cloning.
But there is a promised land where you don’t have to worry about backups, high availability, disaster recovery, provisioning, patching, upgrades, cloning, or monitoring.
The promised land is in the clouds: the Oracle Cloud, the Amazon cloud (AWS), the Microsoft cloud (Azure), and Google Public Cloud. At NoCOUG, we have seen the light and will showcase cloud technology for the foreseeable future. The theme of the upcoming NoCOUG conference sponsored by Google Cloud Platform on Thursday, November 17 at PayPal Town Hall in San Jose is “A Bright and Cloudy Future for All Your Databases.” The conference is free for NoCOUG members, PayPal employees, full-time students, guests of members, and first-time attendees. Click here to review the agenda and register.
P.S. If your future is not in the public cloud, then it will be in a private cloud. Check out Building Database Clouds in Oracle 12c if you are considering a private cloud.