As the editor of the NoCOUG Journal for ten years and counting, I’ve had the interview to interview some of the best minds in the Oracle world. These are a few of my favorite things they said:
[Tim Gorman] You are old, father Gorman (as the young man said) and your hair has become very white. You must have lots of stories. Tell us a story!
Well, in the first place, it is not my hair that is white. In point of fact, I’m as bald as a cue ball, and it is my skin that is pale from a youth misspent in data centers and fluorescent-lit office centers. It is a mistake to think of wisdom as something that simply accumulates over time. Wisdom accumulates due to one’s passages through the world, and no wisdom accumulates if one remains stationary. It has been said that experience is what one receives soon after they need it, and experience includes both success and failure. So wisdom accumulates with experience, but it accumulates fastest as a result of failure.
About four years ago, or 26 years into my IT career, I dropped an index on a 60 TB table with 24,000 hourly partitions; the index was over 15 TB in size. It was the main table in that production application, of course.
Over a quarter-century of industry experience as a developer, production support, systems administrator, and database administrator: if that’s not enough time to have important lessons pounded into one’s head, then how much time is needed?
My supervisor at the time was amazing. After the shock of watching it all happen and still not quite believing it had happened, I called him at about 9:00 p.m. local time and told him what occurred. I finished speaking and waited for the axe to fall—for the entirely justified anger to crash down on my head. He was silent for about 3 seconds, and then said calmly, “Well, I guess we need to fix it.”
And that was it.
No anger, no recriminations, no humiliating micro-management.
We launched straight into planning what needed to happen to fix it.
He got to work notifying the organization about what had happened, and I got started on the rebuild, which eventually took almost 2 weeks to complete.
It truly happens to all of us. And anyone who pretends otherwise simply hasn’t been doing anything important.
How did I come to drop this index? Well, I wasn’t trying to drop it; it resulted from an accident. I was processing an approved change during an approved production outage. I was trying to disable a unique constraint that was supported by the index. I wanted to do this so that a system-maintenance package I had written could perform partition exchange operations (which were blocked by an enabled constraint) on the table. When I tested the disabling of the constraint in the development environment, I used the command ALTER TABLE . . . DISABLE CONSTRAINT and it indeed disabled the unique constraint without affecting the unique index. Then I tested the same operation again in the QA/Test environment successfully. But when it came time to do so in production, it dropped the index as well.
I later learned that the unique constraint and the supporting unique index had been created out of line in the development and QA/test environments. That is, first the table was created, then the unique index was created, and finally the table was altered to create the unique constraint on the already-existing unique index.
But in production, the unique constraint and the supporting unique index had been created in-line. When the table was created, the CREATE TABLE statement had the unique constraint within, along with the USING INDEX clause to create the unique index.
So when I altered the table in production, disabling the constraint also caused the index to be dropped.
After the mishap, I found the additional syntax for KEEP INDEX, which could have been added to the end of the ALTER TABLE . . . DISABLE CONSTRAINT command because Oracle recognized the difference in default behaviors.
But that was a discovery I experienced after I needed it.
As to why my supervisor was so calm and matter-of-fact throughout this disaster, I was not surprised; he was always that way, it seemed. What I learned over beers long after this incident is that, in his early life, he learned the true meaning of the words “emergency” and “catastrophe.” He was born in Afghanistan, and he was a young child during the 1980s after the Soviets invaded. His family decided to take refuge in Pakistan, so they sought the help of professional smugglers, similar to what we call “coyotes” on the Mexican-American border. These smugglers moved through the mountains bordering Afghanistan and Pakistan at night on foot, using camels to carry baggage and the very old, the sick and injured, and the very young.
My supervisor was about 9 years old at the time, so the smugglers put him on a camel so he would not slow them down.
During the night, as they were crossing a ridge, they were spotted by the Soviets, who opened fire on them using machine guns with tracer bullets. Everyone in the caravan dove to the ground to take cover. Unfortunately, they all forgot about the 9-year-old boy on top of the 8-foot-high camel. My supervisor said he saw the bright tracer bullets arching up toward him from down below in the valley, passing over his head so close that he felt he could just reach up and grab them. He wanted to jump down, but he was so high off the ground he was terrified. Finally, someone realized that he was exposed and they pulled him down off the camel.
As he told this story, he laughed and commented that practically nothing he encountered in IT rose to the level of what he defined as an emergency. The worst that could happen was no more catastrophic than changing a tire on a car.
I’ve not yet been able to reach this level of serenity, but it is still something to which I aspire.
[full interview with Tim Gorman]
[Jeremiah Wilton] My daughter’s piano teacher likes to say that practice makes permanent, not perfect. Just because I’ve been a database administrator a long time doesn’t qualify me as a “senior” database administrator—or does it? Who is a “senior” database administrator? Do I need a college degree? Do I need to be a “syntax junkie?” Do I really need experience with Oracle Streams or ASM to claim the title?
To me, senior means that you have used a lot of Oracle’s features, solved a lot of problems, and experienced a variety of production situations. Do these qualities necessarily mean that I will want to hire you? No.
Of far greater importance than seniority is a DBA’s ability to solve problems in a deductive and logical manner, to synthesize creative solutions to problems, and to forge positive and constructive business relationships with colleagues and clients. For years at Amazon, we simply tried to hire extraordinarily smart people with a strong interest in working with Oracle and others. Some of Amazon’s most senior DBAs started with little or no Oracle experience. I believe that the focus on experience in specific technologies and seniority causes employers to pay more and get less than they could when filling DBA positions.
[full interview with Jeremiah Wilton]
[Kelly Pot’vin-Gorman] A young Oracle professional in India named Ranit Biswas asked Cary Millsap, “How did you learn so much about Oracle?” and Cary responded at length on his blog. What advice would you want to pass on to young Ranit?
Although it’s impossible to top Cary’s excellent answer, I can add the following:
[full interview with Kellyn Pot’vin-Gorman]
[Steven Feuerstein] I often wonder what advice I would repeat to my child if I had only a few minutes left on earth. We only have a few minutes left in this interview. What advice do you have for us—your fellow human beings?
As Spidey’s uncle said, with great power comes great responsibility. We in the U.S., each one of us, actually do have great power. But we have largely abdicated responsibility.
We should do everything we can to re-establish a real democracy, one that benefits the majority.
Finally and more generally, our focus and priority should be on our children, the future of our species. Make sure you spend lots of time with children, your own and others, enriching their lives, sharing your experience, giving them unconditional love and support. The world will be a far better place for it.
Thanks for providing me with this soapbox. I do so love to go on.
[full interview with Steven Feuerstein]