John Weathington’s Golden Gem of Compliance Development:
Expect that your requirements will change --  a lot.

I embraced the attitude of “agile” developers more than a few years back now, and it has saved me time and time again in my development efforts. This is actually a golden rule to carry with you on any IT project. I very rarely see a situation, where the business knows exactly what they want at the time of project initiation.  This is especially true on a compliance project, because usually the urgency forces you into action fast, then change as you go.

I’ve actually done a lot of thought leading ( there are others, I’m obviously not the only one ) on developing database solutions using the agile philosophy. My work is practical, not theoretical, and I’ve been able to get demonstrated success where there was no other way to get success.

This is not an article on agile practices, and I’m not going to try to build a case for agile here. This is however, a primer for database people that need to support an agile project. This article assumes that your project manager is actually pretty advanced in his thinking, and wants to take an agile approach to building a compliance data system.

Here’s a few things you need to know about, to prepare you for how this will execute:

• As I stated in my Golden Gem, the requirements will move extremely fast. Every two to four weeks ( the iteration length ), you can expect to have a new set of requirements. These requirements may be in line with what you were thinking, or they may come completely as a surprise, forcing you to re-architect your entire solution. You won’t know the requirements ahead of time. You will learn of them on day one of the iteration. You have to get over this, and just be open-minded. The techniques in this article will help you “cope.”
  
• At the end of every iteration, you will be required to deliver a complete, working solution. No prototypes, mock-ups, or “half-way” solutions. This means you will need to be amazingly focused, and every hour counts.
  
• You will need to reframe your thinking of “re-work”. On an agile project, there is no re-work. Everything is an exploration to a better, evolving solution. Don’t think of work that’s “wasted.” No work is wasted, everything builds on discovery.
  
• Plan on working face to face with people, with the primary mode of communication being in-person and verbal. You may even have a partner attached to your hip all day, if your project manager is employing “Pair Programming.” If you’re the kind of person that likes “alone time”, or works better by yourself, you will have a very difficult time on an agile project.
  
• I hinted to this in the first bullet, but above all, keep an open mind. It would be a very rare case if you completely agreed with everything that’s going on ( unless you have some experience with agile already ). It’s is vitally important that you go with the flow. Most agile projects fail because of attitudes, and nothing else.

Okay, there’s your primer. So once the decision has been made that your project is going agile, here are 4 specific things you need to do right away to survive this:

• Setup a version control system, and practice using it. If you have a version control system like CVS or better yet Subversion in house that’s great, however you may want a separate repository for your development effort. Expect that you will be versioning out every minute or so. That’s right, every minute or so. You need to get into the habit of just blasting out code and not worrying about whether or not it will mess things up. Think of your CVS system as your “super-undo” system. Just hack. If things go bad, you can always go back to a previous version.
  
• Setup a build script, that will build your database, and load it from scratch. All your build SQL should be text based, so this is a perfect fit for your version control system. Make the script universal, so you don’t have to keep tweaking it. It should kill your database, poll your directories, look for SQL, execute the DDL, and execute your transformation routines – all hands-off. This script will be run every night, so you need some sort of scheduler ( i.e. cron ) to kick it off. This is called the “nightly build.”
  
• Setup a test harness, that will test everything in an automated fashion. The best tool I know to leverage in this case is Quest Code Tester. Agilists are huge on testing. In fact it’s not uncommon to write your tests first, then your code ( this is called Test Driven Design ). You will need to execute all of your functional tests at the end of your nightly build, to make sure that the new system is working as planned. If not, you can always rollback – don’t forget the CVS system ( there is a method here ).
  
• Start focusing on a very robust metadata system, that will among other things, generate the SQL that your build script will execute. You should also have a script that can look into your metadata, and create automated documentation. With things moving so fast, there will be no time to spend on documenting, and re-documenting. The system should create its own documentation.

There’s more to consider but these are the keys. Agile projects can be very demanding, but extremely worthwhile. Surviving them as a database developer requires you to setup a version control system, test harness, automated build and test script, and extremely robust metadata. And above all, keep an open mind. Your success depends on it.