Friday, January 02, 2009

Stories from the Field, Part 1

As I promised back in September, I would write about some experiences I've had with consulting.  I expect there to be many parts to this title, so I'm going to set some ground rules.
1. There will be no client names mentioned (or person names).
2. No part of my posts should be interpreted to insult a client or person, I am only reciting the event as it happened.
3. Names may be modified to protect the parties involved.

Now that the disclaimer is done (and I mean it, especially #2), let's start with our first story.

I was on a team working with a few other consultants on a large data warehouse project.  One of the guys (not brought in with us) was an independent consultant.   Let's call him Rob, and Rob is a good guy, very interesting and funny, and his stories caused great laughter at lunch.   Rob had a way of being very straightforward and profane, a quality that does not go well when working at client sites.  It was difficult for him to say a complete sentence that didn't include the F-bomb or using the G-D word, or any other curse word you can imagine.  This 'quality' frustrated a lot of the employees and gave me great pause into engaging him in conversation in front of other people. 

One day in a meeting Rob had a great idea.  In the client's ETL jobs, all jobs were designed to terminate at the first sign of an error, send a page to the duty pager, and have manual intervention to restart the process.  Without getting into whether this is good architecture or not, it is what it is, and Rob had an idea that we could create a routing using an exception handler in all ETL jobs instead of just terminating them.  He said it would be easier to code the jobs this way, and in a way he was correct.  Rob's problem was that he was solving an issue that wasn't a problem for the client - the client was happy with the way the jobs previously functioned, and Rob's hours and effort to retrofit working jobs with this new technique took valuable time away from solving problems that the client had identified.   However, the manager told Rob if he could get a sample done quickly, he would be open to changing the architecture.  

A couple of weeks later and Rob presented his working idea at a team meeting.  I had cautioned him in private about solving a problem that the client didn't seem to have, and my colleague suggested to him that this approach didn't take into consideration all the possible failure scenarios.  Rob pushed on and in the meeting, my colleague again suggested in front of everyone that he wasn't convinced this technique would work as described, and cautioned Rob to test it with every possible scenario before putting this into production.    The manager approved this code to be moved to production only after he had some tests for every scenario that had been described by my colleague.

By now you are probably starting to imagine the outcome of this scenario, and if your outcome includes the word disaster you are correct.  Rob moved this code into production and the first night there was a hiccup, but because he hadn't tested correctly for that failure scenario, incorrect data was loaded into the production tables and the job did not abort as it should have.  The next day people around the company were noticing data that was incorrect on reports and poor Rob had some explaining to do, which he did in a status overview laced with enough profanity to make a sailor blush.  

In the end, after much testing and some more issues, Rob was able to make this routine work correctly but the client largely considered it a failure and chose not to move forward with this approach, now leaving one subject area with this architecture while the remainder used the legacy code.  

For this story, let's analyze what happened here.  Using the flowchart methodology,
Rob identified a process improvement that the customer did not value and Rob did not adequately test this process improvement, causing the customer to suffer delays and bad publicity in their company, thus causing them to be shy on implementing further process improvements.

I'll make the next story an example of success, but the moral of the story here is that if you are going to go out on a limb to perform a process improvement, make sure it's something that the customer values and last but not least, make sure that it works.

No comments: