Databases, Structures, and the Data Itself

SQL Saturday - 3 Years Later

2011-10-03T08:00:00.000-04:00

Last week I attended the SQL Saturday event in Orlando, the site of the original event in November 2007. I'll not forget that one because Andy scheduled me to be the last speaker of the day and in the back of my truck was all my hunting gear ready for a departure to north Florida after the event for a good couple days of deer hunting. My two sessions that day were two of the best I've done, which makes up for the very poor showing on the next day's deer hunting.

The idea for SQL Saturday originally grew out of the code camps. Florida was one of the first states to implement code camps as early as 2005 statewide. The code camps were great and us SQL guys had to settle for having a SQL leaning track at the code camps. I thoroughly enjoyed spending 2006 and 2007 traveling around the state to speak at these events, and I occasionally still do. Brian Knight, Joe Healy, and myself had discussions about putting on a 'data camp' as early as summer 2006 and I even called around to a few hotels to inquire about a meeting room to run the event in October 2006. Well, alas, talk is all it was and in the meantime Andy W. and crew came up the wonderful idea of SQL Saturday and the rest is history. I quickly gathered a group together in Tampa by teaming up with the wonderful and community oriented Pam Shaw and we ran SQL Saturday #2 in Tampa in February 2008. It was a fun event, tiring, and a little stressful, but the first few events led us to where the event is today. Matter of fact, I just signed up to speak at SQL Saturday #100.

There has been some discussion from event organizers wondering if this "code camp" thing is just a fad, and I can thoroughly say that it's not. It's really taken on a life of it's own and now there are code camps, SQL Saturday, Bar camps, Azure Camp, and soon to be Windows Phone camps. As camps split into more specialized areas there are some attendance drops but it also brings out new people who are interested in that particular technology. Techies always enjoy the free or low cost events to network and learn or relearn something, and the post parties certainly don't hurt either.

If you haven't been to one of these events yet, I encourage you to find out when the next one is coming to a city near you, or even better, get a group together and host one yourself. Information on SQL Saturday can be obtained by clicking here.

Data Models and Their Importance

2011-09-30T15:00:00.000-04:00

Just this week I had a discussion with a client who is looking to hire a data modeler. We spoke about his project, the number of facts and dimensions, and what he was looking for in a data modeler. I felt compelled to add the following analogy:

"A data model is like the foundation of a house. Without a strong foundation, no amount of work or rework on the remainder will be correct, and it will frustrate you to failure, so get the foundation right the first time."

Unfortunately I know this all too well. I have been part of a few projects on the ETL side that had poor data models, either models that weren't completely thought through or models that weren't designed for the end solution. A data modeler needs to work very closely with the business/functional analysts to understand the data that will be stored in the model, and develop an enterprise model that can be easily maintained and expanded.

First, let's look at a typical data model. For this example, I'll use the star methodology. This is a simple fact table that contains surrogate keys that point to attributes in the dimensions. If you need to, review the difference between facts and dimensions here.

Our simple model is a grocery store. It has a date, customer, employee, and product dimension. It has a Sales fact table that lists each transaction.

This is a simple model but it carries a lot of information. The data modeler should be thinking ahead as to which dimensions should be added in the future for a sales fact table. There should also be thought as to which fact measures could be added in the future.

I've worked with clients who would design the exact model above and attempt to capture every possible fact and dimensional attribute the first release. DON'T try this! Just capture the majority of the dimensions which are needed to create a usable report, design the metrics (facts), and implement. Dimensions and facts can be easily added later if the tables are designed right, and this allows the IT department to deliver something quickly and get the business vested in the success of the project.

The synopsis, if you have made it this far, is to focus on building a solid data model from the start, containing relevant dimensions and needed facts to get the dimensional model up and running. Never, never, and I'll repeat never shortchange on your data model to get something out the door quicker.

Windows Phone 7.5 - A Year in the Making

2011-09-29T15:00:00.000-04:00

Readers will recall that last year I was excited about getting the Windows Phone 7.5. I even recall Nov 8, 2010 when the phone first went on sale - I awoke in a Marriott in downtown Durham, NC, and after a hectic day made a trip to the AT&T store, only to find that they were sold out of the Samsung Focus which I had decided on. Patience was painful but worthwhile, as a black Friday sale at the local Microsoft store yielded a free-after-contract LG Quantum. It's been a full 10 months since that fateful day and I'm no less pleased with Windows Phone.

Yesterday I was able to upgrade to Windows Phone 7.5. This is the first phone I've owned in almost 10 years that I am completely happy with. The hardware is great (LG), and the camera is excellent. Matter of fact, I don't carry a camera anymore as this one takes excellent pictures. The ability to sync with Zune keeps my podcasts up to date and the phone functions as a de-facto iPod. Except that its' better. I haven't really noticed much with Mango yet except that the colors are more vibrant and texts/emails have a different style to them.

For those who are contemplating a business device but want to stay in the Windows ecosystem, I highly encourage you to check out a WP7 device. I have been asked about it multiple times in my travels and try to spend a few seconds showing the device in a positive light. I am also asked quite a bit about my Amazon Kindle. Did I mention the WP7 apps, there are plenty of apps for many items but I still prefer to use a mobile browser when possible.

Suffice it to say that I'm incredibly excited about Windows Phone 8, Windows 8, and Windows Server 8. The combination O/S, phone, server, and tablet should be the perfect quad of devices. I'm not a Windows fanboy but it's hard to argue that any ecosystem is better in the dual, and often shared realms of business and pleasure.

I'd enjoy hearing any comments about the Windows Phone experiences you have.

Is it 2011 or 2006?

2011-09-28T15:41:00.003-04:00

It's been five years since I originally started this blog, so I decided it was time for a redesign. One of the plans is to move this over within the corporate webpage so the first task was to get the colors matched up. A second tasks was to "modify" the name. I elected to remove the quasi-curse word from the title, and while I believe "damn" is catchy and not overly offensive, the clientele is professional and it's important to retain the professional image.

One of my disappointments has been in the blog postings, mainly the quantity or lack thereof. It's just plain difficult to find the time to do a comprehensive writing, one that actually delivers value to the constituents. Well, here's my mea cupla. I'll try harder to write more often and also more quality postings. Business intelligence is such an interesting and dynamic field and there are many topics that I could write about which would deliver value to you, the readers.

If you have any comments about the new layout, please let me know.

Picking the right laptop

2011-06-28T13:40:00.000-04:00

Another topic back on the laptops, and I hope this will be my last for a while. Earlier this year I decided to pick up a MacBook Air to use when traveling. It's the lightest full laptop on the market and the size is a dream. The problems began quickly - first, I had to spend sometime installing BootCamp and setting up Windows 7 on it with all my software, as most macOS programs are useless for what I do. The problem I had with the laptop was the glossy screen - after just 30 minutes of use, I couldn't see well the rest of the day with severe eye strain. After a couple weeks I put it in the closet and a quick Bing search revealed that the exact same problems had been noted by others as well. I let a friend borrow it a few weeks who didn't have any issues, so just guessing my eyes are a little sensitive and the Apple store said it's perfect, so I resold it and decided to move back to my second choice, a Lenovo ThinkPad.

I finally decided on a ThinkPad T420s. It's a full machine with a 14" screen, which is only 1 lb more than the MBA, with all the features of a laptop (DVD rom, removable battery, anti-glare screen, great screen resolution). Crucial provided me an additional 4GB of ram, so now I'm running a 8 GB ram machine that's a dream to carry and can natively run all the programs that I need. The Lenovo cost the same amount as the MBA so that's a double win. So to anyone who is contemplating a thin, light 14" laptop, I highly recommend checking out the Lenovo lineup.

Types of Data

2011-06-27T13:24:00.003-04:00

This is a question I get asked quite a bit: "What are the different types of data that I can collect in my data warehouse?" Always an interesting topic, so I'll start by saying there are three basic types of data: empirical, anecdotal, and derived.

The basic case for data warehousing starts with empirical data. This is data that is collected - Eg. "I sold 10 widgets this week". Most data warehouses are built off this type of data, because it's really a "fact", meaning that it's true. Not to be confused with dimensional modeling, as an address is also "empirical" in nature.

A second type of data is derived. This is data that is created from another type of data. An example of derived data is "I sold 10 widgets this week for $1 each, therefore my total sales are $10 for the week". Derivation is the only way to perform a computation to get a picture. Think of aggregations as derived data.

The third type of data, and one that is less common, is anecdotal. This is data that is oberserved or believed but without any scientific basis. Anecdotal data often has applications in business. Think of the example that a salesman is selling widgets to a retailer, which we shall call Mega-lo-mart, and the salesman knows through discussion with the Mega-lo-mart manager that they don't intend to buy widgets this year, anecdotal data would be the salesman's oberservation that "mega-lo-mart doesn't indend to buy widgets this year because they aren't selling well". There is no scientific evidence this is true, but think of the business case, where a salesman is wasting time trying to sell to someone who will not buy the widget. Thus, there is a case that anecdotal evidence could be used in a data warehouse application, as long as it's documented as such, to help drive decisions.

I find these data types fascinating, especially the anecdotal nature. Sometimes it's difficult to determine which data type is particular type of data is, based on the way it was collected. That's our jobs as architects (typically a data modeler) who would analyze the data, work with business users to determine the applicability of the data, and build a dimensional model that contains all three types of data to present as a business intelligence applicaiton.

Using Windows Tools to Maximize Productivity

2011-01-09T20:30:00.002-05:00

One of the most important things for consultants to do is to maximize their time. Fortunately, there are a number of tools available to assist in this endeavor. I'm specifically focusing on Microsoft tools, namely Windows Phone and associated services to get more done in less time.

Smart phones have really taken off. Of course, some of us have been using "smart" phones long before Apple came along and took credit for inventing it.

Here's a list of three Windows tools that can be used to maximize productivity:

Windows Phone - Syncs with Hotmail, Hotmail calendar, and can use Outlook connector to sync with Outlook on a desktop. Use Zune software to sync up podcasts and music.

Windows SkyDrive - Upload your documents and photos to this secure online storage area, where they can be accessed from your WP or any internet-connected computer. This is a lifesaver when needing to access business documents on a client site or to edit using the WP.

Office Online - Last but not least, the ability to create, edit, and save Office documents on computers without the client software is huge. Office online doesn't have the full functionality of the client tools but it's enough for basic documents and has saved me more than once.

I hope you consider using these tools to become more productive. They are all available for low or no cost and provide high return on time.

The Importance of Goals

2011-01-03T14:45:00.002-05:00

Welcome to the first post of 2011. Ironically, or maybe not, I'm writing about goals today for software techies, and my first goal is to publish posts more frequently.

It is very easy to get into patters of life that just happen and people, like energy, tend to follow the past of least resistance. This is especially apparent in corporate America, where people get hired as employees, suffer through yearly reviews, get raises every now and then, but (often) fail to build a comprehensive goal list. In corporate speak, this is called a "career path". I prefer the term "Career plan", because the employee is the responsible party for developing and executing his/her own career plan. Especially in the era of corporate downsizing, one should not and cannot let their career path be dictated by a single company.

A career plan can be simple or detailed. It must be actionable. One plan might state "Advance to become Software Architect". This seems slightly lofty and possibly unreasonable, but with the right planning becomes a distinct possibility. Compare with the goal of "Advance to become Software Architect by presenting at a major conference, spending 5 hours a week networking with fellow architects, and publishing an article in a journal during the current year". The best case in this scenario is that you now have a plan and become a software architect and the worst case is that if you achieve all the goals you've set yourself up in a strong position to move into the role down the road.

I would imagine that if you are reading this blog you've probably been doing some networking or attended a tech event. These two facts alone will put you in the "creme de la creme" as they both show initiative in expanding your boundaries. But never rest on your laurels, continue moving forward and setting new goals as you retire achieved ones.

The best goals are SMART:

S - Specific (Get a MBA, Publish an article)

M - Measurable (Degree is conferred, article was published)

A - Actionable (Something that can actually be done)

R - Realistic (Something that you really can achieve in the time allotted)

T - Time driven (There is a time limitation in which to achieve the goal)

The fact is that the software development lifecycle is alive and well, and this relates to techies as well. It's a "grow, learn, and achieve" industry versus a "stagnate and die" one. In which group do you consider yourself? What is your SMART career plan for this year?

Windows Phone 7 has landed

2010-12-01T00:52:00.003-05:00

Hello all, in my last post I spoke about wanting to get a Windows phone badly after suffering a year of Android. Suffering is a word I use lightly, it was never "that" bad, but after a year of app-centric software, I was really missing my WinMo phone. I even went so far as to get out my old HTC Touch Pro and carry it around a week to see if I could be comfortable with it, and sadly, while I loved the software, I did not like the phone itself so I chose to ride out the wait. Well, the wait is over.

Last week I picked up the LG Quantum. It's the only Windows Phone 7 device with a slider keyboard, reminicent of my PPC-6700. Matter of fact, it reminds me a lot of that good ol 6700 in form and function, and that's a good thing. So while I'll do a full review, let's first start by looking at my requirements for a Windows phone:
- Ability to sync with Outlook/Hotmail
- Ability to sync Calendar/Tasks
- Windows Media Player
- Decent internet browser

The WP7 platform does all of these except one. Tasks. Tasks are sorely missing from the O/S but I have faith that Microsoft is planning to add in this feature at some point. I've also learned to use the Zune software to sync the phone, and while I've never had a iPhone, it does seem like a knockoff in functionality ported from Apple.

The user interface is great and a welcome relief from the app buttons of Android. I find the entire phone and features to be a breath of fresh air that will only get better with time.

Some features I hope that MSFT includes in WP7:
- Tethering via bluetooth
- Tasks sync
- Additional SD Micro card storage
- Bing Maps with Navigation

Windows Phone is coming, and I can't wait

2010-10-14T19:55:00.002-04:00

In December of last year I wrote about my purchase of a Google Android device (HTC Hero) and how I felt Windows 6.5 was outdated and that Microsoft had missed the boat on the mobile software. Here's my partial mea-culpa: Microsoft did miss the boat for a while at least but it seems to be a good thing, and here's why. (So I was wrong - Windows Mobile 6.5 is still very relevant)

In the past I have owned three devices (all three made by HTC coincidentally) that ran Windows Mobile 5.0, 6.1, and 6.5, respectively. The common problem with these devices was lack of memory and seeming hardware issues. The first phone (PPC-6700) was my favorite of them, as it worked great until I dropped it (my bad) and then I started going through refurb hell with Sprint and upgraded. The second phone (Mogul) was better in all respects but a little dated in screen resolution, and the screen promptly developed large white spots that a quick Bing search revealed to be a common issue with overheating (phone design error) so the screen was unusable. I then was moved to a Touch Pro which worked fine but had a horrible form factor (thick on the depth and thin on the width) which made the screen very small and barely usable (form factor design flaw) along with horrible battery life (about 30-45 minutes of use per charge). After the issues with that phone I wrote off HTC devices and headed to the Sprint store, where a smooth talking sales guy suggested the HTC Hero with Android 1.6 (Cupcake). I figured I'd give it a shot based on the price ($69 w/contract) and was initially impressed. My honeymoon wore off quickly.

Android is an interesting beast and I've since used the factory upgrade to Android 2.1 and I'm no less pleased with it. Windows Mobile 5 was a better O/S with more features that I found lacking in Android. Android, following Apple's lead, is a siloed O/S where everything is driven by apps. This means information doesn't flow well between applications, and the base O/S has hardly no functionality at all. So for everything I want to do, I have to go online, find an app that does it (or not), and install it and hope that it works. Not my idea of good usability.

The biggest flaw is that there is no easy way to sync up outlook (that I've found) and keep contacts, mail, and tasks synced up. There is a download provided by google that runs on my PC to do this, but it's rickety and while it works, I still don't find the calendar app to be anything like what I had on Windows mobile.

I've been watching the news as Windows Phone 7 gets near and now I am very excited about a phone that has full Outlook sync, Windows Live, Bing maps, a good native browser, and the potential for apps if necessary. The only downside is that the phones are initially only available on GSM (not bad in the long run) but the models available by AT&T and T-Mobile haven't caught my eye like some I have seen overseas. Looks like I'll start watching ebay next week (Oct 20 release in Europe) to pick up one of the new phones unlocked. I'd even take Windows Mobile 6.5 on a good device over my current setup but looks like Microsoft might be showing up at the right time to make a decent dent in the smartphone market.

The Damn Data is Going Away....but not yet

2010-03-12T22:50:00.002-05:00

What a fun four years it has been. Last year I started a new push for the website (www.thedamndata.com) and installed forums and whitepapers but it became a full time job to keep the spam, porn, and Rx peddlers off the forums and I ended up shutting them down for a while. The best mistake I made was trying to check the website from my computer while connected into the client's network. It was an honest mistake but a valuable one. The dreaded words "Blocked" came from the proxy server. The browser had a message that it was inappropriate material for a work machine. I had been wondering why it was so difficult to get some more traffic to the site and it suddenly dawned on me that while the name is a little catchy, it's probably not appropriate from a corporate standpoint. I have a new name, a new website, and while this blog address will stay the same for now, you'll soon see the results of a rebranding for the blog and web portion.

Is Windows Mobile Relevant?

2009-12-07T11:38:00.003-05:00

There is an interesting read on BetaNews about the Windows Mobile platform. Before I get flamed for being a Apple junkie, I'll say right now I do not have nor want an iPhone. The closed platform and interface do nothing for what I need.

My current cellphone has been trending downhill for the past few months. It's an HTC Touch Pro device with WM 6.1, and I hacked it to run WM6.5 that I downloaded at xda-developers. Those great hobbyists do a wonderful job at packaging roms which unlock the full value of phones. I've tried a few different roms but the "Energy rom" line was the most stable and best one I've seen.

Now I'm sure you're thinking "the phone worked fine until you hacked it". This is the opposite, the phone out of the box from Sprint was buggy, laggy, and was a fail at basic telephony. Not to mention the habit of draining the battery before lunchtime. By researching and installing a custom rom I was able to get another 6 months of use from the device, else I'm certain it would have ended smashed into a million pieces on a sidewalk somewhere.

This leads us to today - WM is coming out with WM 7 next year. This is supposed to be an awesome system. WM 6.5 excels where Apple fails. It's open development, lots of good resources but on the flip side a lot of the usage is wonky and seems 2003-ish. I cannot wait for a better system in a year from now when I need a smartphone today.

I purchased a HTC Hero last week running Android. Android is made by Google and is quickly gaining market share among smart devices. It didn't have a dedicated keyboard so I thought I would be bothered by the virtual keyboard, but so far it's not too bad. The phone works well and it seems that the flow and use of the software makes sense. It's much more user intuitive than the Microsoft platform as it exists today, but I'll wait for the newness to wear off before I comment on the overall package. If Microsoft is destined to have a presence in the mobile market, WM 7 must be a compelling "wow".

14 Years and 1.2 Million Miles Later

2009-12-06T09:24:00.002-05:00

Last night I had the great pleasure of flying on Southwest airlines into Kansas City. I haven't flown on Southwest much this year but when I had this opportunity I jumped on it. Boarding the flight was painless and I picked an aisle seat right over the wing to try and have a similar experience as before. We arrived at KC Airport at gate 31, and when I departed into the terminal, I had a strong feeling of deja-vu.

It was June 1995. I want to say it was the 8th day of the month, but I'm not sure. A family member had purchased me a ticket to fly from KC-Chicago on Southwest. I had never flown on a commercial aircraft before and was a little nervous. I won't state my age or give away too much information, but I'll say I was "young". I left my car at my uncle's home and they took me to the airport, I checked in my bag, and waited for the flight. At that time Southwest used those old plastic number cards, and I believe I had a 'B' card. The plane was this huge light brown thing, later I learned it was a 737-200. We boarded the flight through gate 31 and I found a seat right behind the wing on the left side of the aircraft. I recall that flight had some seats that faced to the back, and there was a very attractive blonde girl in one of those seats who ordered a beer on the flight. I'm sure I had a Coke. My first feeling was the massive acceleration on take off, and the rest of the flight was an awesome experience for a young boy.

One of the neat things about consulting and building a business is the opportunity to travel to distant locales. Some people enjoy traveling, some see it as part of the job, and some hate it. I understand all these perspectives. Last night as I recollected on my experience, I thought of the many airports I've visited since that day in June 1995. I thought of the planes I've flown on that would make a 737 look like a tiny toy. I thought about the many people I've met along the journey, both attractive ladies and nice guys. I then thought about what I know now vs. what I knew in 1995. It helped me to remember that the world is full of opportunities, that we all have a chance to make what we want, to live our dreams. I am living mine, talking, practicing, and helping companies become more efficient through the use of information technology and business intelligence. In June 1995 I was working at Walmart, thinking of how we (myself and the four other guys I worked with) could get bleach loaded onto shelves faster, work the stock faster, or sell more items through use of displays. Now I am thinking of how I can use IT to help a client complete financial audits. Wow, what a 14 years it's been. I'm certain that I will look back in another 14 years and saw that in 2009 I had no idea how great it would all become.

Update on the Data Modeling Tool

2009-10-30T07:43:00.000-04:00

One of our clients was testing Toad Data Modeler to see if it would suffice as a replacement for CA ERWin to construct a dimensional model - the verdict is in, and it's "No." Toad Data Modeler does regular ER diagrams not not a dimensional model. Looks like CA ERWin is going to be the tool of choice for this client as well as many of our other clients.

Speaking of Toad for SQL Server, I'm not convinced there is a need for the product as SSMS is such a great tool. Microsoft really is sitting in a sweet spot right now by providing SSMS with the SQL Server license. Toad for Oracle is a necessity with the Oracle products.

Designing Quality ETL Processes

2009-10-28T09:00:00.001-04:00

ETL is the heart and soul of the data warehouse, so why does ETL job design and performance always seem to get little attention when schedules are tight? If it runs, it's fine, right? No! Follow the 'Durable Impact Top 5 Rules of ETL Design' below to develop quality ETL jobs and minimize your headaches with SQL Server Integration Services:

1. Always build jobs as modularized, stand alone components.

2. Segregate jobs based on the importance of the data. Most needed data gets loaded first, etc.

3. Build file extracts within a job and make every job restartable without having to restart the ETL process from the beginning. See #1 above.

4. Parallelize only when necessary. With SSIS 2008, don't use manual parallelization, the backend engine is much improved removing the headaches of slow performance on large data sets.

5. Use the tool best fit for the job. Example: When reading from databases, consider using the SQL ORDER BY command to presort data that must be sorted.

By following the 'Durable Impact Top 5 Rules for ETL Design' you can improve your performance and cut down on your support time.

It's a Great time to be in BI

2009-10-26T19:28:00.002-04:00

Yes it is, a great time for BI. Why is this? BI saves customers money, customers spend a little money in a BI application or dashboard but save a lot more with intelligent data and all the good things that come with it.

We just brought on a great person to fill the Marketing Services Consultant position at Durable Impact. Stay tuned for a blog introduction of this person and keep your eyes posted, we're soon going to be looking for at least five more SQL/BI Developers. Contact us on our webpage if you want to be notified when the positions open. It's always helpful to include a short paragraph or two about yourself and what interests you, so we have a good relationship from the beginning.

Website Moderators

2009-10-17T22:20:00.002-04:00

Thanks to everyone who attended my presentation today at SQL Saturday. It was a great crowd and I enjoyed it immensely.

I am looking for one or possibly two website administrators. The position entails spending 1-2 hours per week working thedamndata forums. Your responsibilities will be to approve new users and posts, create postings, and remove inappropriate content. Again, this should only take one or two hours per week. The DI staff is doing it right now but it's keeping us busy dealing with spam bots, porn operators, and illegal prescription peddlers who continually wreck havoc on the website.

This is an unpaid position, but as compensation for your time, you will receive a one-time $100 gift card to the retailer of your choice. We may provide other perks from time to time, but the primary reason we want someone is for their passion to speak and write about data and spend a few minutes assisting the website as required.

Please email me using the contact us form at www.durableimpact.com if you are interested.

Data Modeling Tools

2009-10-16T18:17:00.002-04:00

Most software apps have a very strong following and drive strong opinions from the users, taking for instance Windows vs. Mac. Those who use Windows often don't understand Mac, and those who use Mac despise Windows. Data modeling is one tool where I haven't found this type of approach, almost everyone I've talked to either uses ERWin or has heard good things about it.

I was working with a client this week who needs to get a data modeling tool. I've recently been working with some clients who have their own version of data modeling using Microsoft Visio and powerpoint slides, but this really isn't a good substitute for a true dimensional modeling tool. I've used ERWin in the past at a few clients and it's a great tool, allows one to reverse engineer the database, or better, to create a new model, creating the DDL for the model, and then versioning it for source control. The one problem with ERWin is that is isn't cheap, thus many look to Visio. Visio is a good tool in it's own right, but it's not a true data model tool yet. I suspect this is the direction that Microsoft is trying to go in and maybe with Visio 2010 it will meet that goal.

One tool I was made aware of is Toad Data Modeler. Toad products are made by Quest Software, and I've been working with Toad for Oracle since the beginning. I'm very pleased with the Toad database products, and have already recommended to this client that they buy Toad for DB2, which they have done. I downloaded a version of Toad Data Modeler and my first take of the tool is that it's going to work great for what we need to do, and it's at a very attractive price point as well.

I always stress to clients that they have a printed data model, typically where the developers can see it and visualize the relations between the tables in the model.

Sales and Making Sales

2009-10-06T07:30:00.001-04:00

My friend Andy had a blog posting last week about sales guys in the software field. Selling is always an interesting part of running a business and most likely the part that causes the most headaches, but without sales there is nothing else.

I'm typically distrustful of most commission-based sales where I'm the consumer. This is true for auto sales or buying suits. I bought a suit last week at a major mall-based retailer and I asked the guy (after purchasing, bad on my part) if it was quality. He went to say something and then stopped, smiling and saying "everything we sell is quality". It's blanket statements like that that make me second-guess my purchase.

I've stopped attending many industry events and meetings because they have turned into sales affairs. I attended one industry event where, of the approximately 60 attendees, at least 30 were sales guys hunting for leads and 20 were unemployed guys looking for work. The other 10 were people like me just trying to learn more about the product and maybe meeting a person or two in the process. I also wrote back in April about getting kicked out of an Oracle event that was a "briefing" but truly was a sales pitch. I've found a much different experience at true community events (like SQL Saturday), and while there is always the element of selling involved, it's not excessive. I have received a few unwanted emails from being on mailing lists coming out of these events, but that's more the exception than the norm. I hope that as SQL Saturday events grow they continue to stay true to their roots.

This brings me to a conversation I had with a client recently. Selling consulting services is kind of like dating, sometimes it's smooth and sometimes it's not, sometimes there is a return phone call and sometime there is not. One of my major disappointments when first getting into this business is the amount of sales leads that seem very hot and then turn cold. It does come with the territory and one has to learn not to take it personally. Anyways, the client I was talking to described to me his process of choosing a company to perform the work. He spoke about other firms which had presented and how things had went with them. What it came down to, in his mind, was trust. I'm pretty plain-spoken which has it's strengths and weaknesses, but it worked in this case that this client believed what we (my partners and I) said, and our straight talk gave them a great sense of comfort. I can relate to that, as I stated earlier, I'm pretty reluctant when listening to sales pitches and always trying to understand all the angles of the discussion. It was pleasing to hear that at the end of the day, the time-honored traditions of honesty and integrity were able to overcome bigger names and slicker salesmen.

A Bumpy Flight

2009-10-03T21:54:00.004-04:00

What was your first thought when you read the title "A bumpy flight"? Were you confused, trying to figure out if I was talking about the economy? Or did you think I was talking (as I do) in analogies?

I was talking about both, and a true bumpy flight.

In reality, I just returned from Dallas on this flight. The 2 hour flight was quite bumpy the entire way, so much that the flight attendants were told to sit down multiple times and barely completed the meal services before landing. I find myself to be a fickle flier and the slightest bumps make me a little nervous, not really because of safety concerns, but I always think the next round of bumps will lead to one of those massive bumps we see on CNN causing multiple injuries. Thankfully we didn't twist a tail and I'm here to tell you about the time this past week that I used the "bumpy flight" analogy.

We just started working with a large client on a data warehouse project. I will talk more about that later, but suffice it to say most would have heard of the name. A partner and I spent the past week working with the client on data discovery and trying to get them set up for the project. A critical part of the project is setting appropriate expectations. This company has never had a data warehouse before, so it's a "greenfield" development project. This is the best kind, from my opinion, because we get to do things right the first time. A majority of our projects to date have involved enhancing or repairing someone else's work so this is a welcome change as a different kind of challenge. Part of my duties this week were to work with senior management (CFO and Director level) to get an idea of what they wanted from the project and try to answer any questions they had about the process. I really enjoy working with management and listening to their concerns, that is true consulting at it's best.

I had a great meeting with the chief product engineer. Part of answering the business questions requires a deep understanding of the business. This person had a lot of great comments and thoughts and our scheduled thirty minute meeting quickly turned into a great two hour discussion. Being a good engineer, he was very interested in risk factors for the project and wanted to know what risk mitigation plans I had prepared. It's true that starting any project risks missing a key requirement and the best way to mitigate that risk is to keep the business users as closely involved with the process as possible. He was satisfied with that answer and as a follow up comment I used an analogy that brings us to the title of this posting. I explained to him that a new data warehouse project will have it's growing pains and to expect them, I told them that "it's similar to a flight from Dallas to Tampa where it's bumpy the whole time. There will be times that we all wonder why the air isn't smooth and there will be bumps along the way but the safety of the aircraft is never in question and we have complete confidence we will arrive safely at our destination."

I love analogies, especially when they come true.

SQL Server DBA / Reporting Services

2009-06-09T16:36:00.002-04:00

Hello all, I'm going to keep this short today. DI is needing to grow our stable of consultants, and specifically I'm looking for a couple positions with the following:

- SQL Server DBA experience

- Experience with 2005 and/or 2008

- Prefer 2005 to 2008 upgrade experience

- SQL Server Reporting Services experience

- Reports in 2005 and/or 2008

- Using analysis services for cubes

We will sponsor your MCDBA or other Microsoft certifications. The first test is to see if you can figure out the email address below!

Please email me at: resumes at durable impact dot com so we can start a discussion.

Employee vs. Consultant, Topic: Insurance

2009-06-01T05:47:00.001-04:00

Today we'll talk about the differences in insurance between working as an employee and working for yourself. Yesterday I explained the difference between contracting and consulting, but both types are considered (by the IRS) to be self-employeed unless they receive a W2, and then you're an employee

Companies began providing insurance as a benefit to attract workers after the shortages in WW2. This benefit has grown into the extreme mess we call "health care" today in the United States. The problem here is that we spend the most on heathcare of any country yet do not receive the best care. Bear in mind, there will be a lot of economics discussion today, but it's necessary to understand the point.

Employers, being the de-facto providers of health care, have a few advantages going for them. The first is adverse selection. Every company will have some people who require recurring care (cancer, diabetes, etc) and some people who require no care, so the insurance pool overall will skew towards the median care requirement being right in the middle. Thus, the portion of insurance for each person will be say, $200, of which the employer picks up $150 and the employee pays $50. (Assuming a single person here). The employee portion of $150 is considered a benefit (and a tax writeoff) to companies.

Let's take the same person under a self-employed scenario. Now, this person knows their own health. We call this moral hazard. They know their favorite foods are cheese fries and buffalo wings. They also know their family history of heart disease. They do not reveal this to the insurance company though when this person purchases insurance for $200 a month, of which the insured is responsible for the entire amount.

There is another self-employeed person who is very fit, has no history of family disease, eats healthy, but balks at paying $200 for heath insurance the person does not think he needs. This is where adverse selection comes into play, as only the sick people or people who believe they will need care buy insurance, and the healthy do not. That is why private cost plans are cost prohibitive. There are some ways around this, as credit unions and other firms try to "pool" people together to buy insurance at discounted rates, to "pull" both of the people (heathly and unhealthy) to buy insurance and average out the risk again. However, this still presents major problems for insurers, who are in the business to make a profit.

Now we enter into the political debate as to whether to force people to buy insurance on the private market, providing the pooling mechanism, or to have the insurance provided by the federal government which again provides pooling. I'm not going to get into this futher except to state that either of those options is preferable to me versus the current option of employers providing insurance.

For those who accept the risks of self-employment and need insurance, there are multiple things you can do. First is to buy a full plan on the open market, which is very cost prohibitive. There is something called a HSA (Health savings account) which when linked with a high-deductible plan, allows people to retain insurance for catastrophic occurences but also has high deductibles to discourage the overuse of the insurance. (If you're interested, this is what I do). If you're married to a partner who has corporate insurance, maybe the best option is to use that person's plan.

If you're going to consider the cost of health insurance in your criteria, factor in the full cost. Employee contibution is $50 every two weeks, and private insurance is $100 a month. Therefore, the premium expense is equal. There are other costs to be factored in here, such as office visit deductibles, but remember that cash is king, and while on corporate insurance I had a prescription once that was a $37 co-pay, but only $21 cash price.

If you're thinking of becoming self-employeed understand that there are options available to you, and with the looming discussion of health-care reform, the options may be more attractive and remove it as a key decision making point in the future.

Consultant vs. Contractor

2009-05-31T05:33:00.001-04:00

Before we go any further in the "Employee vs. Consultant" discussion, I want to clarify a few things. There is a difference between consulting and contracting. Many times the words are used interchangably but this is not correct. I'm certain others will have a differing point of view, but here's mine.

Contracting: hired to do specific tasks, not always a specific range of subject matter expertise needed, for a given duration, very little overview

Consulting: hired specifically for expertise, asked to advise on processes and output, may do some tasks, but work closely with management and employees to instruct and guide

When I first started out I was contracting, but then moved to consulting. Consulting is where you can really make a difference, contracting is more like being a temp employee. Just today I was participating in a conference call with a client, discussing the way a dimensional model is built. One of the attendees asked me why I put both the surrogate key and business key in the fact table, and I replied that I did it in case the surrogate keys were corruputed, to aid in rebuilding them and also for research. Another voice popped up and said that was a good standard for moving forward, and now it looks like that client will be using that process in the future.

Contracting is different though, it's like signing a contract to run a couple thousand test plans against a new piece of software. That requires experience to execute the plans and report results. The difference is that during the requirements gathering and design review, a consultant who had worked with other systems in the industry was brought in to review and provide thoughts for product enhancement. It should be clear that there is a difference in executing a script and advising business process owners and management on features.

There are key differences between contracting and consulting, and it's good to keep them in mind when reading my posts.

Employee vs. Consultant, Topic: Stability

2009-05-30T05:04:00.001-04:00

It's been about a year ago when I wrote my first post about the differences and trade-offs between working for someone and working for yourself. My conclusion then was that you should add up the costs and benefits of everything and then make the best decision. The problem is that the true costs and benefits are obscured and it's hard to make an informed decision. I decided to expand on the topic and have a couple different posts exploring different dimensions of making the best decision for you.

In December 2008 I received a phone call from a friend, the type of call that you never want to receive. He informed me that he had been let go from his corporate job. He's not too far from retirement and almost had his house paid off and was well thought of at his job, but the management had to make a decision that I'm sure was not easy. He was shell-shocked, but quickly recovered and found something with little overall loss in pay and benefits. His statement to me "I thought it would be my last job" was telling. In his mind, he was relatively safe, having traded some dollars in pay (though not much, he was doing quite well) for perceived stability. All of my clients have let people go since last year. I wonder how many of them believed they were safe too. More to the point, I wonder how many believed they were safe and traded a decent amount of salary for that security which they did not receive?

What would you do if you have the choice of two jobs that
- Company A: $78,000 a year, history of mass layoffs and hire/fire shop
- Company B: $70,000 a year, history of never having a layoff
Consider all other benefits as being equal. What would your decision be?

This is a true life scenario of another acquantance of mine. She picked company B, and two years later the company, well known for it's stability (both good and bad) was sold and everyone was laid off. In that two years, she traded $16,000 of salary for perceived stability. After taxes, you have a new Ford Focus. To be fair, there is no guarantee that she would have made two years at company A, but the economy was growing and during that time A did not have any known layoffs.

The point of my exercise, in case you missed it, was to try and remove stability from the equation as much as possible, as stability is a factor that is so far outside of an employees' control. Look at local Tampa company WellCare, a stable company that just did a massive layoff due to some fraud charges. How many DBA or business analysts had any control over that?

Consulting hasn't been too much better over the past year. Many projects have been cut, cutback, delayed, etc and some consultants and contractors have been enjoying time at home. A consultant I know who lost a project was quite concerned and told me he only could survive 2 weeks without a steady gig, if that. Without knowing the details or being judgemental, that's not a good position to be in any way, but in that case being an employee might be a better route as generally layoffs include some kind of package or unemployement, where self-employed does not. (In case you're curious, he went exactly two weeks between projects).

In conclusion, I recommend completely ignoring any discussion or talk about job stability and focusing on other aspects before you make your decision.

Coming up next: Health care, the most dreaded expense.

Managing Servers

2009-05-29T08:00:00.002-04:00

Today I want to write about my server management policy and philosophy. This is one cost area that doesn't offer a competitive advantage for us, so it's best to go with the cheapest options and minimize expenses.

My company's policy on server management is that anything production-capable will be managed by someone else. That is, we host everything with a professional provider. We're in the business of consulting and data management, not hardware providers. I don't feel like we can adequately procure and maintain the hardware as well as a company that is dedicated to this task. The inspiration for this decision was David Ricardo, a famous British economist who is most commonly known for the theory of comparative advantage. That theory goes like this: if China is best at making clothes and the United States is best at making computers, then China should make clothes and trade them to the United States for computers. That's a very simple way to describe it, but we all know what we're best at and I want to focus the business on consulting and not hardware management, so we outsource that area.

Hosted servers are expensive on a month-to-month basis but much cheaper in the long run versus buying hardware, paying for a COLO setup, and buying software licenses. I'm certain that the costs will continue to go down with the introduction of Hyper-V on Windows Server 2008. Durable Impact works with both Unix and Windows-based environments, but our hosted servers are running Windows Server 2003. At some point we will move to Windows Server 2008, but only once our hosts do.

A Trip down Memory Lane

2009-05-28T04:27:00.001-04:00

While working at a client site the other day, one of the people I was talking with had a memo from 10/13/1988 hanging up at his desk. This memo concerned how to use the office lighting system (which must not have changed since then), and was yellowed with age, but still readable. I joked with him about the memo and we got into a discussion of the days when "there was one modem for every 10 people". That reminded me of my first PC.....

....to tell the entire story, my parents purchased a Tandy 1000HX system in November 1987. I don't remember much about that computer, but we only had it for a while before my dad traded it in for a Tandy 1000SX. We bought it at a Radio Shack store in their "Tandy Computer Center". It had a 40 MB hard drive, 512 KB of Ram, and 5.25" and 3.5" floppy drives. It ran MS-DOS and my brother and I had a bad habit of just turning it off, which caused the hard drive to crash, thus we got a hard scolding to type C:\PARK to park the heads before shutting it down. We used this computer for years for games and general Word Perfect 5.1 applications and with work and high school I didn't use it for a long time until I got my own computer.

It was a Packard Bell desktop purchased from WalMart for $1399 in 1996. It had a 15" screen and a blazing-fast Pentium 75 with a whole 8 MB of ram and a 640 MB of hard drive. I saved up my own money and purchased a Canon BBJ-4100, one of the first color bubblejet printers for a cool sum of $318 after taxes. I still have both of the receipts, but I recall that printer cost an entire week of work. When I arrived at college shortly thereafter, I discovered that all my dorm mates loved the printer and cartridges rarely lasted a month, and no one could afford to chip in to replace it. I also realized that Packard Bell wasn't so hot, and by summer of 1998 I sold it for $350.

My next machine was a step back, but really a step forward. It was a used 486DX/66 that my dad bought at a garage sale. I loaded it with a cool 20 MB of ram, a 4.3 GB hard drive (which seemed entirely huge at the time), and a decent video card. The only thing it didn't do well was play games but it was remakably snappy for being a 486. Being at the university I had accesss to 10 Base T ethernet and enjoyed the web in high speed.

In spring 1999 I built a new machine using my mothers' credit card (thanks, Mom although I know you weren't happy at the time). It was a Celeron 300A with 128 MB Ram and an actual legal copy of Windows 98. If you remember, the 300A was the unlocked processor that was highly sought after for builders, and I picked one up cheap. I was able to get it to run at 550 Mhz at one point, but I generally kept it at 450 Mhz using my Abit BH7 motherboard. Throw in a 10 GB hard drive and a 17" flat CRT monitor (pretty good size for that time), and I was computing in style.

That computer lasted until 2002 when, after multiple hard drive crashes (now I know why OEM drives were so cheap online), I purchased a Dell Dimension 8200 desktop with Windows XP home edition. It was a P4 2.53 with 256 MB of RAMBUS ram. The computer was a great deal ($750 incl 17" CRT) but only because Dell was trying to liquidate their RAMBUS ram because it was so expensive and DDR had just come out as a competing technology. I paid $200 just to put 512 MB of ram in the computer, but by the time I sold it in 2007 for $350 I had added a 19" LCD (again, when I bought it that was HUGE and rare), a super nice 4xAGP 256 MB Radeon video card, and a 160 GB hard drive. That was the last (and best) desktop I owned and now I'm only using laptops.

In 2003 I picked up a Dell Inspiron 1100 laptop, Celeron 2.0Ghz, 512 MB ram, 40 GB HD, Intel 945 graphics card. It was cheap ($800) but had a 15" screen (1024x768). I could never get used to the screen resolution as everything was too big, and the computer was a beast, but worked well and survived two years of being totally abused in my backpack being trucked to work and back. Somehow I was able to avoid the problems that plauged that lineup (especially the 5100 and 5500) models that used full Pentium chips, somehow the CPU cooling ducts were on the bottom and would suck in dust and clog up baking the CPU. I never had any issues, and the computer worked great until the day I sold it.

In 2005 I picked up a Dell Inspiron 9300 P-M 2.0, Windows XP pro, 1 GB ram, Nvidia 6800 256MB, and 80 GB HD. This is the computer you will still see my using often as it has proven to be a true workhorse. It was one of the first 17" laptops on the market and I'm very thankful to have it, as it's been worth every penny and them some. I just used it today to do a client presentation. The only complaint I have is that the screen resolution is only 1440x900, but starting with the 9300 Dell discontinued the anti-glare HD screen and if you're read this blog before you know my opinions on that matter. I've been watching ebay and I can pick up a matte HD screen for around $200 but it doesn't seem worth it.

I won't even write about the mistake of an HP laptop purchase in 2008, but I also purchased the Apple MacBook Pro 17 in 2008. This is by far the nicest and best spec'ed out machine I've every owned, although it will be hard pressed to beat the Dell 9300 overall. It has a C2D 2.0, 4 GB ram, 320 GB HD, Nvidia 8400 GS 256 MB, and HD matte screen. I'm not crazy about the apple software, accessories, or fashion statement component that came with the purchase, but it's hard to argue with the form and design of the machine. We'll see how long I keep this one, but so far I think it was a good deal overall.

Well, that's about it for my trip down memory lane, now it's time to get back to work. Hope you enjoyed it and think about how you got started, comment on it below. Let's see who has the oldest rig still in daily service. (2005 here)

Rates - What is right and what is wrong

2009-05-27T16:12:00.000-04:00

The other day I mentioned that the economy was having an effect on the rates consultants can charge for products and services. In most cases, the effect is a negative one. This will be a good topic for today because the rates are always the most important part of a project.

Part of being a consultant and business owner is trying to figure out the right amount to charge clients. At DI, we have a range of rates that we charge depending on the services requested. In general, these rates rise with time as the company grows. I have also seen it go the other way though, as a new technology arrives the first few people can command very high rates for knowledge but then as more competition enters the market the rates are forced down.

I had an interesting exchange with a partner last week. This partner was talking about a potential project for a fairly low rate. The plus side was getting in a project at the ground floor with the potential to develop a long term relationship. The negative side was establishing a benchmark rate that is significantly lower than our existing work and likely severely limiting to any upside we might have with this client in the future. In speaking about this issue with one of my advisors, I called it the "GM" problem. The so-called problem with GM is that the car company is selling discounts instead of cars. When customers think of DI, I would prefer them to think of "quality work" and "excellent value" instead of "cheap". The advisor started laughing and agreed while pointing out that I drive a GM car.

This is a problem that doesn't have a cut-and-dried solution. In my head I have a rough formula that I follow but it's different for every situation.

The key point to remember is that rates are by and large a function of the economy, demand for the skill, supply for the skill, and any other factor (such as brand reputation, client sense of urgency). We can decide what the bottom line is and concentrate on projects over that line, and where your line is begins with you.

It's Us vs. the Economy

2009-05-19T23:18:00.003-04:00

The other day I received a positive comment about a blog post I wrote last June titled "The Great Debate - Employee vs. Consultant". The recent events in the economy with layoffs and project cancellations has made it an even more relevant topic today, although some people might not find much choice but to do contracting/consulting due to the lack of full time employee positions available. To be more specific, there are a lot of jobs available but even more candidates (competitors) for those jobs.

The past 12 months have shaken even the most die-hard consultants and employees. For us techies, it hasn't been as bad as 2001, but for those who have lost their jobs, that's not a very comforting thought. The economy has pressured DI some, but we're all staying busy and producing revenue which is always a plus. The positive thought is that those who can survive through this recession have a great shot at coming out on the winning side in the recovery.

Part of every recession is a drop in projects (i.e. work) and billable hours and rates. From my perspective, clients are being much more cautious with their money and scrutinizing hours and expenses much more than the past few years. Part of DI's value proposition is that we try to always think of the customer, as I did tonight when choosing a dinner entree at the local TGI Fridays, I opted for the wrap instead of the more expensive steak. That might be a silly example but it's the expenses at the margin that make a huge difference overall. Our consultants are also taking more time than ever to work directly with client managers to ensure we are providing the maximum value. Rates clients are willing to pay haven't changed much for us, but we've found some of the competition willing to undercut rates in a major way to drive revenues. In the end, we do not expect any negative long term effects from this behavior and believe this kind of competition will end later this year.

One major casualty of the economy has been the training market. Inquiries from training had been a decent part of the business but have become a very minor part. I expect the training business to be very poor well into the recovery and probably not good until we're nearing full employment, when companies will once again use training as a perk to attract and keep employees. Right now, people are just thankful to be working and not necessarily concerned with getting professional training.

At the end of the day, the only thing we can do is to continue moving forward. In the next couple days I'll be doing an update to the "Employee vs. Consultant" post that is more relevant to the current times, so stay tuned.

My Oracle BI Experience

2009-04-24T11:31:00.003-04:00

A colleague of mine invited me to an Oracle event that was held yesterday (Apr 23) in Tampa. There were two events going on - a SOUG Tech Day at the HCC, the noted speaker was Tom Kyte (asktom.oracle.com) and I was looking forward to seeing his presentation. A second event was held concurrently at the Ruths' Chris Steak House in Tampa, titled "Oracle Business Intelligence Briefing". I agreed to meet my colleague at Ruth's Chris to listen to the new, updated Oracle Warehouse Builder presentation.

For some background information, I have a book about OWB on my bookshelf and the first version of the tool never really took off, and I very rarely get inquiries about it, but they do come from time to time so I thought it would be a great opportunity to give the tool another look.

The meeting was from 11 AM to 3 PM and I arrived at 10:30 AM. There were about 20 cars in the parking lot. For reference, last month Information Builders held a lunch at the same location going over their new BI product, there were about 50 cars and a good 60-70 people in attendance, and it was a great lunch and good information. Back to the story.... I walked in the door and went to the registration table. There were a couple suits walking around eying everyone but not saying much. The lady at the table didn't have my information, but took my card and gave me a nametag and directed me to the breakfast area. I noticed about 20 people sitting at tables.

The whole place had the feel of a funeral. It was quiet and no one was saying much except two guys talking about flying a lot between Ft. Lauderdale and Tampa. My goal was to hear about the product and make at least 3 contacts, so I decided to join their conversation. A great segway was about flying. One guy mentioned that he was flying on that route 2-3 times a week on Southwest; I often did the Tampa-Miami run on American, sometimes 2-3 times per week. My record was 5x in one week last year. So this is a good point in common so I walked up, recounted my story, they laughed and sympathized with me and introduced themselves. The guy said he worked for Oracle and asked me what I did, I told him BI consulting. What he said next astounded me.

"You have to leave". I laughed and resumed the conversation but he just looked at me. I said "Is this some kind of joke?" He said "No, serious I need you to leave you can't attend the meeting". No one else was really saying anything and the Oracle guy wanted me to walk to the door with him so we could talk. After we walked outside, I asked him why Oracle would have listed on their website a BI briefing and then ask people to leave. He replied that this event was paid for by a consulting firm and they instructed that anyone not a client or potential client be asked to leave. He said that it's a sign of the times and the client paying is afraid of other companies stealing clients. Fair enough, I left just kind of surprised by the whole deal.

Interestingly, my colleague (a friend in the business, not a co-worker) was sitting at the table talking to people enjoying his breakfast when he was asked to leave.

This was definately an interesting impression on the Oracle BI product, and not the one I expected to receive.

The good news was that I was able to attend Tom Kyte's presentation at HCC and ran into some former co-workers and it was good chatting and listening to them.

Orlando Code Camp Review

2009-03-29T20:12:00.001-04:00

Yesterday I gave a presentation at the above mentioned event and it was the best one I have done so far. Had a great room, good audience participation, I felt awake and alive, and everything seemed to flow quite smoothly. I want to thank the Orlando organizers for hosting another great event. I look forward to my visits there, between the fall SQL Saturday and spring code camp, it's one of the highlights of my talks each year.

Sprint Phone Comparison

2009-03-19T13:15:00.002-04:00

I thought that today I could write a bit about cell phones. I prefer to use PDA phones since it's easiest to keep up with the business from these phones with respect to emails and such, and to disappoint a few, no I do not use an iPhone nor have any plans to get one.

I was using a Sprint 6700 until August of last year, when it failed me on a business trip. Dealing with refurbished models isn't something I enjoy, so I bought a Sprint Mogul brand new off Ebay. The Mogul was a good phone, I really liked it but it had issues with memory management. The phone would also shut off at random times, leading to missed calls and interrupted calls. I downloaded all the software updates but still ended up doing a hard reset on a daily basis. As a last ditch effort, I found a custom WM6.5 rom on the internet and loaded it on the phone. I must say that WM6.5 is a great software update and I can't wait for the official release, but it was too much for the Mogul to handle, and required twice daily hard resets to clear the memory.

Sprint offered to upgrade me to a Touch Pro, the newest line in PDA phones they offer. Because the price was right, I elected to try this. It meant that I had to turn in my Mogul though, as it was an exchange. My first thoughts of the Touch Pro is that the screen is beautiful, lots of resolution, but i miss the wide stance of the 6700 and Mogul. The Touch Pro is a little thinner, more the size of a traditional cell phone, but I have become used to the wider phones and that has taken some getting used to. The functionality is flawless, the phone works, email works, and it has plenty of memory which is much appreciated. The form of the phone seems fragile though, I can't help but think that one drop and it's toast.

For now, I'm pleased to have a phone that works, but I find myself missing the Mogul. I just found a Wm6.5 rom for it so I'll do my own software update and try to give this one a month to see if my satisfaction improves. If not, it will be back on Ebay and I'll move to something else.

The Future of ETL

2009-03-17T21:04:00.003-04:00

Yesterday I wrote about the arduous task of installing IBM Websphere DataStage. I mentioned that DataStage has been the tool of choice for high-performance ETL, and I wanted to elaborate on this topic today.

In my talks that I do at Code camps, I always take a minute or two to talk about ETL tools. This is a very important part of data warehousing, perhaps the most important part (aside from the data). Too often, the ETL tool choice is made after hearing sales pitches but before really studying the differences in the tools.

There are three main tools on the market: Informatica, DataStage, and SSIS. I haven't directly worked from Informatica, but I'm told that it's similar to DataStage without the hashed files (Server edition). DataStage has some very, very high performance characteristics when using Enterprise edition with large data volumes on a partitioned tables/database design.

However, Microsoft SSIS is really coming up on the inside performance-wise. The 2005 version was very good and offered bulk loaders to load data, and the 2007 release of the native connection packages for different databases is a huge step in this direction. The 2008 version of SSIS is even better and has the cached lookup, which is a big leap for performance as well.

I've written before (in 2006) about some misses in SSIS such as the lack of an easy way to create surrogate keys. I know it can be done using scripting, but there isn't a transform function to do this (or use sequences/identity columns). However, these small misses shouldn't be enough to not consider the tool.

I believe that SSIS is the future of ETL as far as a platform because the price point and the features combine to create a powerful platform. As SSIS grows, other tools will become obsolete and SSIS will take a larger and larger portion of the ETL product market.

Installing IBM Websphere DataStage 8.0.1

2009-03-16T11:06:00.002-04:00

Howdy folks, I thought today I would write about my installation of the IBM ETL toolsuite on my server.

IBM Websphere DataStage v8 is the newest version of the IBM ETL toolsuite. DataStage has for years been the leading high performance ETL tool in the market. DataStage was created by Ardent Software in the mid-1990s, Ardent was renamed to Ascential at some point, and Ascential Software was purchased by IBM for a tidy sum in 2004. In 2005, DataStage 7.5 was released as an Ascential product, and version 8 (v8) is the first true IBM release of the suite.

IBM has made a bunch of changes to v8 to integrate it more with their WebSphere web services product and to position DataStage as a software-as-a-service model more than the traditional batch product. For those of us just looking at DataStage as a nightly run process, this is a negative thing. For those trying to integrate DataStage into webpages, real-time data integration, and right-time data warehouse loading, it's a positive thing.

I installed DSv8 for a client on thier machine last year. I've installed DSv7 on Unix machines and Windows as well. I was able to trade a couple hours of database work for an extra server that one of my clients had sitting around, and I feel it was a great trade. The server is a HP pentium itanium 2 gz x 2 processors with 2 GB of ram. It's slow by today's standards, but it is a 2004 model. The worst part is that it doesn't have a DVD drive or USB2.0 which made putting DS on it a lot of fun.

First, I had to start with a fresh O/S wipe. I installed Windows Server 2003 Standard R2. This took about 45 minutes. I then took a couple hours to transfer the DataStage files over from my laptop using an external HD.

For those curious, I purchased a software package from IBM that includes a development license of DataStage v8. It wasn't cheap, and it's not easy to find, but it's there. I wish that IBM would make it easier and the license terms less onerous. I don't have an extra $350k sitting in my software budget.

I used my laptop to read the installation manual instead of printing it out (saving the environment), and it was pretty intense. There are a lot of settings, permissions, users, and environment variables to configure. I then ran the installation package for DS, which installs DB2 9.1 for use as the metadata repository. Once DB2 is installed, the actual DS installation commenced. It ran flawlessly to completion, and took about 75 minutes in all.

Once I had the software set up, I had to start the websphere server. However I still wasn't able to log into DS. Puzzled, I began to research. I discovered I needed to configure the user accounts in the Administrative console, and while it wasn't easy, it wasn't that hard either. There are a lot of screens and options in this area.

I was then able to log into the Designer and create my first ETL package on my home server. I'm still trying to detemine how I will handle my servers, so I'll write more about server management at some other time.

For now, happy coding.

New Website Functional

2009-03-04T17:57:00.002-05:00

Hello all, I have a good announcement today

The relaunced website is up. The address is www.thedamndata.com. For now, this blog will still be hosted at http://thedamndata.blogspot.com.

The main addition to the website are two major components: Forums and News

The Forums are a discussion area that can be used to discuss anything from SQL databases to your best friends' dog fetish. Just kidding on the second part, but there should be a post for every want and need relating to data. If you feel we're missing something, please let me know and I'll make adjustments to them.

I'm searching for two moderators for the forums, please email me if you're interested.

The other exicitng part is News. The news feed is still being finished but news regarding data will be posted and continuously updated.

I hope you like and use the new website. Please refer your friends as well.

Thanks for the patience as the website was redone. I'm not a graphic designer and I'm sure there are a lot of changes in store for it in the future.

Market Update

2009-02-20T10:00:00.003-05:00

Hello again, I wanted to spend some time today giving a pep talk on the economy. It's a tough time, right? Lots of local events (SQL Saturday, South Florida Code Camp) that I've attended lately have taken on a ghastly pall as tech people talk to those who are out of work or those who are afraid they might be. I can say from my perspective that people aren't in as good of a mood as they were a few years ago. The lack of recruiters at the events is a telling sign as well.

My training as a an economist will help me put another twist on this topic. My small business in software has been hurt a bit by the economy a bit. I know some poeple who have been laid off. I talk to customers or potential customers daily. Things have definately slowed down in the market. But where does this go?

UP! Up, you ask? How? Have I lost my mind? Some would like to say yes, but I really don't think so. In my discussions with clients noted above, I have yet to talk to a single client who has cancelled a project. Most of the managers are cautious and content to push them into later in this year. Budgets have been crimped and a lot of money isn't available as it used to be. YET the work and the necessity of the work is not diminished. This bodes very well for the future of IT.

It's really a shame that many companies and managers view IT as an expense and not an asset. My company specializies in Business intelligence, this is a raw asset. Every penny that you invest in my company building your BI tools will easily return it's cost, and if it doesn't do significantly more than that I have failed. Some parts of IT have expense components - but the net results are assets.

For a look at the numbers, the official unemployment rate is about 8%. That means 92% of people are working. Economic growth is always sluggish in the winter months. It might be a slow spring and hopefully a better summer, but the work is out there, it's building, and the best thing one can do is to use this recession as a time to build up for the coming good times. Build your network, work on self-training. Read that SQL book that you're had for three years on your bookshelf. Attend a lot of the free events in our area, meeting new people, learning about new technologies that are going to be key in the next boom cycle.

A recession is a horrible thing to waste.

Website Update

2009-01-02T17:28:00.002-05:00

It's a new year and time for an update to this website.

While the update is in progress, I'm going to turn off the blog, so I've put my next posting live a couple days early. I hope to have the upgrade done next week so be sure and check back. It will probably take a couple of weeks to get the kinks worked out.

Thanks for the support.

A Positive Story from the Field

2009-01-02T16:00:00.000-05:00

Yesterday I relayed a story about a consultant who made a process improvement of little value to the client and in doing so, created a lot of headache. Today I'll recount a positive story of a value-added process improvement that made a noticeable difference.

One of the clients I was working with asked me to take a look at their database performance. They felt that it was pretty poor and they were right. One of the things I noticed immediately was the over-analysis of tables. To explain, they were using Oracle and had an ETL job that analyzed the tables as soon as the load job finished. The particular instance had a small amount of daily updates but the statistics were going stale on a daily basis. When I started looking at the load job, I found that it did not differentiate between true and false updates. A true update is when a data component changes, and a false update goes down the load option without being changed from its' prior state.

Generally in good ETL architecture, the architect will put an escape clause to only load the true updates by doing a before and after comparison. This keeps the load process free from only changing the update timestamp column as was happening in this instance.

We had to change this job to only load the true updates in the target table, and in doing so we solved the stale statistics problem and also cut the load process by a factor of 10x. By changing this job we were able to disable the analyze table routine thus saving more processing time. The client was quite happy in this instance as the total time savings will be in the hours by the time this architecture change is propagated across all their subject areas.

Seconds don't sound like much until they begin to aggregate over a period of hours. In my next post, I'll talk about separating DBA maintenance tasks from ETL processing.

Until then, happy data warehousing.

Stories from the Field, Part 1

2009-01-02T09:07:00.000-05:00

As I promised back in September, I would write about some experiences I've had with consulting. I expect there to be many parts to this title, so I'm going to set some ground rules.

1. There will be no client names mentioned (or person names).

2. No part of my posts should be interpreted to insult a client or person, I am only reciting the event as it happened.

3. Names may be modified to protect the parties involved.

Now that the disclaimer is done (and I mean it, especially #2), let's start with our first story.

I was on a team working with a few other consultants on a large data warehouse project. One of the guys (not brought in with us) was an independent consultant. Let's call him Rob, and Rob is a good guy, very interesting and funny, and his stories caused great laughter at lunch. Rob had a way of being very straightforward and profane, a quality that does not go well when working at client sites. It was difficult for him to say a complete sentence that didn't include the F-bomb or using the G-D word, or any other curse word you can imagine. This 'quality' frustrated a lot of the employees and gave me great pause into engaging him in conversation in front of other people.

One day in a meeting Rob had a great idea. In the client's ETL jobs, all jobs were designed to terminate at the first sign of an error, send a page to the duty pager, and have manual intervention to restart the process. Without getting into whether this is good architecture or not, it is what it is, and Rob had an idea that we could create a routing using an exception handler in all ETL jobs instead of just terminating them. He said it would be easier to code the jobs this way, and in a way he was correct. Rob's problem was that he was solving an issue that wasn't a problem for the client - the client was happy with the way the jobs previously functioned, and Rob's hours and effort to retrofit working jobs with this new technique took valuable time away from solving problems that the client had identified. However, the manager told Rob if he could get a sample done quickly, he would be open to changing the architecture.

A couple of weeks later and Rob presented his working idea at a team meeting. I had cautioned him in private about solving a problem that the client didn't seem to have, and my colleague suggested to him that this approach didn't take into consideration all the possible failure scenarios. Rob pushed on and in the meeting, my colleague again suggested in front of everyone that he wasn't convinced this technique would work as described, and cautioned Rob to test it with every possible scenario before putting this into production. The manager approved this code to be moved to production only after he had some tests for every scenario that had been described by my colleague.

By now you are probably starting to imagine the outcome of this scenario, and if your outcome includes the word disaster you are correct. Rob moved this code into production and the first night there was a hiccup, but because he hadn't tested correctly for that failure scenario, incorrect data was loaded into the production tables and the job did not abort as it should have. The next day people around the company were noticing data that was incorrect on reports and poor Rob had some explaining to do, which he did in a status overview laced with enough profanity to make a sailor blush.

In the end, after much testing and some more issues, Rob was able to make this routine work correctly but the client largely considered it a failure and chose not to move forward with this approach, now leaving one subject area with this architecture while the remainder used the legacy code.

For this story, let's analyze what happened here. Using the flowchart methodology,

Rob identified a process improvement that the customer did not value and Rob did not adequately test this process improvement, causing the customer to suffer delays and bad publicity in their company, thus causing them to be shy on implementing further process improvements.

I'll make the next story an example of success, but the moral of the story here is that if you are going to go out on a limb to perform a process improvement, make sure it's something that the customer values and last but not least, make sure that it works.

A Year in Review

2009-01-01T13:57:00.002-05:00

First I would like to wish everyone a happy new year. In many ways it's difficult to comprehend that another year has come and gone, a year that was challenging for many. I am pleased to say and say it with some trepidation that 2008 was an excellent year for myself and Durable Impact. DI is making some real progress in acquiring projects and clients and now I have a very good springboard for launching into 2009.

One thing I didn't quite accomplish this year is giving a lot of presentations. By my count, I only did five events this year. I was very busy building DI up and traveled about 32 of the 52 weeks. Traveling is not conductive to building a brand by giving speeches, but it is better to travel and make money than to give talks for free. I have been and will always try to find a happy medium.

SQL Saturday last February and the Day of Data event were great successes. Pam Shaw and I are hosting another SQL Saturday on January 24 at the KFORCE facility in downtown Tampa. Unfortunately, I was not able to put together a Day of Data to go with the Saturday, so maybe I'll host one in the spring.

With 2008 we saw a launch of SQL Server 2008 and some great improvements to the SSIS tool suite. With the coming launch of Windows 7 in 2009/early 2010, it's looking to be another good year for Microsoft.

As for my personal goals, I'm just interested in working, building up DI, helping out the community where I can, and having a general all around good time. If you're ever in Tampa and want to chat, drop me a line through the DI website (linked on the left) at the Contact Us form. I will also try to put up more blog postings, as I have a fairly strong amount of readership and don't like to disappoint.

Here's to a great 2008 and an even better 2009.

Degree Complete, Now What?

2008-12-17T10:00:00.002-05:00

Those who have followed me know that I've been completing a Master's degree in Economics for the past couple of years. I had to study at a slower pace due to job demands, and I'm pleased to announce that I received my diploma this past weekend, so your's truly for now is done with schoolwork.

A common question I receieve is "What are you doing to do with a Master's in Economics?" It's a good question as I primarily work with computers. I considered getting a M.S. in MIS, but thought this would be too redundant. I wanted to go the econ route because it's always interested me, I can shift in that direction with my career one day, and studying econ gives a very good understanding of the vagaries of the economy which affect every single one of us. Working with data, it's also given me a huge leg up with regards to efficiency and understanding business practices of the clients I work with. I was told that my degree had been shopped to a client considering a large data warehouse project because they wanted someone with both datawarehousing and analytical experience, and the degree became the differentiator that put me in a strong competitive position.

In short, it's like having a quiver full of arrows and I just added an additional arrow. Short term benefits are limited to the extra money I can make with the education, but long term benefits are unlimited. I'm glad I did it.

Extracting SQL in Data Warehouses

2008-12-16T21:57:00.002-05:00

In working with potential clients, I often get the following question in tech interviews, and the question is "what is the best way to extract data from a data source?". To use the name of Andy Warren's blog, the answer is "It Depends".

I was asked this specific question today and gave the response above. Knowing that's an incomplete answer, I elaborated. The response was something like "It depends on the machines that are running both the database and the ETL process."

It goes something like this. Let's say our source system is an order entry system that is a mainframe DB2 system, and our data warehouse is a Windows-based SSIS implementation running 16 dual-core processors with 32 GB of RAM. We can run the extraction a couple of different ways, including:
- Perform a simple select statement and pull all rows out of the database on the source
- Perform a simple select statement and pull rows with a constraint
- Run a compplex query on the database engine to pull the minimum amount of data.

The key differentiator in this dilemna is how much data do we need? I've worked on ETL processes that pulled 20 million records in a single ETL jobstream only to use a majority for lookups and then updated only about 1 million records. I've also worked on ETL jobs that looked like the third case above, running a complex query with where clauses out the wazoo and returning just a couple rows to the ETL process. I generally try to avoid the first choice above unless we need all those rows. It's also possible and common to constrain the query and have indexes and other tweaks on the database side to speed query performance.

The way I would do it is dependent on the performance of the two machines. I always try to minimize long running network pulls, but if we run a significant query on the source system we may negatively impact their performance. Thus, it's such a localized question that is dependent on the environment.

Some good benchmark rules for extracting data are:
- Don't run queries on source systems that have negative performance impacts
- Don't have long running network connections to pull data
- Do utilize the machine with the most power for complex tasks
- Use all processor available to complete tasks faster
- Use specialized database connections (vs plain ODBC) to improve performance

Finally, always remember the correct answer is "It depends".

Computer Update

2008-12-01T23:21:00.002-05:00

Hello again, as if you're not tired of hearing my long running discussion about work computers, here's another one and probably the last for a while.

After my latest reformat of my Dell computer, it's been running like a champ. Running so well that it seems to be brand new. I'm very pleased with XP Pro and every day I'm reminded why I don't like Vista. I was visiting one of my clients the other day and a mass comm email went out to the group saying that in 2009 there will be no upgrading to Vista, and this is a company that has thousands of PCs. I have yet to work with any clients running it, and it seems that Microsoft acknowledges the issues with Vista and is pushing Windows 7 out for an early 2010 release. From the specs I read a few weeks ago, Windows 7 is going to be a great system and use significantly less resources than Vista.

My biggest concern with laptops is the screen. I prefer to use 17" lappies, and the screens vary from good to poor depending on the model. When I bought the Dell in 2005, I didn't get the HD screen since it had a glossy finish, and there is not a single Windows-based consumer machine on the market today that doesn't have a glossy finish. I purchased a cheaper HP laptop this year to see if the gloss would be as bad as I thought, and I'm sorry to say that it is. The colors are very vivid, but that computer is worthless is a majority of the settings in which I need to do work, typically corporate offices or outside on a beach. Dell offers a business 17" model (Precision line) which is very expensive and the same form factor as my current comp, which looks dated. My searching led me to the Apple MacBook Pro 17, and with the current new line of MacBooks having went only to glossy screens, it was now or never for the 17, lest Apple only offer glossy on that one.

I'm typing this post from my new Apple MacBook Pro 17 with a HD screen (1900x1200) with matte finish. I must say this screen is the most beautiful computer I have ever worked on. It's powered by LED so it's bright and easy to read, and offers a lot of real-estate for doing remote desktops and development apps. It has a 2.53 Ghz C2D processor, 4 GB ram, and 320 GB hard drive. I installed VMWare Fusion 2.0 and run Windows XP Pro on it. It took me less than 2 hours to have this computer set up exactly like I wanted, and I even have a W2K3 virtual machine with all my servers on it. In short, I finally own my dream machine.

I'm sure I'll catch hell from the guys for using an Apple, but the screen sealed the deal. I would only be happier with this purchase if it was free.

For now, the laptop search is over and I'll try to get back with some consulting stories. Remember it's all about the Damn data.

Code Camps vs. SQL Saturday

2008-10-21T00:10:00.001-04:00

I had a good conversation with our developer evangalist (Joe Healy) the other day, and he mentioned that SQL content is way down at code camps due to the pull of SQL Saturday. As I was preparing my latest presentation for this weekend, I felt a little excited. I then began to wonder why I was excited to prepare slide decks, and realized that I hadn't given a talk since May, and I've only done four this whole year (February, March, May, and October.). Feb. was in Miami, and May in Jacksonville, and March and October in Orlando. . In 2006 and 2007 I did 7-8 talks per year, so either people are getting tired of hearing me talk, or I am not getting out as much as I used to.

After a few minutes of thought, it dawned on me that Tampa code camp was moved to December, and I missed Jacksonville and Tallahassee due to traveling to client sites. So the problem is not that I'm doing less, it's just that I'm doing different things. I would definately rather work and make money than give talks for free, but I do get a great return of satisfaction having passed on some knowledge at these events.

Going back to Joe's comment, he is correct that as the events have pretty much split in two, there is less database content at Code camps. I begged and pleaded with Keith to give me a SQL track at Tampa code camp a few years ago, and while we had a good representation, there was never the content flow that I think would be an awesome winner. I'm hoping to keep some sort of presence for the code campers who want to know database stuff but don't know or aren't interested in it enough to go to a SQL Saturday.

As I learned when I was a kid, if you can't find a road, make one. Keep your eyes tuned to the upcoming 'Day of Data' event.

Installing SQL Server 2008

2008-10-20T00:06:00.000-04:00

I recently installed my first version of SQL Server 2008. I used the Enterprise edition and installed on on Windows Server 2003 running in a VPC. This is certainly not the ideal setup for any kind of benchmark analysis, but it was a good experience.

The installation proceeded very easily. I first had to install the .NET 3.5 Framework, reboot the machine, and start the full installation. I chose to do Windows authentication and set everything to be default and the same user accounts. This is not a good practice for production environment, but if my laptop is considered a full-scale production center, consider me guilty.

Having done a 'select all' on the components, it took just over 1 hour for the full software to load, but I was very pleased to see all the services come up without so much as a reboot. I've been playing around all night with Change data capture (CDC for short) so I hope to write about it soon.

We're up an running, and that's a start.

SQL Saturday - Orlando

2008-10-19T10:00:00.001-04:00

I'll be attending the 2nd annual Orlando SQL Saturday event on October 25. Andy Warren and the OPass crew host this event and they did a heckuva job last year and I'm sure this year will be even better.

My session is all new: "Building a Data Warehouse using SQL Server 2008". This session is all new with a great new data warehouse example. I have been using the airline example, but we'll step it up a notch and look at not only how a major airline can use SSIS, but also how a major manufacturer can as well. I'm really excited about this presentation, as I believe SSIS 2008 will be the version that finally gets the respect it deserves in the marketplace.

I already have the slide deck done and posted on the website (wow, that's a record for me) so if you want to print it off beforehand you can get it at http://durableimpact.com/resources.aspx

Hope to see you there.

Complete System Reformat

2008-10-18T19:01:00.003-04:00

I have written before about my desire to replace my Dell laptop that I use for business with a newer model. After months of fruitless searching, I found the only suitable replacement - an Apple MacBook Pro 17. The steep price tag of the Apple had me reconsider exactly why I needed a new laptop, and I came up with the following reason. My Dell is slow. Slow is hard to quantify, but the hard drive is constantly seeking, and then I had an ah-ha moment. I remember this Dell was quite snappy under Windows XP Pro but seemed to labor under Vista. What if I took it back to XP Pro? Would the speed be acceptable to get another years' life from this machine which has served me well?

Today I found the answer. I just spent 6 hours of my life reinstalling everything from Windows XP Pro to Office 2007 to SQL Server. I was able to save my virtual machines, and VPC is running 10x faster now than it did before. The only negative I have seen is that the Windows Mobile Device center is no longer and it's back to Active Sync for the cellphone.

Why do so many people have problems with Vista? I think the software itself is not too bad, but it is definately a bloated O/S. My Dell is maxed out at 2 GB ram, and with Vista and VPC I would fill it all up. Doing the same on XP Pro I am still at 1.2 GB with everything running. I think I will be very happy with this reformat, and thus be able to squeeze another year out of this machine.

Aside from dealing with a slow Vista system, I have had some problems with WM 6.1 on my Sprint Mogul. This version of software seems to have memory leaks such that the memory fills up and necessitates a reboot every 3-4 days to keep everything running OK. I'm slightly surprised and disappointed that Microsoft has not yet fixed these errors. The 6.1 software is 10x better than the old 5.0 that I had on my last phone, but I am definately looking forward to Windows 7 for both the laptop and phone. Let's just hope that MSFT puts a lot of work into slimming down both systems to make them run better.

IE 8.0 Review

2008-10-11T22:23:00.002-04:00

Hello readers. It's been a while since I posted and I apologize about that. Let me start with a comment about Internet Explorer 8, which I have installed on my Dell laptop. Yeah, this is my main computer since I still have not found anything on the market to satisfy my needs (See earlier post about Laptops). Apple is coming out with a new line this week which might be promising.

Anyways, I installed IE 8.0 which is supposed to be more stable and quicker than IE7. Let me say that I use IE for everything so I can't compare it to Firefox. IE8 does have some nice right click menus (called Accelerators) which allow different functions (Blog with Windows Live is one). The concept of accelerators holds a lot of promise. With regards to websites, I've had no issue with about 95% of the pages, some of them work best in compatibility mode, which is a great mode inclusion for the browser. It does seem quicker with less overhead, and the 3 or 4 times that it's crashed, it pulls itself right back up where it left off, which is nice cleanup work. The one complaint I do have is how much the brower slows down with Flash pages, and Adobe PDF files. I'm not sure why that is, but let's hope it gets fixed for the final release.

All in all, a nice upgrade from IE7.

The Life of a Traveling Consultant

2008-08-31T12:51:00.003-04:00

Hello from here and there. Today I was flying on AA, sitting in my seat waiting for us to takeoff when a beautiful AA Boeing 777 came by on departure and the percussion from the jet engines caused my heart to skip a beat, and as I watched the beautiful jet gracefully defy the forces of gravity and lumber into the sky, I thought it would be a good post to write about life on the road, so here it is.

I had often dreamed about the glamorous life on the road, thinking it would be all about staying in nice hotels, flying commerical airplanes all the time, eating in different restaurants and meeting some very interesting people, and I'm pleased to report that's exactly what it's like. In the past I would just stand in awe at the airport watching the lucky business travelers work their way through the system and know the behind the ropes' scene of current day air travel and now I'm one of them. I had a great innoculation due to the excessive personal travel I've done the past few years, so much that I had already attained status on airlines and hotels prior to becoming a modern day road warrior.

Being a road warrior is not for everyone. For instance, my car spends more time at the airport and in the garage at home that it does being driven - for me that's a plus since I'm not a fan of manejando (driving in Spanish). This kind of lifestyle becomes readily apparent when working at a client site. I have more than once found myself with a corporate mentality of "heck it's 5 PM time to go home" only to remember that I'm not home and leaving would just mean more hours sitting in a hotel room reading, so why not stay at the office a little longer and get more work completed. Thankfully most hotels today have wonderful service and beds and I have found myself enjoying the accomodations quite well, even if sometimes I would like to just be in my own home, watching a baseball game while sitting on my couch.

Another benefit (or not, depending on your perception) is continuous learning of how other companies operate. Just like any corporate job, I have to get in there and play a game of politics to find out who are the power players, who gets stuff done, who knows what, and importantly, who not to listen to. In past experience, I've found some most interesting information from people in the last category, as they might be clammed up in a corporate world, but have a lot to offer to someone who is an outsider (like my role). One thing I've learned is that each organization has it's own quirks and monkey hoops that you have to go through to get even something menial completed, but accepting this is just part of the game.

I'm going to write more on this later and tell stories of some of the battles I've fought and share some of the knowledge I've learned, but now I have some free time and personal time is short for us road warriors, so I ask that you carry on, enjoy your life, appreciate the small things, and have a great Labor day.

TDWI Membership, Is it worthwhile?

2008-08-04T16:10:00.002-04:00

The primary organization in data warehousing is TDWI - The Data Warehouse Institute. Many of you might also be aware of PASS for SQL Server, or the Oracle User groups for Oracle. Today I want to talk about TDWI and get some feedback from you as to the value you find from your membership.

I was a TDWI member but am not currently. I attended two TWDI World Conferences in 2005 and 2006 and while I learned a lot at the first conference, the second was interesting but almost the same content. A lot of the comments I've heard from contats are that TDWI conferences often have the same courses time after time which is good for new attendees but not so good for repeat attendees. The biggest complaint I have about the conference was that my contact information was sold and I received about 20 phone calls per week for six months after the conference, and at least 1-2x a week for 2 years following my attendence.

Along with the membership comes access to reports and website information. There are some good case studies and other reports and I often receive emails requesting to fill out a survey about salaries and projects. One complaint I have about these surveys is that by filling it out, I give them information which they resell to others without receiving any information that helps me in my competitive situation. Most salary information is also given at salary.com. To be fair I have found some surveys to be worthwhile (companies starting projects, etc), but again not worth the $299 per year.

Now TDWI is trying to break out into forming user groups. My premonition is that their membership is declining and they are looking for more ways to reach out to the community (and more membership sales). The brilliant part of this strategy is that they get a huge networking community together and that definately has a lot of value for the membership.

In short, I think TDWI is useful and worthwhile to attend a conference every couple of years, but I'd like to hear and comments about the value you have received from your membership.

Continued: How many packages do I need?

2008-07-29T21:08:00.000-04:00

The other day we looked at a scenario that described the difference between one and many packages. I also put a ghoulish picture of a nasty looking package in my post to scare you away from doing a large package when organized, smaller packages are the better choice.

Let's look at the following case scenario. Great Plains Foods, Inc has hired us to create a database load process for them. Their requirements are that the data is loaded from a flat file every day by 10 AM and any failures need to generate an email to their production support control. We are to load 5 data files into 5 tables. The information below shows us filenames, expected daily row count, and the destination tables.

File Name Expected Count Target Table
Orders A 5500 DAILY_ORDERS
Orders B 1200 DAILY_ORDERS
Production 7500 DAILY_PRODUCTION
Employees 350 DAILY_EMPLOYEES
Shipments 900 DAILY_SHIPMENTS

From observing the data, we see that the orders files A & B load into the same table. A question for the client should pop into your head at this point, which is "What is the difference between Orders A and Orders B files?" Our business associate at Great Plains tells us that the files are the same but A comes from computer orders and B comes from telephone orders. Now, from an architecture standpoint, how many packages would you create for this job?

I would create 4: Orders, Production, Shipments, and Employees. The Orders job will read from two files while the remainder will read from the single source file. I have seen single packages load all these tables in a single job, but that kind of architecture does not leave any room to manually load a single file or to handle data related issues and failures.

What would you do in this scenario?

Package Failure Notification

2008-07-28T09:18:00.001-04:00

One way to guarantee disagreement among a room of ETL developers is to begin a discussion on how to notify when a package fails - and that argumentative topic is the subject of our discussion today.

When I'm working with clients designing ETL jobs one of the first questions I ask from my fact finding checklist is "What is the package failure notification requirements?". This answer not only varies by client but also by team members within the same organization. This simple question often has to be escalated up to the senior manager to get the correct answer. Developers often want to be notified by email, preferably next-day notification while team leaders and managers want people notified by pager or phone calls when the package is a critical data task. I've also found developers on the same team desire next day email and immediate phone call from the operations control even at 2 AM.

There has to be a middle ground in notifications. When business critical metrics are at risk of not being met, immediate notification is a reasonable requirement. If no one will be looking at the data until Monday and it's Wednesday, an email should be sufficient. Thankfully, SSIS has a Send Mail task that can be used to generate emails for failures or abnormal operation. A good example of using Send Mail task is when a process runs but only processes 10,000 records when the daily average is 100,000. In this case, a proactive email can be sent to the developers to do some research before the business metric is published wrong and senior management notices. DataStage 7.x also has notification capabilities when using the Sequencer and a notification task.

The decision on notification activities should be as consensus as possible among an organization and thankfully the major ETL players in the market provide easy development of notification processes in ETL jobs.

Tomorrow we will do a deeper dive into yesterday's topic of splitting up ETL jobs so stay tuned.

One, Two, or More Packages?

2008-07-27T17:11:00.005-04:00

For today's topic we will look at a a major ETL architecure consideration: how many tasks should we have in an ETL package?

One of the beautiful things about ETL is visual data flows that really help the architect consider the best way to move and transform data. Alas, each ETL architect has a different opinion on when to split tasks out into a separate package or to put everything in a single package.

I'll use another consulting experience that I have seen - clients who want to put all kinds of data derivations in a single package. My general rule of thumb is split off packages where there is a logical split in the data - have a separate package to handle order data and sales data. Sometimes there is other reasons to have separate packages - maybe you want to have a package that loads up a SQL Server database and another package to write to a Oracle instance. Normally I would just use Expressions to keep from creating multiple jobs, but consider the case where there are two processes that are similar but not exact, where using an Expression would not be practical.

Another key consideration is for restart capability in event of failure. I don't want to design a package that loads up data but has no ability to restart in case there is a referential integrity violation in the middle of the load.

For more information, see my ETL Architecture design document and/or ETL Architecture considerations presentation I did in June 2007 on the Durable Impact website to get a better idea of some of the considerations used to determine how to break up processes into separate pacakges.

As a final note, don't have a package that looks like the one above.

Laptop as primary work computer?

2008-07-26T13:55:00.000-04:00

Today I'll diverge a bit from talking about software and discuss hardware and a topic that everyone as a strong opinion about. Does a laptop suffice as a primary work computer for us techies?

For my situation the answer is yes, but. Yes, but includes a nice external widescreen LCD monitor and extrenal keyboard and mouse for the house. My personal config is a 24" Dell LCD and ideally when I get a bigger desk I'll get a second 24" LCD, but two monitors might be problematic with laptops that generally have one VGA out. Some of the nicer laptops have VGA and DVI, but more on that later.

I prefer to use laptops because * ding ding ding * I bet you can guess the answer - they are portable. The newer generation of laptops are sufficiently powerful to run every piece of software that I need to run and it keeps me from having a server stack in my office, which is good because I sold them all last year.

This topic came up today because I am looking for another DTR laptop (dtr = desktop replacement). My choices are the Dell Studio line and the HP dv7t. I'm not overly excited about either one of them because they cut a lot of important things like DVI output and high end speakers and removed the option for mattee screens which is a nice option in an office setting. I don't buy the business machines because they do not offer a 17" screen for whatever reason.

My ultimate goal is to have two laptops set up to be ready to go at any time ....1 set up with the Microsoft BI suite and another set up with the IBM BI suite. Once I choose a computer and get them setup I will write a review about it.

Do you use a laptop in place of a desktop?

DataStage 8 New Features

2008-07-25T10:32:00.000-04:00

I have recently taken a good look at the IBM WebSphere DataStage version 8 product offering. v8 has a very tight integration with WebSphere product and is the first true IBM iteration of the previous Ascential toolsuite and it shows. Installation on a Windows 2003 Server product was a really painful process. First, I read the entire manual which has a lot of steps to create users and set permissions on files and folders. Then I installed the software and had repeated failures when configuring the WebSphere product. The installation was failing for unknown reasons, and each time I would have to remove DB2 and WebSphere and restart the process. Finally I was able to successfully complete installation and get the software running.

v8 installs both DB2 and Websphere server but it's possible to use SQL Server or Oracle for the MetaData server instance. DB2 is not a bad database for those who haven't used it but there are definately some quirks (especially with dates and times) for those used to the other platforms. For right now I am using v8 with DB2 to keep things simple, but I will demonstrate the v8 platform using SQL Server, DB2, and Oracle as the MetaData repository to test the performance of each platform.

After I finally got the software installed and rebooted the machine in preparations to prepare screen shots for this blog, I was unable to get the WebSphere server running which means I cannot log into the Information Server portal. Once I troubleshoot this issue I'll post a fix, as I suspect I'm not the only person to have this problem.

Business Intelligence Development Studio, or BIDS for Short

2008-07-24T11:15:00.000-04:00

Part of the allure of the Microsoft platform for business intelligence is the BIDS, or development environment used for the platforms. BIDS is directly part of Visual Studio and will make an easy transition for anyone familiar with VS. I actually like BIDS although I do have some gripes and I understand there are a number of changes in store for BIDS 2008 that will be detailed in a later writing.

One of my main beefs with the BIDS is that sometimes it's overly complicated to do something. Microsoft has always been known for giving developers full control of the environment and BIDS is no exception to this, almost to the point of there being so many options in the tool that it's easy to get overwhelmed. I always put the options I use most often on the toolbar so I don't have to wade through menu after menu of options. It would be great if Microsoft used the ribbon in BIDS 2008 as they have used in Office 2007. The first time I saw the ribbon I was intrigued and soon became a fan and don't even like to use older versions of Office anymore.

The main positive about BIDS is that there is full control over aspects. A key annoyance is all the windows that are docked but it really is a love/hate relationship for me, as I love having access to it, and I like to hide the windows, but I find myself constantly pinning and unpinning the windows when working with my laptop, although it's not so bad when I'm working on the 24" LCD in the office.

What changes are in store for BIDS 2008? You'll have to wait a couple of weeks to find out - I am giving a presentation on August 11 about the 'Features of SQL Server 2008' that you will find interesting.

The Importance of Data Quality

2008-07-23T15:08:00.002-04:00

The key concern in data warehousing is quality data. If you have seen me present, you probably won't forget my emphasis on having data quality mechanisms built into ETL processes. Let me tell you about a scenario I was dealing with at a client recently.

The client wanted and ETL process built that would do the following:
1. Pull all records from table A with a insert date of yesterday into a flat file
2. Delete all records from table A
3. Run this process daily

Okay, so this is simple enough. I set up an ETL job that read the rows from a table and put them into a flat file. I then prepared a second job that would do the delete with the same date criteria. The purpose of doing this in two jobs was to ensure the backup file was created and populated before any delete process executes.

I tested this process and the counts looked great. I implemented in production and checked the counts of the two processes in the morning and they were different. I thought something must be horribly wrong because I had tested this process. After doing a little research with the client it was determined that Table A was being updated on a 24-hour window process, so by the time my process ran in the morning, all the records had not been loaded from the previous night, resulting in records being deleted that were not in the backup file.

The solution to the problem was to set the date back to 2 days prior and voila, all the record counts between archive and delete matched up. The issue with some records not being in the archive was not a problem due to the information being replicated on the source system.

The purpose of this posting is to describe again that data quality and doing simple checks of record counts between processes (especially delete processes) is a pressing and fundamental part of the data quality process. As data systems architects and developers, it is our responsibility to ensure that we build and check processes to ensure requirements are met. It might have been an issue if I have never checked the process after moving it into production. It may have never been discovered. It doesn't matter though because it's part of the data quality stewardship that is so important in today's business intelligence systems.

Datastage Lookup vs. Join

2008-07-11T11:25:00.001-04:00

Today’s topic will do a deep dive into ETL architecture considerations for the IBM Websphere DataStage product. Specifically, I am going to discuss the difference between the Lookup and Join stages in version 7.5. For those who might be confused, version 7.5 will cover 7.5.1, 7.5.2, and 7.5.x2, 7.5.3, etc. I've worked on clients will different variations of the tool but the functionatlity for these stages is the same.

This is a real-life case study of a problem I had desiging a job for a client. The job read from a source table with a driving table containing rows that were to be pulled.....a generic SQL query to do the same thing would look like:

select * from BIG_TABLE where record_type IN (select * from REF_TABLE where process_date = current date)

In DataStage I could have done a couple different things. I could have used a database stage to pull the rows with the join in a single stage and had good performance by forcing the DB engine to do a majority of the work. However, for the purpose of maintenance I chose to use a couple different stages to do the job because the client liked having stages on the pallete for people who support the jobs to easily see what is happening. Fair enough....

Thus I had the option of using Join, Lookup, or Merge stages. Merge is designed to be used to have two similar datasets (original and update sets) and merge them together. Since both tables had different metadata, this was not the best tool for the job. Eliminated from consideration.

The Join stages has options to perform a left, right, inner, or full outer join. Without going into extreme detail of the four types of joins (research SQL joins), I could set up the driver as the left table and the reference as the right and match the rows based on the date column. Okay, so this solution would work but it wasn't really what I wanted to do, so I went to the Lookup stage.

Lookup is similar to join except that it looks up a row in the reference dataset for each row in the source dataset. This is memory intensive and should be avoided on large datasets and I was dealing with millions of rows. I knew that my reference dataset would only have a maximum of 9 rows per run and there is this nifty little option to have the lookup table loaded into memory. Ah ha!!! Now I can load the reference table into memory, perform a lookup on the rows in memory, and get some great performance. For very large reference datasets the join will use less memory, but it worked perfectly for this application.

With some further tuning of the job that was initially running 45 minutes per run, I was able to get it down to 6 minutes. Huge performance increase for a little time spent thinking about better ways to architect the job.

Do you have a good example of where a simple ETL architecture change saved you a large amount of processing time?

The Great Debate: Employee vs. Consultant, Part 3

2008-07-01T14:37:00.000-04:00

This is part 3 of my series on the debate between being a W2 Employee vs. a 1099 Contractor as it relates to the technology world. In my previous posts, I concentrated around the following points:

- There is too much focus on provided benefits and no easy way to attach a dollar value to them

- Without a dollar value on benefits, its’ hard to determine your real wage

- It’s hard to make a good decision on what to do without knowing your real wage

My earlier conclusion is that many people would be better off as contractors instead of employees if they truly looked at the dollar amount of benefits but this summation comes with the following major caveat: the person should be entrepreneurial in nature and be willing to accept the uncertainty of continued employment. Working as a W2 employee you have some protections against zealous bosses and other events out of your control that you do not have as a 1099. Furthermore, many companies have a vested interest in promoting a stable workforce for support purposes and having a staff of consultants who come and go does not help with this goal. This is a key concern of many hiring managers I’ve talked to – they are concerned they will get a 1099 who will come in, develop some stuff, and bail quickly to go make $10 more an hour on another project. It’s a valid concern because this has happened a lot in the past, or the consultant walks away without documenting what he/she did leaving the rest of the people in a lurch. These kind of situations can be rectified by open, honest communication between both the supervising manager and the consultant, but the key point to remember as a consultant is that unhappy clients do not result in additional work.

Some of the feedback I received from part 1 & 2 revolved around the uncertainty (risk) factor. Using economic techniques risk can be modeled if you know the probability of each outcome, but there is no real way to know the outcomes in most job situations, so it’s best to plan on billing the client a rate that takes into effect your desired risk premium. For an example, let’s look at a DBA who can work as W2 for $80,000 or as a 1099 for an hourly wage. This DBA decides that he needs a risk premium of $50/hr to do the job, putting his hourly rate at $150/hr. This is a pretty high rate for a DBA so unless there are special skills involved, it’s not likely this DBA will find many clients so the DBA should probably stick to a W2 job. Another example (quite common) are the consultants who like to work 6-9 months a year with some downtime to travel the world or do whatever. In this case, the risk premium would be substantially less because the person is hoping to be out of projects for a time. Someone wanting to work 6-9 months a year is definitely not a candidate for a W2 position.

The biggest and most desired benefit is health insurance. If I were to put on my economic hat, I could come up with 10 reasons that employers should NOT provide health insurance (and a few reasons they should, mainly adverse selection), but since this is a technology blog, let’s accept that most employers make health insurance the key benefit in their compensation package. My first W2 job provided ‘free’ health insurance and it was a good plan. I’ve seen good plans and I’ve seen bad plans, but this is definitely a consideration for anyone who has a family to support. I’m not going to say a whole lot more about this benefit except look at the plan closely to try and attach a free-market price to it and use this in your job search. Two plans which have a similar bi-weekly deduction may have drastically different out-of-pocket expenses or doctor networks. Caveat emptor!

By and large, the biggest benefit I see for companies is the ability to have flexible scheduling. Having a flexible, motivated workforce is key to economic growth. This is why you will mostly see large development projects staffed with 1099s who have the specialized knowledge to perform the work and leave the after implementation support to the employee base. The best projects I’ve worked on have done exactly that – design, develop, and build the solution, then train the employees and move on. These projects are very rewarding and part of the key reason that I enjoy working as a 1099 worker.

In synopsis, working as a W2 employee vs. a 1099 worker can be drastically different depending on your personal values and associated preferred wage. I like to manage my own career, benefits, and time off versus having to follow a corporate template. Good luck in whatever YOU decide to do, but I hope this candid discussion has helped inform you of some of the considerations.

Community Speaking - Thoughts

2008-06-18T06:00:00.000-04:00

My buddy Andy Warren over at End2End put up an interesting blog about what drives people to speak at community events. I thought I would share my own story here.

The first time I spoke it was about increasing my own knowledge of the product. If you can teach it, you’re a master of it so the saying goes. Teaching has forced me to stay sharper and more on top of my game (less complacent) than I would otherwise be.
Andy surmises that some people speak to do de-facto networking. I will vouch for this – I have had many great discussions with people who’ve attended my sessions asking about future projects or my thoughts on a topic. He also talks about the couple guys who just love contributing to the community – and there are a few of them out there, with the key word being few. Some people just love giving something to others and speaking about technology is a form of community service, if you will.

I can say from personal experience the sheer amount of events in the state of FL is starting to become overwhelming. The code camp circuit has recently expanded to add Southwest Florida (now totaling 6), with 4 SQL Saturday events per year. Add to the local user groups who need speakers, and by the end of the year I will have spoken in Tampa twice, Jacksonville twice, Tallahassee twice, Orlando three times, Miami, and Naples. That’s a lot of money spent on travel but IMO it’s worth every penny. My primary concern about the high number of events is speaker fatigue – most of us are either working schleps in one way or another and have to give up what little free time we have to give back to the community. It’s worth it for now – but if there are another layer of events added I will have to pick and choose which ones provide the best forums for both myself and my audience.

Employee vs. Consulting, Part 2

2008-06-17T23:22:00.001-04:00

Let me begin by clarifying that W2 workers are employees and 1099 workers are contractors. 1099 workers are not employees for anyone as defined by the IRS, so I want to make that very clear in my previous post where I was too liberal in my use of the word employee.

My earlier synopsis was that most people would be better off working as 1099 contractors. There are numerous reasons, but the first is to make transparent the cost of benefits. The true cost of seeing the doctor when you sneeze twice in the morning is much higher than your $20 co-pay. Cash consumers also have a lot of leverage because they eliminate the middleman insurance company. Vacation time is very hard to value – some people don’t use it all and worse yet, some companies don’t allow their employees to use accumulated vacation time. For those economists out there, there is a labor/leisure scenario that goes like this: “As a 1099 I make $800 a day when I’m in the office, and if I want to take a day off work to go fishing I need to get at least $800 value out of my fishing, else I should work”. It’s pretty easy to put a valuation on leisure activities when you know your wage, but as a W2 employee it’s more like: “I get 2-weeks off a year so I’m going to be fishing at least 2 weeks this year and hopefully 10 more days when I’m supposed to home sick in bed.” It’s pretty clear to see that scenario 1 is more efficient for both parties as to the valuation of time and money.

There are also some rules that companies must abide by with 1099 workers that are not true for W2. For starters, 1099 workers set their own schedule. They are also given project milestones but not direction. They are sometimes provided with office equipment. W2 employees are told when they can/can’t work, are given training, direction, and specific instructions, and are provided everything they need to complete the job. The reason for these rules is that 1099 workers do not collect workers compensation insurance. One of my friends had a 1099 worker who didn’t pay his personal taxes and came back and claimed he should have been a W2 employee. My buddy was not impressed and it’s sad that some people reap the benefits and expect others to pay the piper. As a 1099, I prefer to use my own laptop during client visits because I have all the software I need to document and develop and I always have unfettered email and internet access via my Sprint card. At some client sites it’s difficult to carry on a phone conversation because of privacy issues and email is a confidential and quick substitute.

I also know people who have done the contract-to-hire routine. This routine is generally touted as a way for both the employer and employee to get to know each other before committing to a long term (W2) relationship. Unfortunately, I think it’s more biased to the employer. The employer can decide whether or not to extend an offer just as the worker can choose whether or not to accept it. But the worker is given a hard deadline of a time to convert or leave, and this is generally at the whim of the budget of the hiring manager. I’ve known friends who wanted to roll W2 but were waiting 8 months to start the project they were K-to-permed for, and didn’t feel comfortable rolling until the project started. That situation turned out to be a lose/lose for both parties because the company didn’t get a good worker and the worker lost a good long term relationship because the contracted ‘dating’ period wasn’t long enough to provide any benefits.

This post is becoming long enough for a part 3, so I will post more about this in a couple weeks when I get some time. Next I’ll explore the difficulty of finding 1099 work vs. W2 work, and where that trend is heading in the future. Stay tuned and happy data warehousing.

The Great Debate - Employee vs. Consulting

2008-06-12T20:09:00.001-04:00

Thought I would share an interesting dilemma I found myself dealing with the past couple of weeks. This is the consultant dilemma of whether to stay on as a consultant (or 1099 employee for commonality) vs. going directly and working with someone as a W2 employee.

First, there are numerous benefits to being a W2 employee. The true benefits are the ability to receive workman’s compensation in event of unemployment or injury, and the protection of having to deal with corporate HR departments if there is a personnel issue. Other ancillary benefits are generally medical insurance, vacation and sick time, and a retirement plan. The drawbacks of being a W2 employee are that it’s more difficult to move on to other projects quickly and generally some amount of pay is given up to provide the above benefits.

There are also benefits to being a 1099 employee. The major benefit is freedom. This freedom is being able to pack up and leave at a moments’ notice – which truthfully isn’t used that much if one wants to maintain satisfied clients. There are also other benefits like being able to manage your own career vs. following a corporate career path and receive much more flexibility with scheduling. The drawbacks are that you are responsible for providing all benefits and there is no nanny organization wanting to take care of you – i.e. companies do not have a vested interest in the well being of consultants as they do their own employees.

All that said, I’ve been studying and practicing economics long enough to know that there is a certain opportunity cost associated with being an employee. I did a little research into the origin of medical insurance being provided by corporations and it generally was a benefit provided after WW2 to attract the best employees in a brutally competitive market. Another offshoot was the good ‘ol pension plan which has almost universally disappeared from corporate offerings. Vacation and sick time provisions were also introduced around this time and now we have a model for the current workplace.

Is this model broken? I’m inclined to believe yes. Here’s the economic argument for it: Job seekers blindly look at salary and benefits to determine their offered compensation package. One thing is that it is very difficult to rank different packages side by side: if company A is offering a salary of $90,000 + $200 a month for health insurance + 2 weeks vacation + 2 weeks sick time per year, is that better than company B which is offering $93,000 +_ $300 a month for health insurance + 3 weeks vacation + 2 weeks sick time per year. Out of the door B looks better because of the salary difference ($3000-$1200 health care diff = $1800). However, what if company A would roll over vacation and pay out upon departure but B does not? Now the equation is different. Job seekers also don’t know that company A gives 4% raises per year while B gives 2%.

Everyone who has insurance knows full well that the costs have risen substantially in the past 10 years. A good portion of those costs can be attributed to the prevalence of insurance that subsidizes unnecessary procedures. An invention to fix this, the HSA account and high-deductible plan is designed to put more consumers in charge of their own insurance on the assumption that someone paying cash will do only necessary, value-oriented procedures. This has been proven time and time again that in a cash society the consumer is a winner.

Being a 1099 worker is not for the faint of heart – it’s easy to see steady, well paying work cancelled at the drop of a hat. A couple years ago I worked with a number of contractors who were brought in for a project, moved from out of state, and cut after 2 days on site due to cancellation of the project. The benefit is generally contractors are paid an hourly wage that includes the premiums of acceptance of this kind of risk and the loss of benefits. Someone who is smart can take good advantage of this situation to provide themselves with a higher level of salary and benefits at a much lower cost than what could be provided by a corporation.

Notwithstanding the obvious corporate goal of promoting employee stability and retention, I believe most people would be better off as 1099 employees. This would allow the following:

1. People are judged by their output and not padding the clock by showing up on Sunday to clean desk for 4 hours

2. You don’t get paid to socialize around the water cooler discussing last-night’s episode of Survivor (Reduction in office politics)

3. You are able to best manage your finances and benefits which in turns reduces the cost of providing insurance to all

4. Employers are best able to keep around high performers and lose those employees who do not perform

One major hurdle to this is recent changes from the government that more narrowly defines the W2/1099 world to keep employers from having contractors that are de-facto employees.

Obviously this is going to be a controversial post, but stay tuned for part 2……

Presentation Series

2008-06-07T11:22:00.001-04:00

For the past 2.5 years I’ve been traveling around the southeast US giving lectures on Data warehousing and ETL. The first presentation I did was “It’s all about the Data: Building a Data Warehousing using SQL Server 2005”. Since then I have done numerous variations of that presentation. In addition I have done various deep dives into ETL design, architecture, and development for local organizations. Thus, I have decided to organize my presentations into two distinct areas: It’s all about the Data and Data in a Nutshell.

The “It’s all about the Data” series will focus on general overviews of data warehousing and ETL. These sessions are less technical in nature and a lot of overview of the subject matter and some demonstration of data warehousing processes. In contrast, the “Data in a Nutshell” (or DINS) series will be technical deep dives will a lot less overview and more hands on action designing and building data structures. Some DINS talks will include DBA tasks such as permissions, replication, table design while others will strictly complement the “It’s all about the Data” series by doing technical deep dives into ETL architecture and design.

It will take a couple events to get everything lined out but I’m sure when it’s done there will be a clear delineation of the subject matter and I hope to build these two talks into must-see sessions at events around the country.

Thanks again for your support and I welcome any comments.

Windows Vista Update

2008-06-07T11:13:00.001-04:00

In a previous post I had written about my displeasure of working with Windows Vista on my old Dell laptop. After installing service pack 1 the computer has sped up noticeably. The batteries are so old that they don’t last more than 20 minutes per charge so SP1 didn’t make a difference there, but the boot up speed and hard drive access issues have noticeably improved. At this time I have shelved plans to go back down to Windows XP Pro. I’m also working with a Virtual PC image of Windows Server 2008 and I’ve been quite impressed with the performance and reliability, more to come on that.

Orlando Tweener

2008-06-07T11:10:00.001-04:00

I’ll be speaking at the Orlando Tweener event at the Orange County Convention Center in Orlando, FL. This event is making use of the TechEd facilities during the weekend to provide free education to the 1300 registered attendees.

My session is “Data in a Nutshell: Using SSIS to Solve Common Business Problems”. This session is the first in my Data in a Nutshell series (see other posting for an explanation) and will cover the SSIS tool package and use it to solve common scenarios that may be found throughout businesses.

I hope you join me!

Windows Vista.....Is it good for business?

2008-05-08T11:57:00.001-04:00

Hello again. Today’s topic is a discussion of Windows Vista. I use my trust Dell Inspiron 9300 for all my consulting work. It originally came with Windows XP Professional and worked pretty good, but in my experience Windows has a way of mucking up the file system so the computer drastically slows down over time. This has happened with every machine I’ve had from Windows 95 to present. Two solutions to this problem are not to install/remove software all the time, and do a fresh reformat about once a year.

So the Dell came with XP Pro and I moved it to Windows Vista Ultimate 32-bit in January 2007. I bought the Dell because it has a great video card for gaming (although in the 2.5 years I’ve had it I can count on one hand the number of times I’ve played games)….so the computer was ready for Vista. Running the Windows Experience Tool shows the following numbers:

The processor is the slowest part of the machine and the rest is A-ok.

The Dell has struggled a lot under Vista, especially when running SQL Server and BIDS. It’s also become slower booting up. An error a couple weeks ago lead me to do a complete wipe and reinstall of basic applications. However, I will still not pleased with the performance (and especially battery life). The biggest dealbreaker was that I have a difficult time being able to dock my computer to my 24” LCD at home. The video card crashes and then it takes a lot of reboots to get it running, but there is not an ‘official’ Nvidia driver for my card (since it came with XP).

Keith suggested that I give Service Pack 1 a try – see if it will help with my performance issues. I installed it without a problem yesterday and my first impression is that the computer is definitely much faster. SP1 supposedly addressed many performance and battery life issues so I’m going to give it a couple weeks to check. I also haven’t docked the system yet to see if that was affected.

Overall, I can certainly understand the reluctance of businesses to not deal with a Vista upgrade. MSFT continues to create a monster OS each time they release a new version. I have found Vista (and XP) to be very stable…..I am able to crash Vista if I remove my broadband card while it’s running – but otherwise it’s been stable. I also appreciate the extra work on security. I’m sure more people will begin to use Vista in corporate offices, but I’m going to wait and give Windows Server 2008 a try before I comment further. Stay tuned.

BBQ Quest

2008-05-03T09:58:00.001-04:00

Howdy all.

Amazingly enough, I am speaking at SQL Saturday – Jacksonville (#3) today. As usual, I rode up to Jacksonville with Joe Healy (Microsoft dude) and both of us being from the rural part of the country enjoy bbq. When I say BBQ, I don’t mean chicken smothered in BBQ sauce like you will find at your local Applebees (priced $12 for $2 worth of food + $1 of preparation expense). I’m talking good, southern BBQ – the kind that a local guy owns a smoker and cooks ribs, pork, or chicken all day in smoke at 250 degrees until the meat is so tenderized it falls off the bone. On our road trips in the past we have discovered some local gem BBQ places, generally on the old country roads between major cities. There’s good places we’ve discovered outside of Tampa, Lakeland, Suwanee, and Gainesville.

Interestingly enough, we planned our route to intersect a new BBQ join. We drove up I-75 to Ocala, 301 to Gainesville, SR 20 to Palatka, and SR 17 to Green Cove Springs outside of Jax. Ironically, we passed a grand total of two bbq places, of which exactly 0 were open. We arrived at Tommy’s BBQ just north of Ocala on 301 at 3:05 PM. They were open but stopped serving lunch at 3 PM, and the lady was eager to head on home, so we moved on up the road. Passing two Sonny’s BBQ later, we arrived in Jax without having succeeded on the mission. A BBQ Bills’ place north of Palatka was closed as well.

I’m not a huge Sonny’s fan – but it would have sufficed given the precarious situation, but Joe was hell bent on finding a real place or going hungry, so we arrived in Jax about 5 PM starved and disappointed. I planned the route home on different roads, hoping to intersect with a new, hidden gem of a BBQ restaurant.

Update from Busy World

2008-04-21T14:37:00.001-04:00

Hello folks. It's been a while so I thought I would update on what's been happening in my life related to data warehousing.

1. I'll be speaking at the Jacksonville SQL Saturday on May 3, 2008. This should be a really good event and a new one for north Florida. Durable Impact Consulting is also a sponsor of the event and is providing pens and goodie bags.

2. I've revamped my entire lineup of code camp presentations. Now that it's up to about 7 per year, I'm consolidating all of them into the "It's all about the Data" series. I've done this lineup at Miami and Orlando and it gets more popular every year. This year in Miami it was a standing room only which leads me to wonder why throw away a good thing.

3. SQL Saturday Tampa was a great success. I had promised to write a full review of the event, but I'm just too tired so I'll paraphrase. Logistics turned out to be a nightmare but the 220 attendees and 15 sponsors were all around pleased with the event. I'll probably not be running the event next year - instead I am planning to run smaller, quarterly events. The Day of Data was great, 41 attendees all had great remarks about the program. Look for a future event soon.

4. I'm not so happy with my HP laptop anymore. It's really fast but I've had some issues with the network connections and the battery is lucky to last 1.5 hours. In retrospect, I should have went with the Dell Latitude line and I'm strongly considering going this way and selling the HP. I want a computer that can go three hours on the battery. Plus the HP gets REALLY hot.

5. Thankfully my trusty Dell is still going strong, notwithstanding the dreaded "Cannot find bootloader" message I received after a failed Windows Update last week. I had to reformat the entire computer (and lose a few documents and pictures in the process) which took about 8 hours. I found the best solution is to set Windows Updates to manual, and backup full to external drive before each update. I hope this computer will last another couple years as I'm looking to buy another battery on Ebay.

6. Still haven't heard anything about the Tampa Code Camp, but I assume it will be held in July and I'm holding July 12 free for the event. Last I heard from Keith, they were having issues getting the venue straightened out.

7. There is a great program shaping up: the "All-Florida" Code camp to be held the weekend between the two weeks of Tech Ed (in Orlando). I won't write the date because it's not 100%, but if this event gets off the ground it will be awesome. Look for me to be a track leader at this event to provide another great opportunity for the community.

Thanks and stay tuned for some more posts.

SQL Saturday Jacksonville

2008-02-19T19:42:00.000-05:00

(Check back later this week for a full write up on the Tampa SQL Saturday)

SQL Saturday has moved on to Jacksonville on May 3, 2008. If you were at the Tampa or Orlando event, Jacksonville should be more good presentations and knowledge from a lot of the top speakers in the southeast United States. This is going to be another top notch event being driven by Andy Warren, Brian Knight, and co.

I plan on attending and speaking at this event. I always go in August for the Code camp, and this is another chance to have a great time in the fun city of Jacksonville. Hope to see you all there.

SQL Saturday Update

2008-02-13T19:30:00.000-05:00

Well folks, we’re just two days away from SQL Saturday – Tampa. Our venue is still the same (DoubleTree Suites – Westshore) and the DT and us are working hard to put together a good event. We do have one predicament I wanted to share to those who follow this blog. Arrive early.

One of the problems with free events is that people who have no intention of coming can sign up without recourse. This does two negative things for code camps: it causes us to have inaccurate counts for event goodies (including prizes, food, giveaways), and when registration cap is reached, it keeps people who want to come from signing up. This is a typical economist dilemma (on one hand, on the other hard)…….

We have over 380 people signed up for the SQL Saturday event and another 20+ on the waitlist. While experience has shown these kind of events to have about a 40% no-show rate, I made the decision to close off registration for a couple reasons. One, our event is set up to handle 200 attendees. We have lunches, seating, and prizes confirmed for about 200 people. We can handle more people, but at a certain rate we will bump up against fire code regulations and have to cut off attendance in sessions. If we don’t manage the crowd, our facility will and this isn’t good for us or attendees. We will be providing lunch tickets for the first 200 people to register and indicate they will be staying for lunch. Experience has shown us that people come and go during the day, so I expect that with 200 lunches we can handle maybe 240 attendees since some will leave early, some don’t want lunch, some want to go out, etc. There will be plenty of water and soft drinks for all starting at 11 AM.

We’re not having any keynote or closing session either. Prizes will be given out at the end of the day (3:45 PM). Just show up, go to sessions, spend a little time lounging around the pool if that’s your thing.

I’m going to give a little reasoning now why we chose to do our event at a hotel. This kind of venue poses unique budgetary constraints and would be a horrible choice had it not been for our great sponsors. Thankfully we were able to negotiate a rate that was comparable to the rates that other events have paid around the state and we’ll have a unique atmosphere. It’s going to be free, but the hope is that it will feel like a professional conference that people pay thousands to attend. I’ve attended the TDWI Conference in 2005 and our event will be very comparable on a micro level. Our venue is nice and professional, our volunteers are rehearsed and will present a uniform appearance. We were hoping to have speakers stay at the same hotel, but there was an issue with being able to do that so hopefully next time. We’re hosting the Day of Data on the Friday before, which allows a more specialized learning experience for those willing to pay a little extra.

To handle the ‘phantom registration’ problem, there are discussions going on around organizers about charging attendees a nominal fee $5 to $20 to hold a reservation. The money could be used to pay lunch, or more importantly, so people have ‘skin in the game’. Read my sister blog about economics to see what happens to a housing market when people have no ‘skin in the game’, but I digress. I’m not fond of charging to enter events, but I feel a small fee (closer to $5 than $20) is fair to hold registration and provide lunch. If everyone who attends SQL Saturday had paid just $10, we would have been able to provide a full day buffet for everyone. But it’s something to think about for the next year.

We’ll be paying close attention to speaker evaluations, but please give us some room to handle the crowding and food situation. Looking forward to seeing you at the events.

Wes

Visual Studio 2008 Launch

2008-01-29T14:01:00.001-05:00

Visual Studio 2008 is here finally! Early reports from the battleground have a lot of people excited about the reduced level of effort. One local company reported they thought that VS 2008 would reduce development time by 50% this year for their products. As an economist, that’s always a good return on the money.

I installed a copy last night at the Install party Florida.NET user group meeting. If you want to get a free copy of VS 2008 Professional, you have the chance all week to attend some events and install the software. There are events tonight, tomorrow, and Thursday where Joe Healy is providing copies to those who bring computers (Laptops or desktops with peripherals) to the Microsoft office in Tampa and install the software.

Check out Joe’s blog for more details: www.devfish.net

Happy hunting.

Day of Data

2008-01-14T14:51:00.000-05:00

Introducing my newest creation: a SQL Server training series called the ‘Day of Data’.

DoD is a one-day conference matching the top speakers on SQL Server to attendees willing to pay a small free to acquire knowledge and get some one-on-one time with presenters.

The idea for ‘Day of Data’ came out of my original idea of a SQL Saturday (data camp). We have the event on Saturday in a hotel. Speakers and attendees from out of town are staying at the hotel on Friday night, so why not have an event during the day Friday for people in the following two categories:

1. People who work M-F but can’t attend SQL Saturday because of time constraints

2. People who have training budget to spend on SQL Server training but contemplate the general lack of local training

3. People who want to attend the SQL Saturday and also get a little more in-depth knowledge than a one-hour session.

The first DoD is going to be held in the DoubleTree Westshore in Tampa. We have two great speakers lined up: the infamous Joe Celko and also SQL BI expert Rushabh Mehta. We’re hoping that we can get enough people to fill both of the rooms and possibly a third room with another speaker who is interested in joining for a trifecta.

The price of the event is $99. This amount merely covers expenses. We provide all attendees with a continental breakfast, coffee and drinks, a good plate lunch, and a snack in the afternoon. We think this event is a great value.

Check it out at www.dayofdata.com

The long range goal is to have these events at cities around the country.

New Year, New Computer

2008-01-14T12:20:00.001-05:00

Happy New Year everyone. I wanted to start off the new year with a story about a computer I purchased.

I sold both of my desktop computers last year and swore off buying any more, thinking I would want to stick primarily with laptops. I still agree with that assertion and have been watching the notebook design wars with interest. My current notebook, a Dell Inspiron 9300, has been a great computer. It has a 17” screen, is quite peppy, and still 2.5 years later, the Vista Ultimate ratings for the video blow the doors off the ratings of the best computer for sale in Best Buy (the Dell has the Nvidia GeForce 6800 dedicated card – I specifically purchased this computer for that card. Regardless, as great as the large screen has been for work and using the computer at home, it’s not quite as portable as I wanted, and I’ve been specifically contemplating a purchase of a 14” laptop to supplement it. To give you a breakdown of my computer replacement schedule, I buy a good machine and depreciate it over three years. Thus, the Dell will be replaced in December 2008. So it came to my thoughts that I should buy something small that would be compatible with any accessories I buy for the Dell replacement.

My specific requirements were:

1. Intel Core 2 Duo processor

2. 2 GB + Ram

3. 250 GB Hard drive

4. DVD Burner

5. Good battery life

6. Less than $1000

7. Light and thin

8. Ability to be carried without breaking my back in a backpack for long periods of time.

I began looking at computers during the holiday season. First stop: Dell. My last three computer purchases have been Dell and I’ve influenced quite a few more among family and friends. However, I strongly dislike the current design of Dell laptops. The Inspiron line is bland and bloated. My Inspiron 9300 looks a lot better than the 1721 that replaced it, even if the 9300 has an obvious miss in lack of a 9-key pad. The Dell XPS was a nice looking computer, but the 13.3” screen was a tad small for my liking. Thus, Dell was eliminated from consideration. That leaves Gateway, HP, and Toshiba as the only other computers I will consider. Gateway looks cheap, eliminated. Toshiba looks good but only offers integrated video on the 17” lappys, so it’s eliminated. Thus only HP is left. (Apple, for all its pluses, is out of my desired price range).

I bought an HP. I began to look at HP computers after buying and setting one up for a friend and I was impressed with the build quality and design. Yes, the HP computer looks great. The 14.1” laptop has integrated video but I don’t care because I’ll only play games on the big boy. The HP dv2XXX series met specifications 1-8 above and as a plus, included 3 GB of RAM. I could get the dv6000 series with 15.4” screens for less than a 14.1” lappy, but it’s too close in size to the Dell so I don’t really gain anything. I started looking in stores. Best Buy had a computer of my liking, but it had AMD Turion X2 64 and Nvidia. The windows experience score was 2.5, a horrible rating on the graphics card. Combined with lack of Bluetooth, Best Buy’s offerings were eliminated from my choices. I went to Circuit City. They had the perfect computer with all the specs I wanted, it had a WEI rating of 3.5. No problem. However, the computer was silver and white, big problem. I specifically wanted something that looked great and black/silver looks are preferred to white/silver, although the Dell is white/silver. So I checked around and played with the computer for a while to determine if I could live with the colors. I decided to buy it, and when I get the HP d9000 series 17” laptop at Christmas, I’ll make sure to get that one in black/silver.

I’m already pleased because I purchased a notebook dock that will work with the dv2000 and dv9000 series. Power adapters are also compatible. Sweet, I’m already happy. Now let’s hope that HP doesn’t change the dv9000 too much before I can purchase one. After a complete reformat of the computer to remove the bloatware, I’m real happy with everything and the way it runs.

SQL Saturday Update

2007-12-05T21:43:00.000-05:00

Happy Holidays!

Pam, Mike, and I are working very hard to put on a top notch SQL Saturday event in Tampa. We've had a couple of roadblocks but think we've worked them out.

First, the event will now be held on February 16, 2007. It was originally scheduled for 1/12, but the event venue neglected to mention a cheerleading tryouts that will be taking place that same day. We didn't want to compete with 300 screaming, pre-adolescent girls so we moved our event. Thought we had everything in line, and then the DoubleTree raised our rates for the day. That roadblock puts a serious dent in our budget, but I think we'll be OK. Yesterday I received a phone call from them stating that the hotel was almost full and we coulnd't get enough rooms for our party, so I was quite upset and had a good discussion with the manager there. Let's hope that everything else goes without a hitch.

The venue is really nice - if you come, you will have a good time. We're going to hold 5 concurrent sessions at 75 minutes each, giving more time for content. The day should be a relaxing day of knowledge attainment.

I'm also finalizing details for a Friday event put on by Durable Impact for SQL Server professionals. I'll have more to say about that soon - but I'm not ready to spill the beans.

Rest assured that SQL Saturday is going to be a great event - it's been a lot more work just getting it going than I thought it would be, but we have a dedicated team who will get the job done.

SQL Saturday is coming to Tampa

2007-11-22T19:52:00.000-05:00

Data professionals - this is the event you have been awaiting. SQL Saturday is being put on by Durable Impact Consulting on January 12, 2008. Our location is at the DoubleTree Hotel Tampa Airport (4500 West Cypress Blvd, Tampa, FL). We have a lot of great sponsors for this event, including Microsoft, KForce, Visual Gov, End2End Training, and others that will be listed later.

I'm the director of this event and there is a talented Board of Directors who are helping to put this event together. It's certainly not a one man show, and thanks to the sponsors, the event is free for attendees.

I'm really excited about this event - keep watching this blog, and each week, I plan to post a new surprise for the event.

Check out www.sqlsaturday.com for further information.

Happy thanksgiving.

Purpose of Parameters in ETL

2007-11-06T20:30:00.000-05:00

For those of you who don't know, I am now doing software consulting full time. I'll write another post about that, but today I experienced something I wanted to share.

One of my clients has ETL jobs created to populate a data warehouse. Well, for all intents and purposes it's technically a data store, but I digress. I was asked to take a look at some Data Stage ETL jobs that needed to be modified because the database user id had changed. WHAT? You mean people are actually hard coding critical values and passwords in ETL jobs? Why yes they are. I spoke with the ETL architect about the practice and friendly advised him that is a data warehouse worst practice. He was aware of the issue and but a decision was made back in the past to 'get it done now', but thankfully it was an easy sell to put all the user information in variables, but they still wanted variables hardcoded in the jobs, so the battle is only half-way won. I did get a committment to move to a dynamic configuration and I am putting finishing touches on my prototype, but the moral of the story is to not save your user id's and passwords hard coded in ETL jobs. It simply exponentially increases the maintenance costs at a later time.

SQL Saturday - This Weekend

2007-11-06T20:21:00.000-05:00

Howdy folks. It's been a couple weeks since we've last talked but I wanted to remind you that SQL Saturday - Orlando is coming up this Saturday. This is going to be a SQL data-lovin' fest featuring the best and brightest minds in the area presenting talks on data. Yip, you got that right I'll be speaking, but I don't claim to be the best or brightest, I am a sucker for speaking engagements.

I will be giving two sessions - the first, at 9 AM, is "Introduction to SQL Server Integration Services", the new session that I presented in Jacksonville and Tallahassee code camps this year. This session is going to be my staple session for the 2008 code camp season, which officially kicks off in February with South Florida Code Camp IV.

The second, monster session is a full 80 minutes of data warehouse loving discussion. This session is basically the same as I presented at the past two South Florida Code Camps - "It's all about the Data: Building a Data Warehouse using SQL Server 2005". (see my archives for a detailed overview of this session). As much as I think I have saturated the state with my data warehouse talks, I continue to draw a huge crowd each time I present this session, so you will see it in all it's glory finally presented in enough time to really be able to slow down and deep dive into the material. As a note, I hoped to run some SSRS reports off my data, but grad school and Durable Impact have keep me busy enough where I did not have time to complete that portion, so boo hoo.

I'll be at the pre-party on Friday night, but I probably won't stay for Saturday night's party. I'm dying to get some camping in during the cool weather so I'll be heading out immediately after my sessions.

Sneak peak - final disucssions are underway to hold a SQL Saturday - Tampa during the winter months. Hopefully there will be something to announce this week.

Take care and happy data-ing.

Interesting Meetings

2007-10-21T23:03:00.000-04:00

Hello again. I've been meaning to write some more articles, but between work and grad studies I don't have a whole lot of time for that. I would like to share the past couple weeks though because I've meet some very interesting people (in calendar order, oldest to present).

1. I stopped in at a gas station early Saturday morning for a Diet Coke. I found myself standing next to Florida Governor Charlie Crist. We had a brief discussion and he was very friendly and down to earth. I wanted to ask why they renewed the no-fault insurance laws, but being a computer guy, I hate getting asked about computer stuff outside of work, so I chose to give him the same courtesy.

2. Joe Healy invited me to attend ReMix07 in Boston, MA, held at the Hyatt Cambridge suites. As always, I chose to fly American Airlines and routed to Boston via Miami. On the flight from MIA-BOS (757), I received a first-class upgrade, where I sat next to a famous doctor from Harvard returning from a disease conference in South America. He was a really nice guy and we had a great conversation about life, economics, and the medical field. Luckily I was able to swap contact information with him and I'm hoping we'll keep in touch. I feel very lucky to be able to meet interesting people.

3. I was invited to a student luncheon with USF President Judy Genshaft. This luncheon was 6 graduate students who held a roundtable with a few deans and Ms. Genshaft. I learned a lot about the university and found her to be a fascinating person to converse with. Interestingly enough, at my undergraduate alma mater, Missouri State University, I was invited to and attended a dinner with then president John Keiser.

4. To cap off a wonderful couple week span, I was returning from San Juan, flying AA (of course) in business class, and I sat next to Alberto. Alberto is a very interesting guy who lives on a Caribbean island who I met a couple months ago when we shared a row in biz class on the same flight. Two months later, same row, same flight, same seat mate. How often does that happen? I hope to get that experience once again.

I promise my next post will be data related, but I couldn't help but share some highlights from the past two very interesting weeks.

Code Camp Sessions

2007-09-17T23:35:00.000-04:00

My first code camp presentation was South Florida Code Camp II in Feb. 2006 in Miami. It was a couple firsts - first time to present, first time in Miami. All in all it was a great experience and the topic was "It's all about the data - building a Data Warehouse using SQL Server 2005". Much to my surprise, there were 65 attendees. Any doubt I had as to the reach of this topic was immediately put to rest as my nerves began to over-synapse standing in front of that crowd. I took the same session and modified it with different variants for the Tampa, Tallahassee, and Jacksonville code camps of 2006. Attendance at each was quite decent and strong in proportion to the total code camp attendees and other sessions. At one code camp, I had 20% of the attendees in my session.

One of the general rules of code camps is to come up with new sessions each year. Due to some time constraints, I was forced to slightly modify and present an updated verison of "It's all about the Data" at South Florida Code Camp III in February 2007. I expected to see a couple of the same people, but was thinking the topic is getting tired. Much to my surprise, there were over 85 people in the room, people lined up against the walls, sitting on the floor, and quite a few admitted to having attended the same session in 2006. Considering total attendance at the event was about 400, I pulled in close to 25% of the people for that one session. The presentation was significantly modified for Orlando Code Camp to good success. A different modified version was used for Tampa again with mixed success, about the same amount of people in 2007 as 2006.

One of the concerns I have is about satiation. The topic of data warehousing is so broad that it's impossible to cover any substantial part in the typical 60-70 minute presentation. Thus I've had to choose to cover the basics such as the definition of a data warehouse, what kind of tables are used, simple ETL jobs, combined with some kind of code demonstration, which has ranged from advanced ETL to a homegrown frequent flyer EDW I created just for this purpose. Back to satiation - I don't want to talk about the same topic so many times that people have already heard the talk, but at the same time I don't want to deprive those who are attending the events for the first time the opportunity to learn it. For Jacksonville 2007, I decided to decouple from the data warehouse topic and talk more about the SSIS Tool and what it can be used for...this is a direct modification of a presentation I gave in September 2006 to the Tampa SQL User group. Quite frankly, I was disappointed with the overall assessment of the presentation from attendees (about 7/10 in the ranking) and will be looking to tweak the presentation further for the 2008 season.

In short - if there is anything you liked in my presentations and something you didn't like, please leave a comment and let me know. My session in Tallahassee will be a hybrid of the two sessions - a deep dive into SSIS with an emphasis on Data warehousing. I hope to use the results from there and the two sessions I will give at Orlando SQL Saturday in November to develop a unified session for all code camps during next years season.

Code Camp Season Update

2007-09-17T23:19:00.000-04:00

My oh my how time flies. I received an email today from someone who hasn't seen any blog updates from me for a while (since 22 May to be exact), so about 4 months later I'll catch you up on what I've been doing.

Had a number of speaking engagments in the past few months. In June I spoke at the Tampa SQL/BI User Group meeting, doing a deep dive presentation into architecting ETL jobs. Late in June I spoke at the inagural meeting of the Sarasota (FL) SQL Server User Group. The topic there was ETL and Data Warehousing with SQL Server 2005. Had a good turnout for both events and they were a break from the typical entry-level talks I give at the local code camps. Also in June I was provided a pass to TechEd, held in Orlando. It was very informative and I was able to suck in a lot of knowledge about SQL Server, data warehousing, and implementing best practices. The week of Tech Ed was quite enjoyable - I stayed with Keith at the Disney resort, and the parties I attended that week warrant a separate post. Let's just say that it was a great week all around, capped off by a visit to the Islands of Adventure theme park on a rainy Thursday evening.

July brought the return of Tampa Code Camp III. I was on the board this year and my responsibility was the speaker and post-event party. I was able to round up a significant chunk of change from supporters and we had a great party at the Courtside Grille. Around 50 people came to the speaker party and 120 attended the post-event party held in the same location. I may have found my calling in this group, and Jeffrey Palermo, I think you're a great guy, but I'm going to considering throwing my own party at TechEd the MVP Summit. My session at the Code Camp was Data Warehousing with SQL Server. It was a good session with about 36 attendees.

August quickly came upon us again along with my second annual Jacksonville Code Camp. In 2006, the JAX trip was a lot of fun and kind of a cap to a wonderful summer (search for Jacksonville survival report on live.com and the story should come up from devfish). It was more of the same in 2007 except it seemed to go by quickly. The four amigos (Joe, Keith, Stan, and myself) carpooled up 301 and arrived in time to secure a nice room at the Omni. Those guys in JAX put on a great show and I debuted with a new presentation: Introduction to SQL Server Integration Services. Attendance was less than I expected, but I was up against some strong presenters at 9 AM. The good thing about going early is that I was done for the day and had time to attend some other sessions, plus for the first time, I had my entire presentation finished a good week before the code camp. The highlight of this trip was the steak dinner Keith and I enjoyed at the Ruth's Chris on St. Johns' River.

September brings a continutation of graduate school which is keeping me very busy. However, I continue to squeeze time in for code camps and will be presenting at Tallahassee Code Camp III this weekend. To give you a little background, the first code camp I traveled to attend was in Tallahassee back in 2005. I drove up on a Friday evening, the weekend that Rita decimated the Texas coast. This was right after Katrina and I was concerned about being able to secure enough gasoline for the drive up and back. I told myself that I would fill up my car as soon as I arrived in TLH so I would have enough gas to get home, and sure enough I rolled into town about midnight only to find half of the gas stations closed, and many others were already out of gas. I began to sweat until I found one open with gas and topped off the 'Stang. The Code camp itself was relaxing and uneventful and I had a great time getting to know people at the Paradise Bar and Grill that evening. Another bonus was on the drive home I stopped at the St. Marks wildlife preserve and took all the country roads back.

Which brings us to today. As I previously said, graduate school is keeping me very busy. If I don't post on here too often, just send me a gentle reminder that I have a duty to write. I'm working with Jonas right now on possibly attending the first South American' code camp in Argentina in October. Stay tuned for updates on that exciting opportunity.

Tampa Business Intelligence Group - Revision

2007-05-22T22:21:00.001-04:00

The Tampa SQL BI Group meeting has been moved to June 11 to accomodate those going to TechEd. I'll see you on the 11th.

Konesans’ File Watcher Task in SSIS

2007-05-21T21:33:00.000-04:00

Last week I was asked to help a friend redesign some of his ETL processes. It was a great exercise and quite fun knowing that we had to have it completed in the same day due to some time constraints from his customers.

File watchers are programs that ‘listen’ for a file to be placed in a directory. I’ve worked with file watchers written in C, PERL, and manual kick-off processes. Thus there are many ways to do a file watcher programmatically, and today we are going to look at how it is done in SSIS.
The need was to create a file watcher that waited for two files to arrive, then process them in a loop. It looked like this.

The file watcher was created by Konesans – they have good writeup on their webpage about the properties so check it out for information. This post merely shows the product in a production environment.

The task has the following properties:

I’ll explain key variables.

1. Name = name of the task.

2. Description = put a meaningful description here (‘watches for the files’ is not meaning and states the obvious).
Filter = *.* looks for every file. *.induct would only pick up files like TEST.induct.
FindExistingFiles = true if there may be files in the directory already. If False, then it ignores existing files.
Output Variable name = the variable name we assign in the variable box below.

Then I created a Package level variable called User::FileName to store the file name. You must also create a connection manager to list the directory the file will watch for (or create a new one).

The file number variable was used for the looping mechanism – to loop until two files were loaded. Once the file was found, the filename was written into the variable FileName and available for access further down the chain.

Here’s a screen shot of the process running. It looped twice, running the data flow tasks below each time. This task inserted the file name into an audit table. Once the loop was finished, the last file inducted was processed.

Now you should have a better understanding of the excellent Konesas’ file watcher task and a real-world application of this tool.

Differences in ETL Tools

2007-05-18T00:47:00.000-04:00

Here's an interesting topic, today let's look at some of the key differences between the IBM Websphere DataStage ETL product and the Microsoft SQL Server Integration Services suite. This topic has been written in my notebook for a while but an email I received from a code camp attendee pushed me to finally publish it.

First, DataStage (DS) has been around longer and traditionally has been more accepted within the corporate world. DS is now on version 7.5 with version 8 to be released in the fall. DS 8 will be the first all new version for quite a while and there are some excellent, much needed enhancements in the product.

SSIS got it's start as DTS (Data Transformation Services). DTS is a decent tool but not on the scale of DataStage, and the SSIS product certainly makes it a viable, and in some cases, better competitor to DataStage.

Main differences in the product out of the box is that DS has connectors to many different source systems, while SSIS is limited in the provided connectivity, although that is going to change shortly. DS is a powerful tool that provides for very simple ETL building jobs, utilizing Basic code as the supporting language. SSIS is a very powerful tool with many "stages" and options available, but in some cases I've found it harder than I'd like to create simple jobs. SSIS uses Visual Basic for the supporting language, and has great support for scripts. DS has hash files as high performance data stores, and SSIS has raw files. Both products have bulk loader capabilities, and both products are designed for high performance systems. One significant difference is that, not surprisingly, DataStage works with both Unix and Windows OS while SSIS is only available on a Windows platform.

The primary difference is cost, with DS costing 4-5x as much as SSIS for some applications. This is a key reason I believe that SSIS is going to be the future of ETL tools. Not only does it do a majority of the same functionality as DS, it costs much less. With the growing acceptance of ETL tools as business process drivers, look for SSIS to expand the market for ETL and business intelligence tools.

In short, while both DS and SSIS are very powerful, SSIS's cost advantage makes up for some of the weaknesses of the product.

Systems Analysis and Design

2007-05-17T20:18:00.000-04:00

Most people who are following my blog know that I have an affinity for economics and am finishing up a masters degree in Economics as we speak. One common question I receive is "how does economics related to software engineering?" Well, that's a pretty easy for me. Let me explain.

Our entire capatalist system is built on economics; the buying and selling of good, creating efficient markets and mediums of exchanges. Software is very similar. The goal is to create an efficient program that uses hardware to accomplish a stated goal. Economics comes to play in software: I get to use economics on a daily basis to prioritize my workload. If it's more efficient to use my time tracking down a bug that is causing an overstatement of sales, that is how I will use my time (providing immediate payoff) instead of working on a development project with a 6-month payoff. Another example is using rules to cleanse data. In our data warehouse, we often get dirty data in and checking every column of every row is a hugely intensive process, adding cumulatively hours to our daily run times. Using economics, I've identified some rows that have marginal value to our processing on which we should not run the cleaning, shaving some time off (in the aggregate).

Now that I have went completely off target, my elective class this summer is "Systems Analysis and Design". This class looks at building processes, data models, use cases, normalizing data, etc. All the good stuff that I do professionally today. That's the good news - I can test my brain, add something to the class discussion, and hopefully learn a lot new in the process. Intrestingly enough, I took a similar undergraduate class about 10 years ago, and we used version 3 of the same book that the class is using now. Good thing I kept the old one; I have found it to be quite useful in the past couple years.

I'm going to post regular updates on my class on here and share some knowledge in the process.

Links on Left

2007-05-16T20:50:00.000-04:00

I've edited my links on the left of the website to direct to local user groups and SQL resources.

In addition, I've added the links to four companies which deserve special recognition due to my positive experiences with them.

First, American Airlines. Their line is "We know why you fly"(SM). Why yes AA, you certainly do. Thank you for providing such excellent service on the flights I've been on in the past years and I look forward to many more years as an AA Platinum member. One day I hope to be a big shot CEO and you can bet my company will support AA if I have any say in the matter.

Sprint PCS - I've had Sprint phone service for 6 cumulative years. There was a dark 18 months where I used inferior service from another cell company, but the excellent coverage and pricing of Sprint plans fits right up my alley. The PPC-6700 is an excellent phone.

Dell computer - both of my computer are Dells and I've not had any problems in 5 years, 'nuff said. Amazon, good service and tech books at reasonable prices.

Hopefully I won't receive any hate mail requesting removal from my site, and I recommend everyone who reads my blog support the companies I have listed.

Introduction to Data Warehousing

2007-05-16T20:14:00.000-04:00

As some will recall, I started a 8-part series titled "Introduction to Data Warehousing" last summer on JumpStartTV.com. Based on comments I've found using Google and feedback from attendees, I'm putting my presentation from Orlando Code Camp on JumpStartTV. This presentation will be complete material overview minus the actual code demonstration.

I completed part 3 & 4 of the aforementioned series tonight so look for them to be up on JumpStart shortly.

At the Tampa code camp, I hope to do a webcast of my session and a podcast for people
to access remotely at their leisure. I also have some surprises in store, so stay tuned.

Tampa Code Camp Correction

2007-05-15T20:09:00.000-04:00

Keith called today and told me that the Tampa Code Camp has been moved to St. Petersburg College's EPIC Business Center on Ulmerton Road in Largo. A new venue, but it's more central to the area than USF, plus it's closer to home for me so I'm not complaining. I'm going to set up the post party at my local hangout so it should be a lot of fun. Hope to see you there.

Tampa Bay SQL BI User Group

2007-05-14T20:05:00.000-04:00

I am speaking at the June 4th meeting of the Tampa Bay SQL BI User Group.

Topic Name: "Designing ETL Jobs: A Generalized Lesson in ETL Architecture"

Lesson Description: Learn all about using ETL to populate a data warehouse, some of the pitfalls of designing ETL jobs, and about how to design ETL jobs for maximum performance and maintainability.

During this session we will create ETL jobs in SSIS that meet business requirements and follow a standardized methodology. It should be an interesting and informative session.

For more information, see the group website : http://tampasqlbi.sqlservercentral.com/

Vacations

2007-05-08T22:08:00.000-04:00

An interesting note - I am writing this post from my tent at a Florida state park where I have come to enjoy a couple days of solitude (yeah right) and catch up on some reading and minor work tasks. Seems to be the only place I can get anything done. Hopefully I'll get some kayaking and bass fishing in as well.

Earlier in the week I was in Puerto Rico, and I can tell you it is a great place to visit. There is nothing quite as relaxing as sitting under a palm tree on the beach listening to the waves. Sure I could do that in Florida, but then I feel the pressing need to be doing things at home, visiting people, so I decided to take an island getaway and brush up on my Spanish. It was absolutely thrilling, touring around San Juan, old San Juan, Condado beach, Isla Verde, and all the other sights that PR has to offer. I'm looking forward to a return trip to enjoy some scuba diving and a rain forest tour.

Here's a picture I took in Old San Juan of the forts.

Upgrading IBM Websphere DataStage

2007-05-08T21:53:00.000-04:00

It's official - I have converted my employers' data warehouse to use IBM Websphere DataStage 7.5.2; we were running DataStage 7.1 previously. We use DataStage (DS) as our ETL tool, processing over 20 million records daily in a multi-terabyte data warehouse.

I'm proud to say that I led the project, presenting status reports to the CIO and vice-presidents, communcating between the groups, and actually directing the install and doing a majority of the testing.

The upgrade process ran pretty smoothly, all said and done. There were some issues encountered so I'll go ahead and list them here in case someone else has the same issues and somehow finds this post.

First, we are running Sun Solaris 5.8 on our current production server. This is a 7 year old 22-CPU machine. Needless to say, it's pretty slow. Our test environment was our brand-new 8 dual-core CPU (at 2 GHZ) Sun Solaris 10 machine. Therefore, we have substantially greater performance on our development machine than we do on our production machine - for now. This kind of performance disparity makes it diffucult to test because DataStage has many tunable parameters and while I wanted to mimic the prod enviornment, it would almost be criminal to dumb down our development machine to that level of DS performance.

DS has a file (uvconfig) that contains the specific settings for the application. Many variables in this file (NFILES, MFILES, etc) need to be set up to allow a high level of performance, utilizing the machine and software to it's full potential. Verify that the NOFILES (number of concurrent open files) parameter in the operating system is set high (at least 2048). Then set NFILES and TFILES parameters at least 100 or 130 in the uvconfig file. Make sure the DSENV file sources correctly and populates the library variables correctly.

In the end, our DataStage upgrade went smoothly because of good coordination, testing, and documenting all problems encountered during testing helped us a great deal when performing the actual install.

Now I'm looking forward to moving to DataStage 8 (Hawk) next year - the 8 release looks to have some great toolsets and improvements that will go a long ways toward ensuring data quality and funcationality of ETL processes.

Tampa Code Camp

2007-05-01T21:48:00.000-04:00

Save the date, the Third-annual Tampa Code Camp (www.tampacodecamp.com) will be held on July 14, 2o07 at the Business College at University of South Florida.

I plan on doing two sessions this year: first will be the usual presenation on data warehousing, and the second will be an advanced data topics class, where we will talk about ETL design and architecture, database performance, tuning a data warehouse, and data modeling. As usual, the tool of choice will be SQL Server 2005.

This year I am going to videotape the presentation and post it on www.jumpstarttv.com as a training session, and we'll also be making some podcasts during the event for those who aren't able to attend.

Let me know if there is anything you would like to see covered specifically and I'll see if I can work it into the presentation. See you at the code camp.

What is Master Data Management (MDM)

2007-04-20T22:28:00.000-04:00

Master Data Management is a fairly new concept that has gained a lot of traction in the past couple years. MDM, as it's better known, is a holistic way of looking at data as it flows throughout an enterprise. Case in point, when we build data warehouses, we receive data from source systems and turn it into information that the business can use; painting a picture if you will for the business to see.

The data received has to come from somewhere and this is where MDM comes in. Source systems have data models, entity-relationship diagrams, cardinality models, etc. We know that each order has one customer, but one customer can have many orders (one-to-many relationship). We also know that each order can have many products. A product may or may not have a variant (a NFL Jersey has 30 variants - 1 variant for each team, and each team has many variants - a jersey for each player). So it's easy to see how this could become complicated quickly.

MDM is a methodology for watching data flow through the business, similar to a master data model. The source system contains 'orders' which them flow into the data warehouse as 'orders', and then this 'order' information is placed in reports and dashboards for measurement by the business.

Now that I've repeated myself on MDM, let's compare MDM and ETL. This actually is the reason for this post - there was a question posted on the MSDN Forums http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1426964&SiteID=1 by a lady who was curious why ETL and MDM were not the same. In short, MDM is enterprise wide while ETL is specifically related to the data warehouse. Here is my answer:

"MDM is the overall view of managing data within an enterprise, like an umbrella over all the sources of data within a company. ETL is the process uses to support the MDM objectives for business intelligence purposes. With MDM technologies, ETL will become much simpler. MDM is a fairly young in the broader world of business intelligence and will continue to grow as time progresses."

Hopefully now you have a better understanding of the similarities and differences between ETL and MDM.

Training Courses

2007-04-03T00:43:00.000-04:00

I've decided to put together a training curriculum on ETL using the SSIS tool along with a similar session using IBM Websphere DataStage. I'm thinking of a 7 hour training session to be held in a local venue that will cover data warehouse concepts, best practices, and an actual hands-on demonstration/workshop for all attendees to model data and build sample ETL jobstreams. Many people in Florida have now seen my Datawarehouse presentations at Code Camps, and this class will be a logical, in-depth extension. If the cost was reasonable and the content interests you, please email me if you be interested in attending such an event.

It takes a long time to put together quality training materials, so don't expect it in a couple weeks. Think more along June/July timeframe.

Orlando Code Camp 2007

2007-03-19T14:21:00.000-04:00

Orlando Code Camp

Looks like the damn data man will make it to the Orlando code camp this year. It’s March 24 at Seminole Community College. Website is www.orlandocodecamp.com.

My session is titled “Datawarehousing with Sql Server 2005 Presented by the Damn Data man”. This session will be similar in content to the South Florida code camp session as this week I am way too busy to even think about rewriting the session. There will be some tweaks and enhancements, but we will cover basic data warehousing concepts and look at the Kimball Group’s sample data warehouse as an example of how to build a EDW using SSIS.

The official description is as follows: “Description ~ ever wondered what makes a datawarehouse? Ever heard of ETL? You are in the right place because the damn data man is going to teach you everthing you need to know to get started with your own datawarehouse project. We will use sql server 2005 as the tool for completing our project on time and within budget.” Hope to see you there.

MVP Summit Overview

2007-03-18T23:45:00.000-04:00

Thanks to Joe Healy, I was provided a Special guest pass for the 2007 Microsoft MVP Summit. I am not yet a MVP (I hope it happens soon!!!!) but it was great to go to Seattle for 6 days and learn all about Microsoft, upcoming products, and learn more about the future of SQL Server.

I flew up on American airlines from Tampa-DFW-Seattle. I am starting to sound like an AA spokesman, but I LOVE AA and the fact that I was able to secure a first class upgrade for the trip (being AAdvantage Platinum) helps. We flew a MD80 from TPA-DFW and one of the ex-TWA 757-231 jets from DFW-SEA. It was good to fly on the ex-TWA ship (N705TW) as it will soon be flying for Delta. Then the only TWA birds left in AA's fleet will be 103 MD80s, but that's another topic. The flight home was not as good - my upgrade was not available from SEA-DFW (ex TWA MD80) and in disgust I chose not to upgrade from DFW-TPA. Both the flights were good and the weather was beautiful - I actually purchased an Asian chicken salad from buy on board food and it was pretty tasty. I would have preferred the hot roast beef sandwich in first class but I digress....

Arriving in Seattle I rode a taxi downtown to the Westin. Microsoft did great - they put us in a nice hotel. Two nights I roomed with Keith Kabza - the remainder of the nights I spent solo on the 34th floor of the south tower overlooking the port of Seattle. The views were wonderful.

Sunday Keith and I walked around downtown. We watched the NCAA selection show from Fox sports grille where my team (Missouri State) was once again denied a berth so lesser teams could get beat out of the conference in the first round (Duke???? j/k). Stan, Keith, and I went out for a great steak dinner at Ruth's Chris to discuss building our brands and businesses. Jeffrey Palermo (www.jeffreypalermo) hosted a Sunday night party at Jillians Billiard club. It was at that party when I began to recognize the importance of the summit to my professional career (networking opps). Thanks Jeff it was great to meet you.

Monday started out meeting Joe Healy, Stan Schultes, Keith Kabza, Jim Zimmerman, and one of Joe's buddies for salmon soup in a local fish restaurant by the famous Pike's Fish Market. Have you seen that video FISH! about teamwork? Well I was slightly disappointed to find the actual fish market was kind of boring. I much preferred watching the donut guy fry donuts and catch them in his sack without looking. We then went to the MVP Expo at the convention center. I spent the afternoon looking at MSFT products and meeting with MVPs and fellow community leaders. After having a community happy hour, we all met up at the Americas regional dinner at the Westin. The dinner was fun, especially all the photos. Afterwards, Keith and I played a tough round of darts with Brian Knight (www.brianknight.com) and I socialized with Valeria, a very nice SQL Server 2005 MVP from Paraguay. The evening ended with a great party at Gameworks.

Tuesday was the start of sessions. Bill Gates did the keynote presentation first and had a q& a session. Of course he was very smart and well spoken, and he looked just like the nice guy you see on tv. Matter of fact, I cannot believe anyone would want to throw a pie at him (see Amsterdam 2000). Because of the NDA, I am limited in what I can say, but I will summarize in the following points. Future games for XBOX (Halo 3 especially) look WONDERFUL. I will be buying an XBOX 360 soon. Windows Mobile 6 is coming out and I wish I could upgrade my wonderful Pocket PC phone to it. SQL Server 2008 will have some great features too. The evening was spent at the Museum of Flight, looking at airplanes and enjoying cocktails and great food with many different people.

Wednesday we arose early and were bussed to the Microsoft corporate campus (MSHQ) in Redmond. I was kind of surprised at the campus - it was nice and clean, but not as grand as I expected. I guess I expected there to be super cushy offices with 24" LCDs and digital projectors everywhere, but sometimes I don't live in reality and it would be hard to find something to complain about at MSHQ. The conference center was very nice and we had 5 breakout sessions covering databases. Again, I cannot discuss what I learned, but let's just say that SQL Server 2008 will build on an already successful 2005. the part that most interests me, SSIS (SQL Server Integration Services), will have some new features as well. I was pleased to meet the product teams and was able to ask one of the original SSIS product architects why some features I consider imporant were missing (and got reasonable answers - all related to time and keeping within scope). We had a product dinner in the corporate cafeteria that night where I spent an hour talking to Doug and Mike from the product team (both great guys). After I got back downtown, a bunch of us SQL guys met at the Elephant and Castle bar to discuss the days events over drinks.

Thursday was the last day of the conference. Went to a couple breakout sessions and a closing luncheon. After the lunch, I was escorted to the SQL Server 2005 development building where I was able to meet directly with the product teams. It actually felt like I was back at work - we sat in a conference room and watched slides, demos, and had a q and a session. The MSFT developers are great guys and I enjoyed talking with all of them. Headed back downtown afterwards and met up with the Florida group again for a happy hour and evening entertainment to rehash the conference, what we learned, and make plans for Tech Ed 2007 and next year.

There you have it - my detailed overview of the MVP Summit. It was great and next year I hope I am back as a full MVP. I am so excited about the future with SQL Server 2005 and SSIS, and I am thankful of the decision I made in 2005 to get involved with the local community here in Florida to promote these great technologies. Now if I can only convince my boss to let me off to go to Tech Ed 07 in Orlando.....

South Florida Code Camp

2007-02-18T23:30:00.000-05:00

Welcome to 2007. It's been a while, but I am already on the code camp circuit for the year. My first presentation was "DataWarehousing with SQL Server 2005".

I'm constantly trying to tweak my presenations and a new year equals a new start. One of my concerns is that data warehousing is a HUGE topic and there is no way I can accurately cover everything in a 50 minute timeline. For the 2006 Tampa Code camp, I tried to split data warehosuing into two sessions, beginning and advanced. In my opinion, it was not successful as I had 40 for beginning and 4 for advanced. That could either tell me that the first session was so good no one needed to go to the second, or that the first session scared the hell out of everyone.

For Miami, I gave a condensed overview of DW concepts and launched into the demonstation system courtesy of the Ralph Kimball Group (www.ralphkimball.com) using the demo system built in the Microsoft Data Warehouse Toolkit. (www.msftdwtoolkit.com) Kimball Group has a great line of books covering data warehouses and I thought the sample system would be a great demonstation of datawarehousing. An added bonus is that the demo system is downloadable straight from the website so participants can play with it immediately.

The session went well. There were at least 80 and probably 100 people in the room at the start. I was completely blown away by the interest in the topic. Sadly, I forgot to hand out eval forms so I have only received sparse feedback on the session but some were dissapointed that I only demo a system, not build it during the demo. Trust me, I wish I had time to build a working DW from scratch but it's just not possible during 50 minue presentation. If we ever run a data camp in 2007 (something Brian Knight and I have discussed), I may have a full day track on building a DW starting with the specs and ending the day with a fully functioning system. Not a bad idea.....

So, as usual my slides are not posted online but you can contact me at thedamndata at yahoo dot com if you want to get a copy of them.

Stay tuned for Orlando.

2006 – A Year of the Damn Data

2006-12-28T23:50:00.000-05:00

It’s been almost exactly a year since I started “The Damn Data” (TDD). I have a great memory and love to reminisce about good times in the past so I’ve decided to put a data year-in-review summary below and list my professional resolutions at the end.

January 2006 – Come up with idea for “The Damn Data” website while sitting in a boring meeting at work arguing about data quality, thinking that the problem ‘is just the damn data’. Goal was (is?) to develop a brand about data of all types stored in all types of structures and manipulated by all types of tools.

February 3 – Present first ever session at South Florida Code Camp titled “It’s all about the Data: Building a Data Warehouse using SQL Server 2005”. I’ll never admit it but I was kind of nervous about the appeal of the topic but the 65 people in attendance convinced me that I was onto something. The post party at Tarpon Bend was quite enjoyable, I especially enjoyed talking with the pretty ladies who are also techies, and I think they were from Ft. Lauderdale. Oh well.

March – Much to the angst of Joe Healy, I skipped the Orlando code camp to watch my beloved St. Louis Cardinals play three spring training games in Jupiter, unknowingly watching the precursor to Game 7 of the NLCS (Cardinals @ Mets).

June – Roadtripped in the ‘shaggin’ wagon’ (since sold God bless it) with Healy and Tom Fuller to the Tallahassee Code Camp. Presented “Data Warehousing Essentials with SQL Server 2005” with 14 people in attendance. Slightly disappointed in the post-party at Paradise Bar and Grill due to lack of college students in town during the summer. Strongly suggested to Keith Rowe that TLH Code camp be moved back to September (which I’m told it will be). The best part was seeing city-boy AKA Fuller in his Buffalo Bills visor eating at “Hog Heaven” bbq on the side of U.S. 19, a place where only a redneck would feel at home (like Healy and I are at heart).

July – Wasted a good weekend at the Homewood Suites in Tallahassee working on a contract project and arguing with a batch load guy about the appropriate way to load data into tables. Even worse was the lack of a good bar in close proximity of the hotel. Jumped out of a perfectly good airplane with a fellow techie (Laurie) against my own judgement, but it was well worth it. Attended the Tampa code camp and presented two sessions “Data for Dummies: Parts 1 & 2”. First session went good, about 32 people in the room, some good questions, but a problem with my SSIS demo. DOH!!!!! I must have scared a bunch of people away because only 4 came to the second session. Enjoyed the candid feedback received from the evaluation forms, some of which said “speak more slowly, covered material too fast” and “speak faster, spent too much time on basic material”. MORAL – can’t please ‘em all.

August – Attended the Tampa Bay Technology Forum Summer tech jam (www.tbtf.org) at Channelside. It was a lot of fun and I got a chance to meet up with a lot of recruiters (many of them were very pretty and you know who you are, and I’ll still take you up on that dinner with your friend, Michelle). I also socialized with old coworkers and friends who I hadn’t seen in a long time. At the end of the month, I roadtripped with Healy, Kabza, Fuller, Schultes, and Zimmy to Jacksonville for the code camp (thankfully I did not drive this time). Shared a room with Kabza and Zimmy at the Hyatt on the St. Johns River. JAX is a beautiful town and I highly recommend it. Drank a couple with the guys at the Landing and ended up presenting “Data Warehousing Essentials with SQL Server 2005” as a fill-in in the afternoon with a pretty good hangover. Received an eval comment that said “too relaxed” contrasting with a comment saying “slow down and relax”. Went to the Bucs @ Jags preseason game with Kabza, riding the water taxi to the game; it was fun and worthwhile and one of the highlights of my summer.

September – Presented “Introduction to SQL Server Integration Services” at the Tampa SQL user group meeting with about 16 in attendance. The presentation went well and I enjoyed socializing with the group at Courtside Grille afterwards. Spent too much time chasing women and socializing to think a lot about data…….

October – Didn’t do a damn thing related to data, but I did hop on American Airlines and found myself at Game 5 of the World Series, watching the St. Louis Cardinals decimate the Detriot Tigers for the championship. I thought about data warehousing after noticing that it took 3 days for my AAdvantage miles to post to my account after my trip. In contrast, Delta posts the miles to my account by the time I get home from the airport. Could this be the difference between a batch load and a trickle feed (right time) data warehouse? Interesting…. Oh, and thanks to Kabza and Schultes for joining me at the Green Iguana (after Russ’ MSDN event) and Courtside Grille to watch the NLCS game 7.

November – Courtesy of my employer, attended The Data Warehouse Institute (www.tdwi.org) World Conference at Universal Studios in Orlando. I was able to walk around CityWalk during lunch and found it to be enjoyable. I learned a lot about ETL architecture and data warehouse design considerations. At the end of the month, went out with Tim, Kabza, Vinay, Schultes, and Healy to a good bar (The Independent) and Indian food restaurant in downtown St. Pete. It’s always a treat to socialize with these guys and I value those moments highly.

December – Attended the Visual Studio Team Systems for Database Professionals launch in Tampa at the Grand Hyatt. I unexpectedly was recognized in front of the 120 people there for my tribal knowledge and my wonderful website thanks to the Microsoft guys. Later in the month I attended the Hot Spaces.net Tech Social where my nametag with my name WES @ www.thedamndata.com drew quite a few stares. Met some great people and hopefully a hookup for some Tampa Bay Lightning tickets although she flaked at my first request.

Wow – now that I have written that I feel great and remember what a wonderful year it really has been. The goal now is to make 2007 even better, and tomorrow I will write a post with my thoughts on those goals.

Windows Vista – A New experience

2006-12-27T23:49:00.000-05:00

Those who have seen my presentations know that my life is on my laptop. Therefore, it was fitting that I was exceptionally frustrated when a 10-month old Dell Inspiron 9300 laptop began to chug along like a 4-cylinder truck.

I reformatted the laptop and using my MSDN Universal subscription decided to install Windows Vista Ultimate as my operating system of choice. Thanks to my excellent file management practices (nicely complementing my good data design practices), backing up my projects and personal files took all of 30 minutes. I downloaded Windows Vista and I was off running to the races.

The install process was especially smooth. Friends reported that Vista installs quickly, my experience did not reflect that and it seemed to take a good hour to fully install. Most of the hardware had build in Vista drivers.

The new look and feel is beautiful. Thankfully I purchased the laptop with 1 GB of memory and the dedicated (and at the time, kick ass) Nvidia GeForce 6800 256 MB (dedicated) video card. The fade look and shadows are very nice.

Vista is a bit of a memory hog and I regularly find that 500 MB of my memory is used even before I open up any programs. It will be interesting to see how Oracle 10g runs on Vista and SQL Server 2005 when I am running my demonstrations. With XP Professional, running Oracle and SQL Server 2005 together was risky business.

The only issues I have had so far have to do with my legacy programs. My HP printer will not install a driver, thus it’s practically worthless to me right now. I also use a statistical analysis package called STATA for my economics research, and that program would not install correctly on Vista. Even running as administrator with XP SP 2 compatibility did not work.

Thankfully I can still use my trusty Dell Dimension 8200 (4 ½ years old) to run my legacy apps and print documents. I certainly recommend anyone upgrading to Vista to consider having 2 GB of RAM as a minimum. I am loathe to upgrade the lappy for $200 right now as this machine will be replaced in 11 months with an Apple MacBook pro unless Dell has a more compelling product.

Along with Vista I installed Office 2007 Professional. I love the new look and features of this program and will probably do a separate write up of it later after I have used it more.

All in all, Vista appears to be a great step forward and I look forward to showing it off at the Miami code camp on February 10, 2007.

Data Warehouse opportunities in Tampa Bay

2006-12-04T23:04:00.000-05:00

It seems appropriate that I have been requested to help find some good talent to work with in the Tampa Bay area and the benefit is that you will be able to work with me, thus you will learn from the best (*HA*). Keeping in line with my blogging policies, I will not say the employer but if you've seen me speak you know where I work.

Position requirements:
-------------------------
College degree - no experience is okay, these are entry level positions
Willingness to learn
Good attitude

I've been speaking all over the state of FL this year and certainly there are 1 or 2 people who want to come work in a high-performance data warehouse environment, learn from people at the top of their game, and have a very strong resume builder position on their resumes.

Here's the technology you will be using:
-----------------------------------------
Oracle 9i/10g
IBM Websphere DataStage ETL
Quest TOAD for Oracle

We're also willing to consider entry-level people with experience in other ETL tools such as SSIS and Informatica.

If you are interested in learning more or want to send me your resume, my email is thedamndata AT yahoo DOT com. If you can't figure that one out, then don't bother.

Tampa SQL Presentation

2006-09-15T15:20:00.000-04:00

On Tuesday, September 19, I will be presenting "Introduction to SQL Server Intergration Services" at the monthly Tampa Bay SQL user group meeting.

The presentation will cover the SSIS tool and we will look at ETL design and developing and deploying some small ETL jobs to move data around different databases. I'll also take questions from the audience and attempt to answer them or make up some kind of answer to salvage my reputation!

Hope to see you there.

Tampa SQL Website

Jacksonville Code Camp

2006-08-26T10:40:00.000-04:00

The second annual Jacksonville code camp is August 26, 2006 in downtown Jacksonville. I didn't post this earlier because I wasn't sure of my attendance until the very last minute. A sequence of events (one is a pattern Mr. Rawls!!) transpired that made my travel to Jacksonville possible. My continuous travel has put a damper on housekeeping tasks and my session to speak was not turned in on time, but nevertheless I am giving a 75 minute session today titled "Introduction to Data Warehousing".

Tonight we will be at the Copper Cellar in Jacksonville Landings for the post-party event.

Come support our 'brothers-in-arms' in the Florida developer community at the Jacksonville code camp.