RobMensching.com /Blog
when setup isn't just xcopy

Explaining the value of packaging in California.

We just did the curly-cue take off from San Jose on my way to Seattle by way of Portland (don't ask).  The captain just informed us that the second pass over San Jose Airport is used to climb above the incoming traffic to SFO.  That makes an amazing amount of sense and provides a nice view of the whole bay area.  I just hope I don't miss my connection in Portland.

I was down in San Jose this weekend talking with the external development team that works on Windows Marketplace with us.  The new build and deployment system I mentioned previously is a pretty big departure from how Marketplace is built and deployed today.  I was having difficultly explaining the value of the new system through email so I just flew down to California to explain it in person.  That worked really well.

Since I was just talking about the Windows Marketplace deployment for the last two days I thought I could share some of the values.  Unfortunately, I'm not currently funded to fly out and talk to each and every one of you.  So you'll just have to do with a blog entry and any follow up conversations that happen here.  <smile/>

(Just a reminder, this will not be comprehensive list of reasons packaging has value.  This will just be the list that I can think of while in the air).

The underlying question I was really trying answer was, "Why create (MSI) packages instead of just some batch files to xcopy deploy?"  The following are my reasons, in no particular order, they are all important.

1.  Fully automated deployment.  This is the key goal, removing humans from the process as much as possible because humans are slow and make mistakes.  Perfect achievement of this goal would mean that humans would only be involved to create the packages (and stuff that goes in the packages, of course) and to provide the parameters that are unique to an environment (such as machine names and service accounts).  That means deployments to a known environment can be done by simply pushing the "Go" button.

Since you can't configure everything on a machine using xcopy alone (can't do anything in SQL for example) you'll either have to have a human intervene for the times when non-file based items change or you'll have to create some batch file to do it.  MSI packages plus the CustomAction library that is part of the WiX toolset can configure pretty much everything in a datacenter without human interaction.

2.  Fully configurable.  To make it easy to move from environment (test cluster, pre-production cluster, production cluster) to environment or to add new environments (new dev clusters, new test clusters, new production clusters around the world) the packages need to be configurable.  For Windows Marketplace, that means passing in Properties that configure XML files (web.config, policyCache.config, random .config files), SQL databases (some settings stored in database tables), IIS settings (e.g. SSL certificates), service accounts (AppPools and other services need to run as different users in different environments) and registry keys (yeah, we even settings stored in registry keys).  It is on my "hit list" to reduce the number of settings over time.

Again, xcopy deployment alone can't configure all the things that change from environment to environment.  A batch file or human interaction in each environment is necessary to tweak the little things that are different.  A MSI package can be passed properties that declaratively adapt the install from environment to environment.

3.  Reduce redundancy.  Once you are moving towards fully configurable you can start removing redundancy in the development process.

For example, when deployments are built without using packages I consistently see a separate web.config for each environment.  The problem is that when you want to add or change a setting, you have to go into all of the .config files and tweak the setting appropriately for each environment.  This process becomes insanely error prone as your system gets larger.  Again, using MSI packages means you can pass properties in during the deployment and configure a single template .config to meet the needs of the environment.

4.  Verify configuration.  As you start to rely more on properties to configure your deployments, you'll also want to verify that the parameters are within appropriate bounds.  This verification reduces debugging time as you bring on new environments or train new operators. 

Of course, xcopy deployments can't do any verification so you'd have to fall back to a batch file.  The Windows Installer Condition syntax is much richer than the checks you can do natively in a batch file.  There have been several times where adding a condition to the deployment for Windows Marketplace saved Zac and I several minutes of debugging because it caught a typo in the deployment properties.

5.  Transacted changes.  Ideally all changes to the datacenter move you from one known (working) state to another known (hopefully, working) state.  Creating transactions using a scripted deployment is difficult if there any branches in the logic.  The Windows Installer is a declarative installation engine and thus provides transactions for each package.  That doesn't help us with multiple package deployments much but if you keep your packages isolated from each other (which is generally pretty easy in datacenter applications) you can get "pretty good" rollback by uninstalling the new and installing the old.

6.  Rebuild everything, anywhere, anytime.  This is a power that you might not truly appreciate until you have experienced it.  The idea is that you can take any set of machines and quickly turn them into productive members of your datacenter application any time.  When you have a set of packages that install everything it is a simple matter of just running all the packages on the new machines.

For example, during our testing Zac and I needed to test a few new tweaks but the test team was currently pounding on the test cluster.  So, I bounced over to my dual-proc, dual-core, 2 GB machine and spun up a couple virtual machines with IIS on some and SQL Server on others and Zac typed in the parameters for this new environment.  In about an hour the virtual machines were acting as a new and completely independent test environment.  Zac practically broke down in giggles (he was responsible for setting up many of the environments previously and that was a grueling task).

7.  Quickly inventory a machine.  The Windows Installer provides a very rich set of APIs to determine what is (and is not) installed on a machine.  Even the ability to pop open the Add/Remove Programs control panel makes it trivial to quickly see, "What is on this machine?"  The ability to inventory the machine at many levels is just one more diagnostic tool.

So, that's a bunch of features you get when approaching datacenter deployment with a "package mentality".  What is the value of all that?  Three things come to mind:

1.  Consistent.  Everything is deployed in a package.  Developers know what they need to do to be deployed in the datacenter (build a new MSI or add to an existing MSI).  Operators know what to expect why they do deployments (deploy a bunch of MSI packages).  All of the processes (build, transfer, communication) get streamlined around a single set of concepts.

2.  Reproducible.  Without any human steps in the configuration of your datacenter, install and uninstall operations will always behave the same way.  Being able to consistently do the same thing over and over allows you to tweak out the bad things and optimize the good things.  It also means that if you get the package install and uninstall operations right in your test cluster chances are good the packages will work correctly in pre-production and, later, production.

3.  Confidence.  Once you have a consistent way of doing things and you can always do it in the same way, your confidence in the datacenter will improve.  Already I'm seeing Zac say, "Well, that's a bug.  We'll just uninstall and install the new fix."  He has confidence in the deployment system and knows that we just have to work out all the bugs introduced by the humans (us).  <smile/>

 

The death of the comprehensive blog entry.

Since I started blogging years ago, I've always tried to write comprehensive blog entries.  By comprehensive I mean blog entries that fully cover the topic of choice, are well organized and try to address all of the counterpoints that could be made.  My intention was to create a block of information that could stand all by itself and answer any questions that might occur to the reader.

Since some topics were so long, I tried to create a series of blog posts that would tell the whole story.  That never worked out as well as I had hoped because when I started the series the topic may have been relevant but often the current events moved the discussion beyond that topic.  Other times I'd lose interest part of the way through the series and wouldn't get back to it.

Since I want to enjoy blogging more and blog more often, I'm throwing out the idea that I'll ever write a comprehensive blog entry again.  That means if you read something here, feel free to send me a note (through the contact link) or leave a comment (assuming my blog server is cooperating) or even write your own blog entry (and trackback to mine so I find it).  We'll have a conversation.

Since I won't be trying to write write comprehensive blog entries any longer, I'm going to entertain the idea of writing a book again.  Seems like a book would be a much better place to capture comprehensive content anyway.  This blog will be a stream of consciousness, a book would be well organized and try to cover the topic of choice fully.

Since I can think of nothing else to say on this topic, keep coding... you know I am.

three zero.

It has been two years since I made a blog post on this day exactly.  In my mind and looking back that day held much more significance than today's three decade mark.  But since numbers ending in a zero seem to hold special significance in base 10 it is a nice time to reflect.

Before looking back too far, let me say that Sunday was a great day.  Jenny had (apparently) been plotting a surprise birthday party for me for weeks.  She provided me just enough information to keep me from suspecting anything unusual and had quietly contacted all my friends so that after I picked her up from work on Sunday everyone would be in the house.  Sneaky to the extreme.  I was thoroughly and pleasantly surprised.

As a random tangent, the last surprise party for me didn't go off nearly as well.  My good friend, Denise (who is an event planner in denial), planned a birthday party for me when I turned something like 24 or 25.  Unfortunately, she didn't anticipate me calling up Carolyn and seeing if she wanted to catch up over dinner... because I didn't have any plans for my birthday.  During dinner, my phone kept ringing with several of my friends with lame excuses why I should return home.  Finally, one of them finally broke down and said, "Do you know where I am?  I'm at your house  and you know what?  You're not here so you're missing your party."  I found the whole thing quite amusing (but Denise never really forgave me).  Jenny had her bases covered this time... she invited Carolyn and Steve to the surprise party this Sunday.  <smile/>

But getting back to the reflecting, on the drive downtown with just Jenny tonight for dinner I was thinking about what turning 30 really meant to me.  Over dinner I shared my thoughts with her that I'll share with you here now.

First, for the last 30 years (that does sound a bit weird) I feel like I've been working hard to learn as much as I could to build a foundation that I could operate from confidently.  Elementary school was about learning enough to get into middle school.  Middle school was about learning enough to get into high school.  High school as about learning enough to get into college.  College was about learning enough to get a good job.  Working at my job was about learning enough to be able to actually do something.  Well, now I actually feel like I have enough of a foundation to really do some cool things and learning is just a natural reflex (like reading words on passing billboards).

Second, my parents just turned 60.  They are doing quite well (healthy, active, comfortable).  I'm half their age now.  That (hopefully) means I have a good 30 years ahead of me to do a great many active things.  That means for as long as I've spent learning, I can now spend doing.  That is really quite exciting.

Finally, and this is a feeling better captured by my 28th birthday blog entry but Jenny and I are now married.  That means there are a whole bunch of new experiences and things to do with someone else that weren't possible before.  And I'm just starting to learn what that really means...

Personal Update

Yesterday I took the day off.  By that I mean I slept until about one in the afternoon, woke up to eat something, played some XBox360 and went to bed around ten at night.  When I first started at Microsoft when I used to work a lot.  I'd take every third Sunday off or something like that.  Yesterday felt like one of those Sundays.

I have been working a lot for the last two months.  Monday through Saturday often 7 AM (if I work out) or 9 AM (if I don't) to 7 PM.  On Thursdays I stay out until around 3 AM working on the WiX toolset.  When Jenny was gone for a week in January to visit family I got back on my "early days at Microsoft schedule" of around 20 hours that causes my sleep schedule to rotate to a point where I actually go nocturnal before returning diurnal a day or so later.

Somehow Jenny has put up with it.  It helps that she works Saturday's and I only go nocturnal when she's not around for a few days on end.  I also make it home in time for dinner almost always.  Also we can see the light at the end of the tunnel (this month) and I'm pretty sure it isn't a train.

So why have I been working so hard?  Well, first it was the release of Windows Vista and Microsoft Office 2007 on Windows Marketplace.  One day maybe I'll take the time to write down the story of how that all came to be.  The whole event has been very encouraging to me after my time in COSD. 

The second that I've been working so hard on is a completely rewrite of the build and deployment system for Windows Marketplace.  The previous systems were very human intensive which means they were slower and more error prone than an automated system could be.  For example, I've been able to get average build time of 1.25 hours down to under 10 minutes.  The deployment times dropped from 2 hours (when there were no mistakes by humans in a very long list of steps for humans to take) to approximately 2 minutes.  Most importantly I'm making everything a "two button system".  One button builds.  The second button deploys.  Literally.

The new build and deployment system will be used to rollout Windows Marketplace at the end of the month assuming Zac and I don't fall on our face somewhere in the next week.  If you see the site down at the end of the month then you'll know who's fault it is.

Finally, I'm making a change to my blogging ways.  A long time ago I mentioned I was going to start addressing comments the way I saw Scoble doing it.  That worked for me.  However, I've been trying to do blog entry's the way Raymond does it; only one (maybe two) per day.  The problem is that I don't write a bunch of blog entries up front and have a system for distributing them out one by one day by day.  So, I'm switching to a new scheme that I've seen Dare use.  I'll write when I have time and probably post two to four blog entries of various size. 

Why has it taken me so long to realize that I one blog a day wasn't the model for me?  I don't know.  I think I maybe ascribed too tightly to the guidance that posting regularly was important for a blog.  Ironically, trying posting regularly just made get further behind is sharing all the blog entries I have to write (I literally have a folder with 95 items in it).  Anyway, I have to start the day and get off to a birthday party for one of my cousins.

Keep coding, I'll be at it again Monday.