Meet Microsoft's Ultimate Beta Tester

By Nate Mook and Tim Conneally, BetaNews

July 17, 2006, 4:45 AM

Markezich BigWith tens of thousands of external beta testers examining the latest Microsoft software, little is said about the company's largest guinea pig: itself. Long before a final release, Microsoft has put the beta bits into production -- a process it calls "eating its own dog food."

BetaNews recently sat down with Microsoft Chief Information Officer Ron Markezich, often referred to as "Microsoft's ultimate beta tester," to get an in-depth look at how dog-fooding helps shape the software that hundreds of millions of people use each day.

As CIO, Markezich heads up Microsoft's IT department, which is responsible for managing the massive infrastructure that supports over 63,000 employees around the world. Perhaps more importantly, his team also plays a vital role in product development by actively using Microsoft software before it's complete, filing bugs and providing feedback.

Finding individual consumers to beta test products like Windows Vista is easy; even Google offers much of its software as betas. But businesses are a different story. Corporations rely on mission-critical systems to keep things running smoothly, and any hiccup due to buggy software could have major consequences.

Microsoft has attempted to alleviate this dilemma by encouraging businesses to install its enterprise software during the beta phase through the use of "go live" licenses. However, the company has learned the only way to firmly stand behind a release when talking to partners is to first deploy it internally.

That issue came to a head with the release of Exchange Server 2000. Markezich says the product was not running optimally inside Microsoft, but it was shipped to customers nonetheless. The result was a veritable nightmare: customers experienced the same problems and Microsoft had to rush out numerous hotfixes.

"From that day forward we said we will not ship a product that we sell to the enterprise until we run our business on it," Markezich explained. Since then, the dog-fooding program has extended across a variety of Microsoft products and now serves as a quality checkpoint on the road to a final release.

"Not only do we want to get involved in the beta testing, but we also want to give feedback in the product planning, product designs, requirements, and then all the way through the entire lifecycle," Markezich says.

Unlike a typical beta tester, Markezich's team holds considerable sway over the release of a new product. In fact, he has the power to delay a release to manufacturing (RTM) if a software product hasn't hit goals and expectations the development team had previously established as part of the dog-fooding process.

"If we don't meet these shared goals for a particular milestone, we don't allow the product group to move on to the next milestone, all the way to RTM. If we don't meet the goals that we have laid out for RTM, sometimes even a year and a half or two years prior, we don't allow the product to ship," Markezich said.

For Windows Vista, Markezich has shared goals for BitLocker, enterprise search, strong user authentication, network access protection, and a number of other business-oriented features.

"This organization is an extension of the product team; we have sign-off authority on those products before they ship," he adds.

Continued. . .
1 | 2 | Next >>

Add a Comment (39 Comments)

BetaNews reserves the right to remove any comment at any time for any reason. Please keep your responses appropriate and on topic. Foul language and personal attacks will not be tolerated.

Name (required):

E-mail (required):

Enter Your Comment:

By Malshadow

posted Apr 1, 2007 - 9:52 PM

Sometimes I wish people would just leave Microsoft to die. After all, they're all at least millionaires off of products that work half as good as their free counterparts. Exchange server isn't the only thing to ever be shipped out with problems, almost every one of their products have bad problems, and when they release updates to fix them they often include spyware.

Score: 0

By ramesh_pilani4

edited Aug 2, 2006 - 10:43 PM

How will I become a beta tester...?
Guide me...

Ramesh with hope

Score: 0

By CyberDoc999

posted Jul 18, 2006 - 7:37 AM

"eating its own dog food"
vista beta is now called "dog food"
yum yum
funny how you can't hide the taste of "dog food"
no matter how pretty you make it look....
please pass the "dog food"
(I will pass on that).....

Score: 0

By paulm

posted Jul 18, 2006 - 5:57 PM

CyberDoc999,
You've got to stop eating dog food. Hasn't anyone told you it's not fit for human consumption! ;-)

Score: 0

By mesiex

posted Jul 19, 2006 - 8:29 AM

They are the dog and also the food.

Score: 0

By mesiex

posted Jul 18, 2006 - 1:01 AM

They are the hound and the hare at the same time. Who may be the judge and the defendant at the same time and expect a fair trial?

Score: 0

By ds0934

posted Jul 17, 2006 - 11:52 PM

Brave man. To put his picture and name out there tagged to such a bold initiative as that of saving the face of Microsoft's ailing public perception. He's either got massive gonads or no brain, or both. Or maybe they're holding his family hostage until Vista ships.

Score: 0

By Ciprian.Dobrea

posted Jul 18, 2006 - 4:45 AM

thanks, you made me smile ;)

Score: 0

By mesiex

posted Jul 17, 2006 - 3:40 PM

Monster Megabytes, monster HD, monster OS...for minor tasks? Too many backdoors. Maybe the best would be to simplify things. The biggest the OS, the hardest to control it.

Score: 0

By PC_Tool

posted Jul 17, 2006 - 5:00 PM

Slip in a DOS Diskette.

Simple enough for ya?

Score: 0

By dvferret

posted Jul 17, 2006 - 8:01 PM

lol. dont think you can get much smaller that that.

Score: 0

By Kramy

posted Jul 18, 2006 - 12:12 AM

How about this?

Score: 0

By mesiex

posted Jul 21, 2006 - 6:37 PM

Sorry, I didn't mean that. In my opinion the size of the system since Windows 98 SE until Vista has grown unproportionally to the new advantages. In my opinion a high percentage of dirt and unuseful drivers could be avoided or cleaned unharmlessly.

Score: 0

By 33Nick

posted Jul 17, 2006 - 11:45 AM

I just think it is horrible, inhumanne, appaling, having poeple test their own systems inhouse, and shock of all shock finding out it doesn't work quite as well as they thought. I knew Microsoft was mean but that is just down-right...mean!

Score: 0

By seier

posted Jul 17, 2006 - 5:47 PM

Hi Nick,

Dog-fooding does suck while you're working there and it can have a significant effect on productivity. For example if you're supposed to get 8 hours of testing done, but first you have to download over a slow network (1-3 hours) and then upgrade to the latest beta of Windows. However, now that I don't work there, I can really appreciate how significant of a test that is. I think it's one of their best practices. Other companies will adopt this policy soon enough. It's true a company shouldn't be selling anything to others that it isn't yet willing to use itself.

Cheers,
Christian

Score: 0

By dvferret

posted Jul 17, 2006 - 8:02 PM

Yes, I find this a very good policy and hope other companies follow, as well.

Score: 0

By Spyderloco

posted Jul 17, 2006 - 10:26 AM

I wish we had a CIO that actually knew something.

Our CIO is a freaking idiot and doesn't even know what a T1 is or what a switch or router does. Some days he has more problems with his desktop than the average user here.

He needs fired.

Score: 0

By xyzcb1

edited Jul 17, 2006 - 11:33 AM

The cannot can't fire him. Your company probably probably spent tons on hired him and gave him one of those contact that guarantee millions as long as he shows up to work. If they do fire him, they have to buy out his contract, which probably cost more. I am in a similary situation, my director here know jack-sh1t, and occasionally when he does "work", and that mean trouble for the team. So we are better off when he does nothing, and let us run on our own.

Score: 0

By jshurst

posted Jul 17, 2006 - 12:00 PM

That sucks. My last CIO seemed to know everything. I didn't expect him to know technical details, but he could talk to the programmers in detail like he was one, and was an awesome manager as well. He could then talk to every other department like he worked for them too. I was very impressed by his knowledge, but doubt that I'll find too many people like him in my future opportunities.

Score: 0

By crashoverride

posted Jul 17, 2006 - 9:17 AM

Is it just me or does this guy look a little nutts in that picture?

Score: 0

By ds0934

posted Jul 17, 2006 - 11:54 PM

It begs to have a mustache and glasses drawn on it. Doesn't it? :)

Score: 0

By crashoverride

posted Jul 18, 2006 - 10:18 AM

Indeed it does. :)

Score: 0

By otto888

posted Jul 17, 2006 - 10:09 AM

yep. he looks nuts to me too.

Score: 0

By cannie

posted Jul 17, 2006 - 7:18 AM

No matter how much dog-fooding, hackers are waiting out there. We shall have security patches again, for sure. In my opinion only open code would make possible for experts to know what the system is really doing and prevent leaks by personalizing their own.

Score: 0

By compm375

posted Jul 17, 2006 - 8:41 AM

I'm usually out bashing Microsoft, but I can't really agree with you. Open and closed software alike have security vulnerabilities. Open source has more people to fix the code, but also more to mess it up. The only real way to get rid of vulnerabilities is have companies rethink the way they write code, whether it be open or closed source. They need to have coders that know what they are doing and many people auditing. Sure, it would make code take several times longer to complete, but it would be a lot more secure.

Score: 0

By Kramy

posted Jul 18, 2006 - 12:15 AM

The other option is to simplify the language to the point that errors are obvious. A 1200 line javascript app is much easier to debug than a 50000 line C++ app. The disadvantage there though is speed.

Score: 0

By cannie

posted Jul 18, 2006 - 7:00 AM

Speed is not a problem any more. It means a faster processor and memory upgrade. Your option is a very clever one. Congratulations.

Score: 0

By cannie

posted Jul 26, 2006 - 4:51 PM

Maybe the best: back to Windows 98 SE and Opera.

Score: 0

By PC Rat

posted Jul 17, 2006 - 7:12 AM

...

Microsoft has ~got~ to find some way of getting
a new OS out the door in less than six years.

Obviously, what they're currently doing ain't
working.

Instead of self-congratulatory Puff Pieces like
the one above, Microsoft needs to seriously
re-think it's development model.

...

The Computer Rodent

...

Score: 0

By frankwick

posted Jul 17, 2006 - 11:39 AM

You will see a faster OS release cycle from MS after Vista. Keep in mind that between XP and Vista, MS rolled out Server 2003, R1, and XPSP2. These are just OS releases and don't include complex systems like Exchange and SQLServer.

Score: 0

By PC_Tool

posted Jul 17, 2006 - 1:56 PM

Don't.

Feed.

The.

Trolls.

Score: 0

By Intrusive_Rogue

posted Jul 17, 2006 - 2:07 PM

right on

Score: 0

By PC Rat

edited Jul 17, 2006 - 1:48 PM

...

"Keep in mind that between
XP and Vista, MS rolled out
Server 2003, R1, and XPSP2"

...

That might be an excuse if Microsoft was a little
mom-and-pop organization instead of a huge international
corporation staffed with 61,000 employees (2005).

What's the deal ? Microsoft can't chew gum ~and~ walk
both at the same time ?

...

The Computer Rodent

...

Score: 0

By The Man

posted Jul 17, 2006 - 10:11 AM

Agreed

Score: 0

By donpacman

posted Jul 17, 2006 - 7:04 AM

I wonder how many times his Mom tells him to take out the trash?

Score: 0

By Intrusive_Rogue

posted Jul 17, 2006 - 10:07 AM

Only one more time a week than your's does.

Score: 0

By donpacman

edited Jul 21, 2006 - 12:02 AM

genius!

Score: 0

By jshurst

posted Jul 17, 2006 - 12:02 PM

Too funny...

Score: 0