How Microsoft can clean up the mess in its home and small business server business

If you want your customers to trust you, don't lie to them.

That sounds like common sense, the kind of stuff you should learn in the first semester of business school, but apparently someone up in Redmond skipped that lecture. And boy, are they paying for it now.

Last week, Microsoft announced its decision to discontinue development of one of the core features of Windows Home Server, a technology called Drive Extender. I won't rehash the details here. You can get an overview in this post from my colleague Mary Jo Foley and get more technical details in this thorough explanation from Peter Bright at Ars Technica.

Instead, I want to talk about how Microsoft delivered the bad news, and how in a matter of hours they destroyed their relationship with a large and loyal customer base.

Update 1-Dec 4:00AM PST: Apparently, Microsoft has also alienated its premier Windows Home Server partner. Alex Kuretz, a former HP employee and Windows Home Server blogger, reports that HP has "discontinued the MediaSmart Server and will not be releasing any more models either on the current version of Windows Home Server or the upcoming Vail platform."

The actual announcement from Microsoft came in a blog post by Michael Leworthy, blandly titled "Windows Home Server code name 'Vail' – Update." It's a masterpiece of corporate muddlespeak:

During our current testing period for our Windows Home Server code name “Vail” product, we have received feedback from partners and customers about how they use storage today and how they plan to use it moving forward. Today large hard drives of over 1TB are reasonably priced, and freely available. We are also seeing further expansion of hard drive sizes at a fast rate, where 2Tb drives and more are becoming easy accessible.  Since customers looking to buy Windows Home Server solutons [sic] from OEM's will now have the ability to include larger drives, this will reduce the need for Drive Extender functionality.

When weighing up the future direction of storage in the consumer and SMB market, the team felt the Drive Extender technology was not meeting our customer needs. Therefore, moving forward we have decided to remove the Drive Extender technology from Windows Home Server Code Name “Vail” (and Windows Small Business Server 2011 Essentials and Windows Storage Server 2008 R2 Essentials) which are currently in beta.

As of this writing, that post has some 246 comments, filled with words like stupid, clueless, and incompetent. A follow-up post from Leworthy did absolutely nothing to quell the outrage from the aggrieved Windows Home Server community.

The reason they are outraged is because everyone knows the real reason why this feature was dropped. The Drive Extender architecture had a troubled past involving a particularly nasty data-destroying bug that was finally fixed in late 2008. For the next release of Windows Home Server, someone made the engineering decision to completely redesign the Drive Extender code base. I'll let Peter Bright explain what happened next:

With Vail, Drive Extender was completely rewritten in a manner that should make it both more flexible and more reliable. Instead of using regular NTFS disks, Vail inserts a layer underneath the filesystem. This layer was responsible for distributing blocks of data between disks, replicating them to ensure fault tolerance, and de-duplicating them to make the system more efficient…

Unfortunately, the new block layer in Vail doesn't quite work right. Just like Drive Extender in Windows Home Server, there have been bugs. Different bugs, but bugs all the same. Microsoft hasn't gone into explicit detail about what these problems are, but there were some issues with its ability to correct errors, and some Small Business Server testers reported application compatibility problems.

So instead of fixing the flaws, and potentially delaying the three products dependent on Drive Extender, Microsoft is killing the feature altogether.

Everyone in the Windows Home Server community knows this is absolutely true. And yet Leworthy did not mention a word of those bugs and engineering issues in his bland and incomplete blog posts. Let's call it a lie of omission, because that's what it is.

I spent 30 minutes on the phone with Leworthy last week, in a tense interview that I'm sure was as uncomfortable for him as it was for me. Initially, he stuck with the story that this decision was based on customer feedback, repeating the official messaging that "the implementation [of Drive Extender] didn't fit the needs of our customers." But eventually he acknowledged, "We were worried that we might find data error issues."

Page 2: Do big hard drives really solve the problem? -->

<-- Previous page

With Windows Home Server, Microsoft created a monster. The worst kind, in fact.

As a product, it has built a passionate user base, filled with enthusiasts who have pushed the platform to its limits. As a business, though, Windows Home Server has been a dud, never really breaking out beyond its enthusiast base.

Adding a pair of similar server products aimed at the small-business segment, with the same set of features as Windows Home Server, is a smart business decision. It leverages the WHS development effort and expands the market for small, easy-to-manage servers into what could become a billion-dollar business. If the engineering team developing the Drive Extender technology can't confidently predict that they will be able to deliver a robust, reliable version of the technology to ship with all three of those products, then it is indeed a sound engineering decision to look for alternatives.

Leworthy's original explanation for the decision cited the proliferation of large hard drives of 1TB, 2TB and more. And indeed, for many Windows Home Server users a single large disk would be sufficient to handle backups and shared files with ease. Microsoft refused my request to share their data on exactly how much storage a typical Windows Home Server customer uses, noting only that the average is "well under a terabyte." A knowledgeable source told me that the actual number is around 730GB, which is more than enough to fit on one of those single large drives.

But averages can be deceiving, especially when your most vocal, passionate users—the ones who evangelize the platform—are storing massive amounts of data on their home servers. Via Twitter, I did an informal survey of 18 Windows Home Server users (myself included), asking them to list how much data they have stored on their Windows Home Servers. Here are the averages:

• Data backups: 336 GB
• Data duplication: 710 GB
• Shared files: 2.02 TB

That first number is a tribute to how amazingly efficient Windows Home Server is at storing backups, thanks to its single-instance storage feature. Backups that might gobble up a full terabyte on a device like an Apple Time Capsule can be stored in a fraction of that space because duplicate files (like system files that are identical on multiple computers) are stored as a single instance, increasing efficiency of storage.

The second number, data duplication, is a measure of Drive Extender at work, duplicating files on different disks so that a failure of one disk doesn't destroy the entire server. Surprisingly, this feature isn't universally used. In my admittedly anecdotal and non-scientific survey, roughly one-third of users duplicated 80-100% of shared files, another third used duplication for 40-60% of files, and yet another third used little or no data duplication.

And then there's the shared files category, which includes the most passionate, envelope-pushing group of all. Roughly a quarter of the Home Server users I surveyed have more than 3 terabytes of data. Most of that data, I suspect, is in the form of digital movies, either ripped from DVDs or downloaded from the Internet.

It's noteworthy that more than half of the users I surveyed have a combined total of less than 2TB in data and backups. If you offer a cloud-based backup of critical data, those users might indeed be able to get by without Drive Extender.

Page 3: Is there a solution? -->

<-- Previous page

So what about customers who want a small server that can hold lots of data and whose storage can be expanded on the fly?

In his discussion with me, Leworthy noted that they have been "having discussions for a while" with OEM and software vendors to find alternatives to Drive Extender technology, looking primarily at RAID storage, data duplication, and software management. The trouble, as he acknowledged in our talk, is that hardware RAID is difficult and expensive to implement in a consumer space.

Indeed, RAID is a poor solution for anyone except storage specialists. For consumers, it's a huge step back from the set-it-and-forget-it design of the original Drive Extender technology. But it does make sense for small business server products. That's especially true for servers designed for 20 or more users, which will generally need a reseller and an IT pro to set up and manage the server.

Meanwhile, where does that leave those enthusiasts who disagree with Microsoft's decision and are offended by the misleading and mushy way the bad news was delivered?

I'm told that an online "Bring back Drive Extender" petition at Microsoft's Connect site has more than 3000 signatures so far. That is not surprising, based on the intensity of reaction I've heard from the WHS community. I asked Leworthy if that was an option and got this discouraging response, which sounded like it had been written by a lawyer:

The decision has been made to remove Drive Extender from the platform. We are moving forward without it.

But I don't think the decision is all that cut-and-dried, and Microsoft still has a chance to recover with a few smart moves:

First, optimize the basic Windows Home Server product as a single-drive product, with 1 to 3TB of storage intended for backups and light file sharing in a home setting. This type of device could be small, quiet, and reliable, and based on Microsoft's data it would serve the needs of most Home Server users and even a fair number of enthusiasts. (If I could find a single-drive 3TB Vail-based device next year, I would buy it in a heartbeat.)

Next, bring back the first-generation Drive Extender code and whip it into shape as an optional add-on for enthusiasts who want to build multi-disk boxes. Make it available only for Vail and block its installation on the small-business servers.

Give every Home Server installation a dedicated 50 or 100 GB of SkyDrive storage and use it as a free cloud-based backup service to help overcome the loss of data duplication in Drive Extender.

Finally, start telling the truth. The whole truth. Everyone knows there are issues with the new Drive Extender technology. Acknowledge them openly instead of trying to hide them under a bunch of doubletalk. Microsoft owes no less than that to the customers who've stuck with them for all these years.