Wednesday, October 19, 2011

The Dark Side Of Clouds

In my travels, I'm starting to see a disturbing trend.  I don't like it one bit.
 
The setup is easy.  Business is demanding more from IT.  IT wants to respond by being able to deliver cloud-like services internally.  A group of smart people assemble to try and tackle the challenge.
And then it all goes horribly, horribly wrong.  

The assembled team decides to invest many months and piles of money to go hand-craft their own "cloud" out of commodity components.  Their management team, for whatever reason, allows them to go do it.  Based on experiences so far, almost no one seems to be able to reach the finish line with a satisfactory result.

The first time I saw this at a large financial institution, I thought it was an anomoly.  I've now seen this for the 7th or 8th time, so there's a pattern forming.  And it's grim.

I've been giving some serious thought as to why some IT groups decide to go down this particular road, and others take a more pragmatic route.

And, to be honest, it's not a pretty picture.

Meet The Cloud Building Team A classic example was about six weeks ago -- I had a few hours with the "cloud team" at a relatively large financial services institution.  They had started to assemble a home-grown cloud environment, and they wanted to know what commodity-like storage bits and pieces EMC might have that could help.

Not a good starting position, but I was sort of curious as to why they were going down their chosen path.

My first question -- why are you folks hand-crafting a commodity cloud vs. simply buying a pre-integrated and supported solution like a Vblock?

Obviously, to save money they replied.  They felt that by using open source and commodity components, they had the potential of saving their company a "boatload" in vendor costs.  Amazon had done it, Google had done, why couldn't they?

My second question -- how was it going?
A bit of a pause before the response.  It was taking longer than they had originally promised their management, and they were facing more challenges than they had originally anticipated.  Nothing that more time and more resources couldn't fix.

My third question -- well, how far had they gotten?  
After six months, they had a modest environment of a few hundred virtual machines up and running, and it was sort of working on good days.  They'd be unable to convince too many people to use the new environment -- something about stability problems, application compatibility issues, a less-than-optimal user experience, and so on.  Nothing that couldn't be sorted out.

The big question -- how much money had they put into it so far?  
Not much, they'd reckoned -- some commodity servers, storage, networks, etc.
Hold on -- what about all the time your team had spent so far?  Six months -- how many people?  Long pause.

And what about the opportunity cost?  Wasn't the team six months in with nothing really usable to show so far?  And no firm estimate as to how much more it was going to cost, or how long it was going to take, or even if the results would be usable?  A much longer pause ensued.

I was calling their baby ugly.  You could feel the tensions start to rise.

I thought for a moment as to whether or not I should bring up their perceptions of ongoing support costs associated with their pet project if it ever made it into a production environment.

I wisely decided to hold that thought, and pursue a different course.

Perceptions About The Alternative
OK, I said, you guys probably know about what VCE is doing with Vblocks, and what HP is doing, and so on.  Why didn't you go down that sort of route?

The first objection was predictable: it costs too much.  Really?  How did the cost-to-serve numbers stack up?  Well, they really hadn't done a detailed comparison, but they just felt that by going down the open source and commodity component route, they'd end up saving money.

After all, that's what the big cloud boys did -- Facebook, Twitter, et. al.
The second objection was also predictable: they wanted to avoid vendor lock in.  They felt that if they started with something like a Vblock, they'd be forced to stay with something like a Vblock, especially if they wanted to introduce another flavor of storage, server, network, hypervisor, etc.  They felt that utter flexibility to mix and match components was essential.

That's interesting, I thought.  How did they feel about their mainframe environment?  Their corporate network?  Their email and database environment?  Their purchased applications?  Weren't those largely vendor-specific enterprise-class environments?  

Yes, the claimed, and that was precisely what they wanted to avoid this time around.
I kind of sat back and thought about what they were saying -- I was quiet for a while.  I wasn't quite sure where to take the discussion, really.  I needed time to regroup.

One of the senior guys spoke up after a bit, maybe being a bit too honest.

Look, he said, there are going to be a lot of financial institutions that are going to want to run on commodity clouds in the future.  If they could figure out how to make it work here, there'd be strong demand for that skill set and expertise elsewhere.

Wow.  I really had to think about that one.  I was largely dumbstruck -- ultimately, this wasn't necessarily about doing what was best for the business.  Now it seemed that part of the agenda was about helping the participants create a marketable set of in-demand skills.  Wow.
What Could I Say?

There comes a time in some customer discussions where you assess the situation, and the logical course of action is to beat a hasty retreat.  This particular conversation wasn't going anywhere fast.
They had ostensibly brought EMC in to talk about some of our specific storage products that might work in their environment.  They were curious about what Atmos was all about.  They wanted to know what they upper-end of the Iomega range could do for them, and they had a passing interest in low-cost, high-density and high-bandwidth modular storage arrays. I told them what I knew.

One of the guys in the room kept interrupting me and wanted to know what the effective price per usable gigabyte was for each of the products I was talking about.  I guess I frustrated him, because I just didn't know what the current answer was -- the prices drop routinely as the components change.
After about 15 minutes of that, I did come up with one last proposition to try and salvage the original thread.

The Proposition
Look, your team has been chartered to deliver a cloud-like environment to IT so they can start delivering variable IT services back to the business.  You're under the gun to deliver something usable, and soon.  And you've convinced yourself that the right path is to roll your own.

Since there's a lot at stake, why don't you understand your competition?

Bring in a small Vblock, see what it can do and use it as a benchmark to compare your internal efforts.  Compare costs, compare functionality, compare stabilty, compare integration, compare compatibility, compare support, etc.

If you can do better, faster and cheaper with your own home grown efforts, great.  You'll have accomplished something substantial. But at least you'll know what competitive alternatives can do -- you'll have the facts.  And if you can't do better on your own, you'll know that sooner than later.
They politely thanked me for my suggestion.  I knew that they'd never consider it seriously.

Later Reflections
As I was walking out, I was thinking about all the Vblocks that went in shortly after the Big Homegrown Cloud Project had visibly failed.  I could have shared that with the team, but I don't think it would have mattered.

I could have made the case that Amazon, Google, Facebook, Twitter, et. al. have a very different business model (not to mention application profile!) than large financial institutions.  I don't think that would have mattered either.

I could have made a case that changing how IT operates -- and how it interacts with the business -- is the *real* heavy lifting around clouds and IT-as-a-service.  And, instead, they were spending all their precious resources on re-inventing the wheel.  I don't think that one would have worked.
I could have made a case that getting something workable and usable on the floor probably mattered more to the business people who were paying the bills than their elongated and uncertain approach.  I'd be just talking to the wind on that one ...

The sales rep was looking for some ideas on what to do next at the account.

The first thing that came to my mind was simple: go find some grown-ups, and tell them what's going on.  At some point, a legitimate business requirement had obviously turned into a self-satisfiying science project for the participants.  Their cloud project wasn't going well.  However, their defense and rationalization mechanisms were working quite well indeed.

Based on my experience, the prognosis was not good.  The sooner someone who was paying the bills realized it, the better.

And, no, you weren't going to earn any brownie points with the "cloud team" in the process.
The bottom line?  Had they chosen a different path, they would have been using their cloud in production for several months.  They'd be moving on to the next challenge.  Or, they could look at what VCE had done, and make an informed case to do better themselves.

Instead, it looked like all they had a pile of assembled components, and nothing really useful to show other than some interesting experiences and some more stuff to put on their resume.

Sure, there are cases I've seen where -- yes -- it could make sense to invest in a big team and a big project to create a hand-crafted cloud.  But that's the rare exception, and only after a sober analysis of the costs and benefits of doing so.

The first few times I saw this, it was unusual.  But it seems to be happening more.
And I think that's not too good for anyone.

By: Chuck Hollis