↓ Archives ↓

Category → KPIs and Metrics

DevOps Days Mountain View 2011: DevOps Metrics & Measurement (Video)

Panel at DevOps Days Mountain View 2011 on DevOps Metrics and Measurement.

Alexis Lê-Quôc (Datadog)
Patrick Debois (Jedi)
Vladimir Vuksan (Broadcom)
Brian Doll (New Relic)
Laurie Denness (Etsy)
Andrew Shafer (Cloudscaling)

Moderator: John Allspaw (Etsy)

See all videos from DevOps Days Mountain View 2011

DevOps Days Mountain View:
http://devopsdays.org/events/2011-mountainview/

Special thanks to LinkedIn for hosting DevOps Days Mountain View 2011.

Also, thank you to the sponsors:
AppDynamics  DTO Solutions  Google  MaestroDev  New Relic  Nolio
O'Reilly Media  PagerDuty  Puppet Labs  Reactor8  Splunk  StreamStep
ThoughtWorks  Usenix

 

More Video with John Allspaw at Etsy: Dashboard tour & metrics discussion

Here's another video I shot with John Allspaw at Etsy's Brooklyn, NY offices...

00:00 - John gives a tour of the big dashboards

02:42 - Zoom in on key deployment dashboard (yes, that says 24.6 code deploys per day!)

04:52 - John jumps on the whiteboard to explain what goes into their metrics efforts 

See the previous video with John.

Video with John Allspaw at Etsy: What to put on the big dashboards

I had the pleasure of visting the folks at Etsy recently. Etsy is well known in DevOps circles for their Continuous Deploymet and Metrics efforts.

I'll be posting more videos soon, but in the meantime here is a quick hit I recorded with John Allspaw while he gave me a tour of their Brooklyn HQ. I had just asked John about how they decide what to put on the big dashboards on the wall.

Here is what he told me...

The Five Why’s of Cloud

 

Editors note: This is the first post by our newest dev2ops.org contributor, John Willis.

 

 In my “cloudy” travels from Cannonical, to Opscode, to DTO Solutions, I often ask a seemingly simple question to the companies I meet - “Why do you want to use the cloud?” - and I get an array of amazingly unsatisfying answers.

Some people cite the classic 8-weeks-to-8-minute provisioning use case. Some people regurgitate the analyst answer of "for the ROI of (insert magic number here)" . Of course there is always the CAPex vs OPex answer. Now I am no financial type, but I've been around enough to know that in many types of businesses, OPex is more of a dirty word to the CFO than is CAPex.

Every once in a while I get a more informed answer like “I want to go to the cloud for agility”. But the usual lack of depth behind the answer proves to be just as unsatisfying.

 

The 5 Whys Epiphany

The other day I was having a conversation with my good friend Michael Cote and it hit me as to why I never get the depth of answer that I am am looking for. I have not been asking the right question(s).

Many of of us in our industry have learned and adapted many techniques and methodologies from Toyota’s lean manufacturing models. One of my favorites is the applying the “Five Whys” technique to determine a root cause of a problem. Therefore, I started applying the 5W’s during my informal surveys.

Here is an example of one of those conversations:

Question 1 - Why do you want the Cloud?

Answer: To decrease provisioning time.

Question 2 - Why?

Answer: So we can get servers provisioned in 8 minutes instead of 8 weeks.

Question 3 - Why?

Answer: So developers can get resources quicker to get there job done faster.

Question 4 - Why?

Answer: So they can get features out to customers faster.

Question 5 - Why?

Answer: So we make more money faster.

Aha, so that is why they wanted their cloud, to make more money and make it faster. At this point if I had more time I would start another line of questions (unfortunately I usually don’t) that begins with “Will the cloud do that for you?”.

Any of you who have been working with the cloud in a business for more than a year already know the answer to that question - Not out of the box it won't. But dispelling the myth that Cloud alone solves all problems is not my point here.

My point is that the real question/answer I should have gotten to my initial “Why Cloud” question is that last question/answer after 5W’s. The real answer regarding "Why Cloud" should not focus on technical feature, it should be be based on specific business goals. Ask the 5W's until those true goals are clear to everyone. Don't doubt the power of "Why?".

 

Begin With the End in Mind

They now teach to kids in elementary schools the idea of beginning with the end in mind. Its a lesson we can all learn from.

I like to use my cell phones in Russia analogy. When Eastern Europe opened up to capitalism, they didn’t go out and order land line phones all over the place. They knew from recent history that it was much easier to create new phone networks via cell phones. The began with the end in mind.

All too often, smart people lose site of this simple idea when it comes to the cloud. Getting a cloud is not the goal. The goal is to achieve a specific business objective (i.e., the root cause of why you want a cloud). Therefore if your real goal is “We want to make more money faster” then let’s map out the full path to get there.

I often describe the cloud as a big fence that comes just up to everyones eyebrows. It looks really cool but you can’t quite see what’s over the fence. I try to urge people to use a step ladder to look over the fence and see the longer journey. Most people will see that whats beyond the fence is not the end of the journey, but just the beginning.

Depending on what your business goal is, the end state which probably will include a cloud. But in almost every case, the end state will require far more than just a cloud.

 

In Cloud We Trust

I often hear in my travels the remark that “I thought the cloud did that”. It amazes me that an organization will first implement a cloud and then start asking questions like "where is the autoscaling part" or "where are the automated load balancers" or "where is the push button application deployment". I call this the being blinded by the cloud pixie dust.

When the pixie dusts settles and they examine what their cloud really delivers, typically they will find that they are still missing what my good friend Damon Edwards calls the “abilities”. Around you cloud you are still going to need to create a strong operational infrastructure that delivers scalability, manageability, availability, reliability, flexibility, and - last but not least - agility.

In my forever quest to find the perfect cloud, I have yet to find one that comes with one big red “Abilities Button”. Sure the cloud will help with some of these -- and some clouds provide more than others -- but the cloud is just one tool that you will need.

Just like before the cloud, successful cloud-based operations include a hand full off glue technologies and a whole lot of additional sweat equity.

When analyzing your requirements, don't let you thinking stop at the word "cloud". Focus on what that actually means. Focus on ideas like:

  • Utility based infrastructure
  • Self-service resource allocation
  • Self-managing infrastructure
  • Software development life cycle support
  • Behavior driven availability

The cloud is just one part of the equation. Clouds are useful tools, but not the magic bullet. My suggestion, therefore, is to always apply the Five Why’s to make sure you know where you are going.

How to measure the impact of IT operations on your business (Part 2)

Measurement: a set of observations that reduce uncertainty where the result is expressed as a quantity