↓ Archives ↓

Category → chef

Test Driven Chef Cookbooks With Meez

When cooking a new dish, things get out of control if you try to manage too many things at once

You might face a similar situation when trying to write a new Chef cookbook.

Getting your arms around all those tools and frameworks needed to write solid, tested cookbooks gets you spinning. You need to install Food Critic, Chef Spec, Berkshelf – and the list goes on. This set up can easily take up to half a day or more.

French Chefs arrange all the ingredients which they’ll need well before they start cooking. They call this set up procedure „Mise en place“, or in short „Meez“.

And you should do the same

That’s what Meez is all about. It’s a Ruby Gem which creates a cookbook for you which has puts all these tools in place before you start cooking. Now you can work like a professional Chef concentrating on the recipe at hand instead of juggling new tools.

Meez needs a Ruby installation

Meez manages all dependencies using Bundler. To install it on your box run:

gem install bundler

Additionally, make sure you’ve got VirtualBox and Vagrant installed, too.

Let’s create a cookbook called my_cookbook using Meez

  1. Install the Meez Gem:

    $ gem install meez

  2. Let’s ask meez to create our cookbook:
    $ meez -C "Matthias Marschall" -m mm@agileweboperations.com -I apachev2 my_cookbook
  3. Time to install all required gems:
    $ cd ./cookbooks/my_cookbook
    $ bundle install

  4. Install all dependent cookbooks using Berkshelf:
    $ bundle exec berks install

  5. Time to run all auto-generated test suites:

    $ bundle exec strainer test

Start writing your cookbook

If you add new dependencies to your metadata.rb, you need to run bundle exec berks install again.

Then you can re-run your complete test suite using bundle exec strainer test

Read more about the ideas behind Meez in The Lazy SysAdmin’s Guide to Test Driven Chef Cookbooks written by the author of Meez: Paul Czarkowski (@pczarkowski)

FOSDEM 2014 is coming

and with that almost a full week of side events.
For those who don't know FOSDEM, (where have you been hiding for the past 13 years ? ) Fosdem is the annual Free and Open Source Developers European meeting. If you are into open source , you just can't mis this event where thousands of likeminded people will meet.

And if 2 days of FOSDEM madness isn't enough people organise events around it.

Last year I organised PuppetCamp in Gent, the days before Fosdem and a MonitoringLove Hackfest in our office the 2 days after FOSDEM This year another marathon is planned.

On Friday (31/1/2014) the CentOs community is hosting a Dojo in Brussels at the IBM Forum. (Free, but registration required by the venue)

After the success of PuppetCamp in Gent last year we decided to open up the discussion and get more Infrastructure as Code people involved in a CfgMgmtCamp.eu

The keynotes for CfgMgmtCamp will include the leaders of the 3 most popular tools around , both Mark Burgess, Luke Kanies and Adam Jacob will present at the event which will take place in Gent right after Fosdem. We expect people from all the major communities including, but not limited to , Ansible, Salt, Chef, Puppet, CFengine, Rudder, Foreman and Juju (Free but registration required for catering)

And because 3 events in one week isn't enough the RedHat Community is hosting their Infrastructure.next conference after CfgMgmtCamp at the same venue. (Free but registration required for catering)

cya in Belgium next year..

Chef: The Definitive Guide

Writing books is hard. I used to think it was a lot like writing a blog, only scaled up a bit. But it's much more involved than that. Once you're a published author, suddenly you have obligations and responsibilities to your publisher, and to the paying, reading public. The ante has been well and truly upped.

A few years ago I wrote a slim volume - Test-driven infrastructure with Chef. At the time I wrote it, I was pretty much the only person trying to do infrastructure code in a fashion inspired by the TDD and BDD schools of thought which I practiced as a software developer. When it was published, it was remarkably popular, and despite really being a book about test-driven development, and infrastructure code, it was widely read as a book 'about' Chef.

The problem was, this was the only published book on Chef. Chef as a tool was growing in popularity, and regular complaints were heard about the quality of the public documentation on the Opscode wiki, and about the failure of my volume to be a comprehensive introduction to Chef. Notwithstanding the observation that my first book was never intended to be a comprehensive introduction to Chef, both O'Reilly and I took this on board, and began work on a "Definitive Guide" - a full-length, comprehensive book on Chef. And that's where the problems started.

Chef is a large project, with a very active community. It's also a young project, moving very quickly indeed. Any attempt to capture the 'recommended approach' at a point in time was quickly rendered obsolete. New features were being added at breakneck speed. New community tools were being developed, and 'best practice' was in a constant state of flux. It was clear that writing a 'definitive' guide was going to be a significant undertaking, and quite possibly a flawed one.

At the same time, interest in test-driven infrastructure was exploding. New testing approaches and tools were blossoming, dozens of talks and discussions were being had, and my little introduction to the idea was getting slammed by its readership for covering only one approach, for being too short, and for not being a definitive guide to Chef. Did I mention that writing books is hard?

I had to make a decision. With limited time available, with a busy speaking, training and consulting schedule, and a large family, where should I focus my attention? After discussions with O'Reilly, and friends and colleagues, I decided that I should work on updating the initial Test-driven Infrastructure book. Now, in microcosm, we had the very same problem of a rapidly growing toolset and community to deal with. Tools came and went, underwent complete rewrites, and best practices continued to evolve. I also felt it was necessary to try to give a more thorough Chef overview, in response to feedback on the first volume. I worked incredibly hard on the 2nd edition. Frankly, too hard. I made myself ill, got entirely burned out, and let down family, friends, colleagues and readers. But, by the time of Velocity, this summer, we were pretty much finished. I met with my editor at the conference, and we surveyed the landscape.

The much-hated wiki had been replaced by a greatly improved documentation website. Ospcode had hired a technical writer to work on documentation, and community engagement in improving and maintaining that resource was growing. We remained convinced that trying to write a "definitive guide" at this stage was not a wise choice.

Additionally, Seth Vargo had begun an excellent project: learnchef. Aimed at being the ultimate hands-on quickstart guide, this was immediately well-received, and together with my second edition, filled the requirement for an introduction to Chef. The intermediate, reference-level specification of the core Chef functionality was adequately covered by docs.opscode.com. What we felt was missing was deep-dive, subject-specific discussions. How do I build infrastructure on Windows? How can I build a continuous delivery pipeline using Chef? Advanced programming in the Chef environment. That sort of thing.

We agreed to cancel the definitive guide project, with a view to working on these subject-specific guides. I tweeted about this, more than once, and shared our intention with friends in the community. What we didn't do, either O'Reilly, or me, was make a formal, public announcement. That was a mistake. I can't apologise on behalf of O'Reilly, but I can apologise personally. That was unprofessional of me: I'm sorry.

So, it's now summer, and I'm utterly exhausted. But in the spirit of the invincible superhero, I continued to take on work, travel, speak, and generally over-commit. My physical and mental health deteriorated further, until late August when I had to accept I was at the point of complete breakdown. I took about 6 weeks off, recovered my perspective, got some good rest, and left the editorial process in the safe hands of O'Reilly, and Helena.

Fast-forward to now. The 2nd edition of Test-Driven Infrastructure is out, and early reviews are positive. I'm rested, healthy, and hopefully wiser. I've learned a lot about Chef, and about writing, and am ready to start on my next project... this time with co-authors from day one. I have ideas on what we should cover first, but I'm open to suggestions and requests.

In the meantime, we have good resources available. Use learnchef, join the IRC channels, participate in the mailing list, read my book. Matthias Marschall has just finished his book on Chef, which is also excellent. People who lament the quality of the official documentation - please: give specific examples of where you feel information is missing, the writing is poor, the material is misleading. Remember: this is a community project - if you think you can improve the documentation, submit a pull request, and make it better for everyone. Opscode is committed to great documentation, and the decision not to try to write a definitive guide forces us as a community to build this reference ourselves, openly.

To conclude - I acknowledge that I've let down the people who were so eagerly anticipating "The Definitive Guide". I also accept that we handled the communication of our decision badly. But I think it's the right decision. And I think we're in a strong position to move forward and build on the resources we already have as a community. Will you help?

What Interests People About The “Chef Infrastructure Automation Cookbook”

In October I teamed up with Packt Publishing to organize a giveaway of my book Chef Infrastructure Automation Cookbook. I asked people what interests them most about this book.

Chef Infrastructure Automation Cookbook

My book contains practical recipes on everything you will need to automate your infrastructure using Chef. The book is packed with illustrated code examples to automate your server and cloud infrastructure.

Two lucky winners won an e-copy of the book. Find out what interests people most about this book in the comments below.

If Devs Own Testing, Ops Owns the Environment

The devs are all writing automated tests and some are even experimenting with TDD. Congrats! But what happens when the build server breaks? Who’s taking care that Continuous Integration is running smoothly? Seems to be an awful lot of red in there…

Unlike writing the first basic tests, CI is hard. Did the test fail due to an application bug or is it the environment? Once again, the dreaded chant of “it works locally” is taken up. What most people fail to understand is that the failing test is the first sign of a communication breakdown between developers and sysadmins.

Maybe the dev locally installed the latest version of python because it brings a 25% performance boost to execution. Or the sysadmin upgraded the virtualenv package on test due to a glaring security hole. And never rule out that both these guys made some “quick fixes”! Whatever the cause, your business application doesn’t work anymore. Who’s “fault” is it? And, more importantly, who should fix it?

This situation is the whole point of DevOps. Or, rather, the idea behind DevOps is to prevent these situations from happening in the first place.

An ounce of configuration is worth a pound of troubleshooting

  1. get the build server under control (puppet, chef)
  2. get the dev boxes under control (vagrant => puppet, chef)
  3. failing test on CI means “stop the line” – no more commits, builds from *anybody* until the test passes

By “under control”, the team agrees on a set of tools, applications and version numbers that need to be available on the server. These tools are defined in an automated configuration management tool running on all development, build and test infrastructure. Initially, only the sysadmins have write access to the repository that hosts this tool.

How to handle problems the DevOps way

  • Never blame!
  • Work together to solve problems (helps with situational awareness)
  • Share daily stand-ups and tell folks what you’re doing!
  • Trust, Measure and Publish

Don’t fall into the trap of blaming and arguing – this isn’t a pissing contest! Act like professional adults interested in the success of your company. The other party did what they did for a reason – take interest and find out. Try to work together to explain and understand why the change was made. If at all possible, combine daily stand-ups and tell people you’re going to be patching the test servers.

Avoid these common mistakes

  • don’t give devs write access to the automated configuration management – they don’t have time for sysconfig
  • don’t give devs sudo access – when you have a hammer, everything is a nail!
  • scale down resources for dev boxes – their desktop VMs typically don’t have 16GB of RAM

Ultimately, when development is responsible for testing and sysadmins are responsible for the environment, there is little chance of misconfigurations and misunderstandings. Having clear lines of responsibility empowers a team to focus on what really matters (and what they’re truly competent in). Communicate this rationale to both sides and get these folks pulling together for the business not their individual departments.

Infrastructure as Code – A comprehensive overview

I've been tracking infrastructure as code for a few years now. Over the years it has gotten closer to real code.

Close but no sigar yet.... We've come a long way but when you compare it to real languages it still feels in it's infancy. In this updated overview I gave at the ABUG, I went through:

  • the basic concepts of infrastructure as code
  • the differences/concepts in the languages (chef, puppet, ...)
  • the editors , syntax checkers, highlighting
  • integration with git version control
  • integration with CI systems
  • the different forms of testing (syntax, compile, unit, smoke testing)
  • using vagrant, veewee and the tools in that eco-system
  • debugging , profiling your code

This talk is probably the most comprehensive tool list that I've seen/made about the subject. But feel free to post and add your findings in the comments!

Note: that at the end of the presentation there are many extra links still to be sorted or slight outdated tools.

I've given previous versions of this talk at Devoxx 2012 and Jax2012. Enjoy the Jax2012 video here:

What if Config Management was created by Game Designers

Ever wondered what Config Management would look like if it was created by Game Designers?

Enjoy my Ignite session from Devopsdays London 2013


(n) Steps to Configuration Management Paradise…

Over the past five years I’ve come to experience the delights of Puppet, CFEngine and Chef across a wide range of deployments ranging from a couple of web servers to host two or three hundred sites, to thousands of servers underpinning an OpenStack-based cloud solution. I’d like to share a couple of thoughts on what […]

Vagrant and Drupal, a winning team

While heading back home from DrupalCon Munich after 4 days of good interaction with lots of Drupal folks.
I realized to my big suprise that there are a lot of people using Vagrant to make sure that developers are not working on platforms they invented their own. Lots of people have realized that "It works on my computer" is not something they want to hear from a developer and are reaching out to give them viable solutions to work on shared and reproducible solutions.

There were 2 talks proposing solutions to the problem,

the first one was ..Fearless development with Drush, Vagrant and Aegir by Christopher Gervais He talked about Drush VAgrant Integraion and how extentions to Drush allow for easy vagrant integration , bridging this gap allows rupal developers to use a tool they are already familiar with

The second one was Jochen Lillich who explained how he us using Vagrant an Chef for this purpose his talk titled Use datacenter tools to make your dev life easier has been posted already.

During the Vagrant BOF , I briefly ran over @patrickdebois old slides on Vagrant after which people started discussing their use cases.. 2 other projects came up

First is Project Oscar which aims at providing developers with a default Drupal development environment in a Jiffy. they do this by providing a bunch of puppetmanifests that sets up a working environment.

And the second one is Ariadne which is a standardized virtual machine development evironment for easily developing Drupal sites in a local sandbox that is essentially identical to a fully-configured hosted solution. It attempts to emulate a dedicated Acquia/Pantheon server as closely as possible, with added development tools. Project Ariadne is just like the examples from Jochen Lillich based on Chef

With all of these tools and examples around , there should be no excuses anymore for Drupal developers to hack on their own machine and tell the systems people "It works on my machine" (let alone to hack in production).

My PHPNE Talk on Vagrant

I’m not a public speaker. In fact, i’m normally found either sitting at the back behind a sound desk or running round fixing technical problems.

However, Anthony Sterling approached me the week before the May 2012 PHP North East meetup and asked if i’d do a talk on Vagrant.

In order to give something back to the group, and knowing it was a product I really like that i’d be able to come up with plenty of content, I agreed.

I titled the talk Virtualized Development Environments with Vagrant so I could go into the background and why you would want to use such a tool, as well as how to use it. I also gave a brief introduction to Intrastructure as Code and Configuration Management with Puppet and Chef.

The full contents were:

  • Introduction to:
    • Development Environments
    • Virtualization
  • Why virtualize your development environment?
  • Introduction to Vagrant
  • Using Vagrant
  • Vagrant Plugins
  • Automated Provisioning (Configuration Management)

Watching it back, my public speaking and presentation skills do need quite a bit of work. However, given that it was a first attempt at public speaking, which is not natural to me, hopefully it was a good attempt. The feedback was good, plenty of people had positive things to say, questions and further discussion in the pub afterwards so that nice. I now also have some points to improve on.

The talk was video’d and available on Vimeo:

And the slides available on Slideshare or SpeakerDeck: