↓ Archives ↓

Archive → July, 2010

Puppet Camp – San Francisco 2010

Making machine metadata visible

I’m quite the fan of data, metadata and querying these to interact with my infrastructure rather than interacting by hostnames and wanted to show how far I am down this route.

This is more an iterative ongoing process than a fully baked idea at this point since the concept of hostnames is so heavily embedded in our Sysadmin culture. Today I can’t yet fully break away from it due to tools like nagios etc still relying heavily on the hostname as the index but these are things that will improve in time.

The background is that in the old days we attempted to capture a lot of metadata in hostnames, domain names and so forth. This was kind of OK since we had static networks with relatively small amounts of hosts. Today we do ever more complex work on our servers and we have more and more servers. The advent of cloud computing has also brought with it a whole new pain of unpredictable hostnames, rapidly changing infrastructures a much bigger emphasis on role based computing.

My metadata about my machines comes from 3 main sources:

  • My Puppet manifests – classes and modules that gets put on a machine
  • Facter facts with the ability to add many per machine easily
  • MCollective stores the meta data in a MongoDB and let me query the network in real time

Puppet manifests based on query

When setting up machines I keep some data like database master hostnames in extlookup but in many cases I am now moving to a search based approach to finding resources. Here’s a sample manifest that will find the master database for a customers development machines:

$masterdb = search_nodes("{'facts.customer': '${customer}', 'facts.environment':${environment}, classes: 'mysql::master'}")

This is MongoDB query against my infrastructure database, it will find for a given node the name of a node that has the class mysql::master on it, by convention there should be only one per customer in my case. When using it in a template I can get back full objects with all the meta data for a node. Hopefully with Puppet 2.6 I can get full hashes into puppet too!

Making Metadata Visible

With machines doing a lot of work, filling a lot of roles etc and with more and more machines you need to be able to tell immediately what machine you are on.

I do this in several places, first my MOTD can look something like this:

   Welcome to Synchronize Your Dogmas 
            hosted at Hetzner, Germany
 
        Puppet Modules:
                - apache
                - iptables
                - mcollective member
                - xen dom0 skeleton
                - mw1.xxx.net virtual machine

I build this up using snippet from my concat module, each important module like apache can just put something like this in:

motd::register{"Apache Web Server": }

Being managed by my snippet library, if you just remove the include line from the manifests the MOTD will automatically update.

With a big block of welcome done, I now need to also be able to show in my prompts what a machine does, who its for a importantly what environment it is in.

Above a shot of 2 prompts in different environments, you see customer name, environment and major modules. Like with the motd I have a prompt::register define that module use to register into the prompt.

SSH Based on Metadata

With all this meta data in place, mcollective rolled out and everything integrated it’s very easy to now find and access machines based on this.

MCollective does real time resource discovery, so keeping with the mysql example above from puppet:

$ mc-ssh -W "environment=development customer=acme mysql::master"
Running: ssh db1.acme.net
Last login: Thu Jul 29 00:22:58 2010 from xxxx

$

Here i am ssh’ing to a server based on a query, if it found more than one machine matching the query a menu would be presented offering me a choice.

Monitoring Based on Metatdata

Finally setting up monitoring and keeping it in sync with reality can be a big challenge especially in dynamic cloud based environments, again I deal with this through discovery based on meta data:

$ check-mc-nrpe -W "environment=development customer=acme mysql::master"  check_load
check_load: OK: 1 WARNING: 0 CRITICAL: 0 UNKNOWN: 0|total=1 ok=1 warn=0 crit=0 unknown=0 checktime=0.612054

Summary

This is really the tip of the ice berg, there is a lot more that I already do – like scheduling puppet runs on groups of machines based on metadata – but also a lot more to do this really is early days down this route. I am very keen to get views from others who are struggling with shortcomings in hostname based approaches and how they deal with it.

Devops Meetups and Devops Dojos

Devopsday USA 2010 and the first Silicon Valley Devops Meetup

In late june/early july this year I went to San Francisco for devopsday USA 2010 that I had the pleasure to co-organized with Damon Edwards, Patrick Debois and Andrew Shafer.

I really enjoyed the experience and am glad so many people came to attend the conference (spéciale dédidace to the French Diaspora: Alexis, Olivier, Patrice and Jérôme). I look forward now for another chance to contribute to the next events!

I was still in the Bay Area on july the 6th when the first Silicon Valley Devops Meetup was organized by Dave Nielsen in Mountain View and I decided to join attend their first meetup.

Although Patrick and I have been in contact and working on presentations about "Agile and Operations" and "Continuous Deployment pipelines" months before he pinned the devops term and decided to create the first Devopsdays, we don't live close enough to one another to be able to see each other regularly, and I don't know yet enough devops-minded people locally to be able to start regular meetups, so it was interesting for me to to see what form it would take (sadly I haven't managed to attend to the popular london meetups yet).

The meetup started with a little discussion on the group name and on what the content and form should be for the following meetups.

The first Devopsdays was a 2 days conference with speakers in the morning and openspaces/unconference in the afternoon. I felt it was a nice format since the morning presentation would raise interest on specific subjects and fuel the afternoon debates without restricting them. (We were more constrained by time -only one day- for devopsday USA 2010 and had plenty of speakers so we decided to only have panels and a few lightning talks to raise interest/awareness to other subjects.)

I guess I felt that I was passing on the torch somehow and since some of the topics and discussions that took place during (and after) Devopsdays came back, it was an opportunity to share what was said and done back then. I think that the biggest benefit the devops movement is that it enables people to share their experience with one another, and I believe this is one of the way we can solve the problem I addressed on my first post.

One of the things from Ghent that I mentioned was the very nice experiment by Lindsay Holmwood when he proposed a 1-hour gang-development session on "cucumber as a script language". Not only because the subject was cool, but also because there was actually concrete code produced after this session, and I believe this is great if we can not only exchange ideas but also produce something that goes in the right direction.

Even though the devops movement is very much about people, about having the right mindsets, about breaking silos and about business alignment and change management, it is also about tools. And I think that since developers and ops (and network and security and QA) people meet together during the meetups and conferences, it is also probably the right place for new tools to emerge, tools that can efficiently and elegantly solve the daily pain points and bring people together/help them concentrate on what's really important.

This is why I was really happy to see that the meetup then followed by a nice presentation by Alex Honor on the "devops toolchain project". I'm glad I had the opportunity to meet Alex several time during my stay in the USA as he also had been thinking about those issues for a long time. His work on the toolchain helps pointing the gaps, the same way the "missing tools?" session during Ghent's Devopsdays did and there is a lot to do!

Devops Dojos?

Before I was involved in the devops movement, I was very much influenced by the Agile community, thanks to my friend Raphaël Pierquin (I also met Patrick thanks to him). He is the one who introduced me to the notion of "Coding Dojos".

I'm not sure who invented the Coding Dojos in the first place (it might have been Laurent Bossavit and al), but the idea is roughly "how come you are supposed to become a java expert after a one-week course when it takes a life time of regular training to become a martial art expert?", and as a martial art practitioner myself I find this idea sound.

Still, while I'm sure regular trainings on devops ideas makes sense, I'm not sure exactly how this should be done:

  • Do we need to train on a specific problem, a specific tool or on a specific method?
  • Maybe we could do retrospectives on a problem we've had and the solution we've implemented, to see how others would have fixed it?
  • Maybe this could be an opportunity to design a tool that would solve a specific problem, or a modification on an existing tool so it would be a better fit?

If you guys have an idea about this, I'd be really interested hearing it!

Devops Meetups and Devops Dojos

Devopsday USA 2010 and the first Silicon Valley Devops Meetup

In late june/early july this year I went to San Francisco for devopsday USA 2010 that I had the pleasure to co-organized with Damon Edwards, Patrick Debois and Andrew Shafer.

I really enjoyed the experience and am glad so many people came to attend the conference (spéciale dédidace to the French Diaspora: Alexis, Olivier, Patrice and Jérôme). I look forward now for another chance to contribute to the next events!

I was still in the Bay Area on july the 6th when the first Silicon Valley Devops Meetup was organized by Dave Nielsen in Mountain View and I decided to join attend their first meetup.

Although Patrick and I have been in contact and working on presentations about "Agile and Operations" and "Continuous Deployment pipelines" months before he pinned the devops term and decided to create the first Devopsdays, we don't live close enough to one another to be able to see each other regularly, and I don't know yet enough devops-minded people locally to be able to start regular meetups, so it was interesting for me to to see what form it would take (sadly I haven't managed to attend to the popular london meetups yet).

The meetup started with a little discussion on the group name and on what the content and form should be for the following meetups.

The first Devopsdays was a 2 days conference with speakers in the morning and openspaces/unconference in the afternoon. I felt it was a nice format since the morning presentation would raise interest on specific subjects and fuel the afternoon debates without restricting them. (We were more constrained by time -only one day- for devopsday USA 2010 and had plenty of speakers so we decided to only have panels and a few lightning talks to raise interest/awareness to other subjects.)

I guess I felt that I was passing on the torch somehow and since some of the topics and discussions that took place during (and after) Devopsdays came back, it was an opportunity to share what was said and done back then. I think that the biggest benefit the devops movement is that it enables people to share their experience with one another, and I believe this is one of the way we can solve the problem I addressed on my first post.

One of the things from Ghent that I mentioned was the very nice experiment by Lindsay Holmwood when he proposed a 1-hour gang-development session on "cucumber as a script language". Not only because the subject was cool, but also because there was actually concrete code produced after this session, and I believe this is great if we can not only exchange ideas but also produce something that goes in the right direction.

Even though the devops movement is very much about people, about having the right mindsets, about breaking silos and about business alignment and change management, it is also about tools. And I think that since developers and ops (and network and security and QA) people meet together during the meetups and conferences, it is also probably the right place for new tools to emerge, tools that can efficiently and elegantly solve the daily pain points and bring people together/help them concentrate on what's really important.

This is why I was really happy to see that the meetup then followed by a nice presentation by Alex Honor on the "devops toolchain project". I'm glad I had the opportunity to meet Alex several time during my stay in the USA as he also had been thinking about those issues for a long time. His work on the toolchain helps pointing the gaps, the same way the "missing tools?" session during Ghent's Devopsdays did and there is a lot to do!

Devops Dojos?

Before I was involved in the devops movement, I was very much influenced by the Agile community, thanks to my friend Raphaël Pierquin (I also met Patrick thanks to him). He is the one who introduced me to the notion of "Coding Dojos".

I'm not sure who invented the Coding Dojos in the first place (it might have been Laurent Bossavit and al), but the idea is roughly "how come you are supposed to become a java expert after a one-week course when it takes a life time of regular training to become a martial art expert?", and as a martial art practitioner myself I find this idea sound.

Still, while I'm sure regular trainings on devops ideas makes sense, I'm not sure exactly how this should be done:

  • Do we need to train on a specific problem, a specific tool or on a specific method?
  • Maybe we could do retrospectives on a problem we've had and the solution we've implemented, to see how others would have fixed it?
  • Maybe this could be an opportunity to design a tool that would solve a specific problem, or a modification on an existing tool so it would be a better fit?

If you guys have an idea about this, I'd be really interested hearing it!

Installing Cobbler on Ubuntu

I’ve started to make some changes to the Cobbler tree to enable it to run on other linux distros than the RedHat family. First up, Ubuntu Lucid! Install the required packages: aptitude install python-yaml python-cheetah python-netaddr python-urlgrabber Get the source code from my git tree: git clone git://github.com/proffalken/cobbler.git change to the new directory and run [...]

Monitoring ActiveMQ

I have a number of ActiveMQ servers, 7 in total, 3 in a network of brokers the rest standalone. For MCollective I use topics extensively so don’t really need to monitoring them much other than for availability. I also though do a lot of Queued work where lots of machines put data in a queue and others process the data.

In the Queue scenario you absolutely need to monitor queue sizes, memory usage and such. You also need to graph things like rates of messages, consumer counts and memory use. I am busy writing a number of Nagios and Cacti plugins to help with this, you can find them on Github.

To use these you need to have the ActiveMQ Statistics Plugin enabled.

First we need to monitor queue sizes:

$ check_activemq_queue.rb --host localhost --user nagios --password passw0rd --queue exim.stats --queue-warn 1000 --queue-crit 2000
OK: ActiveMQ exim.stats has 1 messages

This will connect to localhost monitoring a queue exim.stats warning you when it’s got 1000 messages and critical at 2000.

I need to add to this the ability to monitor memory usage, this will come over the next few days.

I also have a plugin for Cacti it can output stats for the broker as a whole and also for a specific queue. First the whole broker:

$ activemq-cacti-plugin.rb --host localhost --user nagios --password passw0rd --report broker
stomp+ssl:stomp+ssl storePercentUsage:81 size:5597 ssl:ssl vm:vm://web3 dataDirectory:/var/log/activemq/activemq-data dispatchCount:169533 brokerName:web3 openwire:tcp://web3:6166 storeUsage:869933776 memoryUsage:1564 tempUsage:0 averageEnqueueTime:1623.90502285799 enqueueCount:174080 minEnqueueTime:0.0 producerCount:0 memoryPercentUsage:0 tempLimit:104857600 messagesCached:0 consumerCount:2 memoryLimit:20971520 storeLimit:1073741824 inflightCount:9 dequeueCount:169525 brokerId:ID:web3-44651-1280002111036-0:0 tempPercentUsage:0 stomp:stomp://web3:6163 maxEnqueueTime:328585.0 expiredCount:0

Now a specific queue:

$ activemq-cacti-plugin.rb --host localhost --user nagios --password passw0rd --report exim.stats
size:0 dispatchCount:168951 memoryUsage:0 averageEnqueueTime:1629.42897052992 enqueueCount:168951 minEnqueueTime:0.0 consumerCount:1 producerCount:0 memoryPercentUsage:0 destinationName:queue://exim.stats messagesCached:0 memoryLimit:20971520 inflightCount:0 dequeueCount:168951 expiredCount:0 maxEnqueueTime:328585.0

Grab the code on GitHub and follow there, I expect a few updates in the next few weeks.

cucumber-vhost goes for release 0.1!

I’ve noticed that some people have shown an interest in my cucumber-vhost project. I’ve just made a few commits that now mean you have steps to tear down virtual servers as well as create them and, as a result, I’m calling this version 0.1 and releasing it for general usage. There are still a few [...]

Self Serve Dev Environment

Introduction

I am currently working in a web development shop. We develop and maintain a range of websites/webapps for customers.
At $WORK we have many varied customers each with their own unique Production Environments (PE from here on in).
Our $DEVs are also free to run any *nix based OS on their workstations.
One of the challeges we have had in the past is making a copy of a customer's PE easily available to our $DEVS.
This used to require lodging a task in Redmine and waiting for a Systems Admin to build you a VM on a central VM server.
This post will show how we are now using common FOSS tools to give enable to $DEVS to have VM's on their own workstations that mimick a customer's PE.

Whilst I make mention of some specific tools in this post they can be swapped out in most places for alternates however I have not tested the
alternatives yet. ie: puppet/chef, mercurial/git, centos/ubuntu.

Notes on the Challenge

  • needs to be simple not an obsticle - if it's slower than just getting a sysadmin to build it for you then its a fail
  • simplicity generally means easy to fix when something goes wrong in the wheel.
  • self-serve - no waiting for sysadmins
  • visability - everything in DVCS and Redmine (project managent software)
  • needed to be repeatable - $DEVs needed to easily be able to build, destroy and build again
  • relatively self documenting - read the kickstart or puppet manifests
  • I hate OS images - They're big, cumbersome and pain in my..err..storage See - Golden Image or Foil Ball?

After spending a fair amount of time on this and looking at many of the VM/cloud management solutions out there I have decided that while some are very nice and useful I do not believe they are suiteable for our situation. Most VM/cloud management tools are built around the "OS Image" and require each workstation to 'register' as a node.

Current solution

After doing the full circle of research we are now using a simple collection of existing tools.
It was all there staring me in the face all along. Libvirt, virt-install kickstart puppet, mercurial and a wiki entry. A $DEV just needs to make sure he/she has libvirt, virt-install, virt-viewer installed.
We are using KVM to provide the virtualisation layer but through the use of libvirt you should be able to use any libvirt compatible virtualisation provider.(virtual box etc)

Technologies used

  • a httpd server (nginx, apache etc) - to serve kickstart + yum repos/mirror
  • Own yum repos + centos mirror ( again ubuntu mirror etc )
  • puppetmasterd ( or other CF tool ie: chef etc ) with autosign turned on (we have a separate puppetmaser for the $DEVS)
  • some kickstart files - I use one per customer and bootstrap puppet from the %POST section
  • libvirtd + KVM/qemu - could be any supported virtualisation software supported by libvirt
  • python-virtinst + virt-viewer
  • dhcpd
  • forward and reverse dns - puppet will fail to work as expected without it ( I use powerdns-recursor for demos as it exports /etc/hosts )
  • redmine - we make use of Redmine's ACL's to visualize the repos for puppet and kickstart files per customer

Devs

The following is the steps needed for a $DEV to deploy a customer's PE.

  • Check network page and grab an available network mac to use (this is used for dhcp & dns so puppet works properly)
    and the name of the customers kickstart file.
  • update wiki page to say that network mac is in use.
  • deploy a VM on their workstation. - See Libvirt tips

    virt-install --connect qemu:///system --accelerate -n virt01 -m 54:52:00:37:2E:B9 -r 1024 --vcpus=1 --disk pool=lvm,bus=virtio,size=20 --vnc --os-type linux --os-variant=rhel5 --network=network:default -l http://192.168.1.250/os/CentOS/5.5/os/x86_64/ -x "ks=http://192.168.1.250/ks/project_customer1.ks"

This will take advantage of the fact that both CentOS and Ubuntu have the necessary PXE files stored in their mirrors for booting the installer.

  • wait approx 10 or so minutes and they have a clone of the customer's PE on their workstation ready to deploy to and hack on. see notes in conclusion below

New customers

The following is what's involved in preparing for a new customer:

  • A new customer has a VM/server provisioned in a DC by a hosting company.
  • I grab the current package list and make a kickstart file to replicate the install locally
  • Create a new project_customer3 in puppet and add details to bottom of the new kickstart file.
  • publish new kickstart file and update wiki entry

Conclusion

I have reduced the time it takes for a dev to get a copy of a customers PE down from days to minutes and its now a self serve solution.
There is still more to refine in this but it's already full of win as I now get to do more of 'stuff that matters'

It's early days for us using this new setup and I am yet to work out an easy, effective way of notifying a $DEV when puppet has finished the buildout. Suggestions welcome.

  • cucumber tests ?
  • using libnotify via Dbus ? ( suggestion made at a recent DevOps Sydney meetup )
  • ??

DevOps (live) at OSCON

Early reports from OSCON are that DevOps is a topic of much discussion. My fellow dev2ops.org contributor Alex Honor and I are headed to Portland this morning to give DevOps related talks at OSCON. If you are there Wednesday or Thursday, please come by and say hello!

 

Wednesday (7/21) 1:40pm in room Portland 251 is Alex's presentation...
Open Source Tool Chains for Cloud Computing 

Thursday (7/22) 10:40am in room D135 is Damon's presentation... 
The IT Philharmonic: How Out of Tune Are Your Operations? 

 

Both talks feature lots of new content (even though the titles and outdated descriptions on the OSCON site are similar to our Velocity talks) 

 

 

Puppet 2.6.0 is here! It’s alive!

The journey was long and arduous and many fell along the way but
Labs is proud to announce the 2.6.0 release!

The 2.6.0 release is a major feature release and includes a huge variety
of new features, fixes, updates and enhancements. These include the
complete cut-over from XMLRPC to the REST API, numerous language
enhancements, a complete rewrite of the events and reporting system, an
internal Ruby DSL, a single binary, Windows support, a new HTTP report
processor, and a myriad of other enhancements.

We’ve included release notes below that you can also see at:

http://projects.puppetlabs.com/projects//wiki/Release_Notes

The release is available for download at:

http://puppetlabs.com/downloads//-2.6.0.tar.gz

And I am sure packagers will be hard at work in the not to distant future!