↓ Archives ↓

Archive → June, 2011

Facter facts from TXT, JSON, YAML and non ruby scripts

There has been many discussions about Facter 2, one of the things I looked forward to getting was the ability to read arbitrary files or run arbitrary scripts in a directory to create facts.

This is pretty important as a lot of the typical users just aren’t Ruby coders and really all they want is to trivially add some facts. All too often the answer to common questions in the Puppet user groups ends up being “add a fact” but when they look at adding facts its just way too daunting.

Sadly as of today Facter 2 is mostly vaporware so I created a quick – really quick – fact that reads the directory /etc/facts.d and parse Text, JSON or YAML files but can also run any executable in there.

To write a fact in a shell script that reads /etc/sysconfig/kernel on a RedHat machine and names the default kernel package name simply do this:

#!/bin/sh
 
source /etc/sysconfig/kernel
 
echo "default_kernel=${DEFAULTKERNEL}"

Add it to /etc/facts.d and make it executable, now you can simple use your new fact:

% facter default_kernel
kernel-xen

Simple stuff. You can get the fact on my GitHub and deploy it using the standard method.

Puppet and DataDog

Some of you may have seen the awesome new metrics, graphing and event platform from DataDog. It’s currently in beta and well worth a look. I saw it at Velocity two weeks ago and knew I had to write an integration with . So here ’tis! I’ve added a Puppet report processor that sends metrics and events to the DataDog API.

To use it install the `dogapi` gem on your Puppet master

$ sudo gem install dogapi

Then install puppet- as a module in your Puppet master’s modulepath and update the `_api_key` variable in the `.yaml` file with your API key and copy the file to `/etc/puppet/`.

Enable pluginsync and on your master and clients in `puppet.conf`

        [master]
        report = true
        reports = datadog
        pluginsync = true
        [agent]
        report = true
        pluginsync = true

Finally run the Puppet client and sync the report as a plugin.

P.S. Also available is a DataDog Puppet module to install the Datadog agent.

Puppet and DataDog

Some of you may have seen the awesome new metrics, graphing and event platform from DataDog. It’s currently in beta and well worth a look. I saw it at Velocity two weeks ago and knew I had to write an integration with Puppet. So here ‘tis! I’ve added a Puppet report processor that sends metrics and events to the DataDog API. To use it install the dogapi gem on your Puppet master

$ sudo gem install dogapi

Then install puppet-datadog as a module in your Puppet master’s modulepath and update the datadog_api_key variable in the datadog.yaml file with your Datadog API key and copy the file to /etc/puppet/. Enable pluginsync and reports on your master and clients in puppet.conf

        [master]
        report = true
        reports = datadog
        pluginsync = true
        [agent]
        report = true
        pluginsync = true

Finally run the Puppet client and sync the report as a plugin.

P.S. Also available is a DataDog Puppet module to install the Datadog agent.

Video with John Allspaw at Etsy: What to put on the big dashboards

I had the pleasure of visting the folks at Etsy recently. Etsy is well known in DevOps circles for their Continuous Deploymet and Metrics efforts.

I'll be posting more videos soon, but in the meantime here is a quick hit I recorded with John Allspaw while he gave me a tour of their Brooklyn HQ. I had just asked John about how they decide what to put on the big dashboards on the wall.

Here is what he told me...

Command-line cookbook dependency solving with knife exec

Imagine you have a fairly complicated infrastructre with a large number of nodes and roles. Suppose you have a requirement to take one of the nodes and rebuild it in an entirely new network, perhaps even for a completely different organization. This should be easy, right? We have our infrastructure in the form of code. However, our current infrastructure has hundreds of uploaded cookbooks - how do we know the minimum ones to download and move over? We need to find out from a node exactly what cookbooks are needed for that node to be built.

The obvious place to start is with the node itself:

$ knife node show controller
Node Name:   controller
Environment: _default
FQDN:        controller
IP:          182.13.194.41
Run List:    role[base], recipe[apt::cacher], role[pxe_server]
Roles:       pxe_server, base
Recipes      apt::cacher, pxe_dust::server, dhcp, dhcp::config
Platform:    ubuntu 10.04

OK, this tells us we need the apt, pxe_dust and dhcp cookbooks. But what about them - do they have any dependencies? How could we find out? Well, dependencies are specified in two places - in the cookbook metadata, and in the individual recipes. Here's a primitive way to illustrate this:

bash-3.2$ for c in apt pxe_dust dhcp
> do
> grep -iER 'include_recipe|^depends' $c/* | cut -d '"' -f 2 | sort | uniq
> done
apt::cacher-client
apache2
pxe_dust::server
tftp
tftp::server
utils

As I said - primitive. However the problem doesn't end here. In order to be sure, we now need to repeat this for each dependency, recursively. And of course it would be nice to present them more attractively. Thinking about it, it would be rather useful to know what cookbook versions are in use too. This is definitely not a job for a shell one liner - is there a better way?

As it happens, there is. Think about it - the Chef server already needs to solve these dependencies to know what cookbooks to push to API clients. Can we access this logic? Of course we can - clients carry out all their interactions with the Chef server via the API. This means we can let the server solve the dependencies and query it via the API ourselves.

Chef provides two powerful ways to access the API without having to write a RESTful client. The first, Shef, is an interactive REPL based on IRB, which when launched gives access to the Chef server. This isn't trivial to use. The second, much simpler way is the knife exec subcommand. This allows you to write Ruby scripts or simple one-liners that are executed in the context of a fully configured Chef API Client using the knife configuration file.

knife exec -E '(api.get "nodes/controller/cookbooks").each { |cb| pp cb[0] => cb[1].version }'

The /nodes/NODE_NAME/cookbooks endpoint returns the cookbook attributes, definitions, libraries and recipes that are required for this node. The response is a hash of cookbook name and Chef::CookbookVersion object. We simply iterate over each one, and pretty print the cookbook name and the version.

Let's give it a try:

$ knife exec -E '(api.get "nodes/controller/cookbooks").each { |cb| pp cb[0] => cb[1].version }'
{"apt"=>"1.1.1"}
{"tftp"=>"0.1.0"}
{"apache2"=>"0.99.3"}
{"dhcp"=>"0.1.0"}
{"utils"=>"0.9.5"}
{"pxe_dust"=>"1.1.0"}

Nifty! :)

Command-line cookbook dependency solving with knife exec

Imagine you have a fairly complicated infrastructre with a large number of nodes and roles. Suppose you have a requirement to take one of the nodes and rebuild it in an entirely new network, perhaps even for a completely different organization. This should be easy, right? We have our infrastructure in the form of code. However, our current infrastructure has hundreds of uploaded cookbooks - how do we know the minimum ones to download and move over? We need to find out from a node exactly what cookbooks are needed for that node to be built.

The obvious place to start is with the node itself:

$ knife node show controller
Node Name:   controller
Environment: _default
FQDN:        controller
IP:          182.13.194.41
Run List:    role[base], recipe[apt::cacher], role[pxe_server]
Roles:       pxe_server, base
Recipes      apt::cacher, pxe_dust::server, dhcp, dhcp::config
Platform:    ubuntu 10.04

OK, this tells us we need the apt, pxe_dust and dhcp cookbooks. But what about them - do they have any dependencies? How could we find out? Well, dependencies are specified in two places - in the cookbook metadata, and in the individual recipes. Here’s a primitive way to illustrate this:

bash-3.2$ for c in apt pxe_dust dhcp
> do
> grep -iER 'include_recipe|^depends' $c/* | cut -d '"' -f 2 | sort | uniq
> done
apt::cacher-client
apache2
pxe_dust::server
tftp
tftp::server
utils

As I said - primitive. However the problem doesn’t end here. In order to be sure, we now need to repeat this for each dependency, recursively. And of course it would be nice to present them more attractively. Thinking about it, it would be rather useful to know what cookbook versions are in use too. This is definitely not a job for a shell one liner - is there a better way?

As it happens, there is. Think about it - the Chef server already needs to solve these dependencies to know what cookbooks to push to API clients. Can we access this logic? Of course we can - clients carry out all their interactions with the Chef server via the API. This means we can let the server solve the dependencies and query it via the API ourselves.

Chef provides two powerful ways to access the API without having to write a RESTful client. The first, Shef, is an interactive REPL based on IRB, which when launched gives access to the Chef server. This isn’t trivial to use. The second, much simpler way is the knife exec subcommand. This allows you to write Ruby scripts or simple one-liners that are executed in the context of a fully configured Chef API Client using the knife configuration file.

knife exec -E '(api.get "nodes/controller/cookbooks").each { |cb| pp cb[0] => cb[1].version }'

The /nodes/NODE_NAME/cookbooks endpoint returns the cookbook attributes, definitions, libraries and recipes that are required for this node. The response is a hash of cookbook name and Chef::CookbookVersion object. We simply iterate over each one, and pretty print the cookbook name and the version.

Let’s give it a try:

$ knife exec -E '(api.get "nodes/controller/cookbooks").each { |cb| pp cb[0] => cb[1].version }'
{"apt"=>"1.1.1"}
{"tftp"=>"0.1.0"}
{"apache2"=>"0.99.3"}
{"dhcp"=>"0.1.0"}
{"utils"=>"0.9.5"}
{"pxe_dust"=>"1.1.0"}

Nifty! :)

Command-line cookbook dependency solving with knife exec

Imagine you have a fairly complicated infrastructre with a large number of nodes and roles. Suppose you have a requirement to take one of the nodes and rebuild it in an entirely new network, perhaps even for a completely different organization. This should be easy, right? We have our infrastructure in the form of code. However, our current infrastructure has hundreds of uploaded cookbooks - how do we know the minimum ones to download and move over? We need to find out from a node exactly what cookbooks are needed for that node to be built.

The obvious place to start is with the node itself:

$ knife node show controller
Node Name:   controller
Environment: _default
FQDN:        controller
IP:          182.13.194.41
Run List:    role[base], recipe[apt::cacher], role[pxe_server]
Roles:       pxe_server, base
Recipes      apt::cacher, pxe_dust::server, dhcp, dhcp::config
Platform:    ubuntu 10.04

OK, this tells us we need the apt, pxe_dust and dhcp cookbooks. But what about them - do they have any dependencies? How could we find out? Well, dependencies are specified in two places - in the cookbook metadata, and in the individual recipes. Here’s a primitive way to illustrate this:

bash-3.2$ for c in apt pxe_dust dhcp
> do
> grep -iER 'include_recipe|^depends' $c/* | cut -d '"' -f 2 | sort | uniq
> done
apt::cacher-client
apache2
pxe_dust::server
tftp
tftp::server
utils

As I said - primitive. However the problem doesn’t end here. In order to be sure, we now need to repeat this for each dependency, recursively. And of course it would be nice to present them more attractively. Thinking about it, it would be rather useful to know what cookbook versions are in use too. This is definitely not a job for a shell one liner - is there a better way?

As it happens, there is. Think about it - the Chef server already needs to solve these dependencies to know what cookbooks to push to API clients. Can we access this logic? Of course we can - clients carry out all their interactions with the Chef server via the API. This means we can let the server solve the dependencies and query it via the API ourselves.

Chef provides two powerful ways to access the API without having to write a RESTful client. The first, Shef, is an interactive REPL based on IRB, which when launched gives access to the Chef server. This isn’t trivial to use. The second, much simpler way is the knife exec subcommand. This allows you to write Ruby scripts or simple one-liners that are executed in the context of a fully configured Chef API Client using the knife configuration file.

knife exec -E '(api.get "nodes/controller/cookbooks").each { |cb| pp cb[0] => cb[1].version }'

The /nodes/NODE_NAME/cookbooks endpoint returns the cookbook attributes, definitions, libraries and recipes that are required for this node. The response is a hash of cookbook name and Chef::CookbookVersion object. We simply iterate over each one, and pretty print the cookbook name and the version.

Let’s give it a try:

$ knife exec -E '(api.get "nodes/controller/cookbooks").each { |cb| pp cb[0] => cb[1].version }'
{"apt"=>"1.1.1"}
{"tftp"=>"0.1.0"}
{"apache2"=>"0.99.3"}
{"dhcp"=>"0.1.0"}
{"utils"=>"0.9.5"}
{"pxe_dust"=>"1.1.0"}

Nifty! :)

Kohsuke Kawaguchi presents Jenkins to Silicon Valley DevOps Meetup (Video)

Kohsuke Kawaguchi stopped by the Silicon Valley DevOps Meetup on June 7, 2011 to give an in-depth tour of Jenkins. Kohsuke is the founder of both the Hudson and the Jenkins open source projects and now works for CloudBees.

Kohsuke's presentation covered not only the Jenkins basics but also more advanced topics like distributed builds in the cloud, the matrix project, and build promotion. Video and slides are below.

Once again, thanks to Box.net for hosting the event!

Puppet backend for Hiera part 2

Note: This project is now being managed by Puppetlabs, its new home is http://projects.puppetlabs.com/projects/hiera

Last week I posted the first details about my new data source for Hiera that enables its use in Puppet.

In that post I mentioned I want to do merge or array searches in the future. I took a first stab at that for array data and wanted to show how that works. I also mentioned I wanted to write a Hiera External Node Classifier (ENC) but this work completely makes that redundant now in my mind.

A common pattern you see in ENCs are that they layer data – very similar in how extlookup / hiera has done it – but that instead of just doing a first-match search they combine the results into a merged list. This merged list is then used to include the classes on the nodes.

For a node in the production environment located in dc1 you’ll want:

node default {
   include users::common
   include users::production
   include users::dc1
}

I’ve made this trivial in Hiera now, given the 3 files below:

common.json
{"classes":"users::common"}

production.json
{"classes":"users::production"}

dc1.json
{"classes":"users::dc1"}

And appropriate Hiera hierarchy configuration you can achieve this using the node block below:

node default {
   hiera_include("classes")
}

Any parametrized classes that use Hiera as in my previous post will simply do the right thing. Individual classes variables can be arrays so you can include many classes at each tier. Now just add a role fact on your machines, add a role tier in Hiera and you’re all set.

The huge win here is that you do not need to do any stupid hacks like load the facts from the Puppet Masters vardir in your ENC to access the node facts or any of the other hacky things people do in ENCs. This is simply a manifest doing what manifests do – just better.

The hiera CLI tool has been updated with array support, here is it running on the data above:

$ hiera -a classes
["users::common"]
$ hiera -a classes environment=production location=dc1
["users::common", "users::production", "users::dc1"]

I’ve also added a hiera_array() function that takes the same parameters as the hiera() function but that returns an array of the found data. The array capability will be in Hiera version 0.2.0 which should be out later today.

I should also mention that Luke Kanies took a quick stab at integrating Hiera into Puppet and the result is pretty awesome. Given the example below Puppet will magically use Hiera if it’s available, else fall back to old behavior.

class ntp::config($ntpservers="1.pool.ntp.org") {
    .
    .
}
 
node default {
   include ntp::config
}

With Lukes proposed changes this would be equivalent to:

class ntp::config($ntpservers=hiera("ntpservers", "1.pool.ntp.org")) {
    .
    .
}

This is pretty awesome. I wouldn’t hold my breath to see this kind of flexibility soon in Puppet core but it shows whats possible.

The case for Augeas

Ever since I met David Lutterkort over steaks at OLS 2007 augeas was this tool in the back of my mind that I couldn't place... I never saw the need for it... or it seemed to be huge overkill for the problem that needed solving .

Till I ran into sipxecs rewriting XML files on the fly .. and putting values in their XML that I could not trace back to an original source. As of Augeas 0.8.x there's an XML lens out there.

Digging innot blah.xml with augtool you can do stuff like

  1. set /augeas/load/Xml/incl[3] /tmp/blah.xml
  2. set /augeas/load/Xml/lens Xml.lns
  3. load
  4. print /files/tmp/blah.xml/profile/settings/param[17]/
  5. /files/tmp/blah.xml/profile/settings/param[17] = "#empty"
  6. /files/tmp/blah.xml/profile/settings/param[17]/#attribute
  7. /files/tmp/blah.xml/profile/settings/param[17]/#attribute/name = "sip-ip"
  8. /files/tmp/blah.xml/profile/settings/param[17]/#attribute/value = "10.255.202.90"
  9. augtool> print /files/tmp/blah.xml/profile/settings/param[18]/
  10. /files/tmp/blah.xml/profile/settings/param[18] = "#empty"
  11. /files/tmp/blah.xml/profile/settings/param[18]/#attribute
  12. /files/tmp/blah.xml/profile/settings/param[18]/#attribute/name = "ext-rtp-ip"
  13. /files/tmp/blah.xml/profile/settings/param[18]/#attribute/value = "auto-nat"
  14. augtool> print /files/tmp/blah.xml/profile/settings/param[16]/
  15. /files/tmp/blah.xml/profile/settings/param[16] = "#empty"
  16. /files/tmp/blah.xml/profile/settings/param[16]/#attribute
  17. /files/tmp/blah.xml/profile/settings/param[16]/#attribute/name = "rtp-ip"
  18. /files/tmp/blah.xml/profile/settings/param[16]/#attribute/value = "10.255.202.90"

and get and set

  1. augtool> get /files/etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml/profile/settings/param[17]/#attribute/value
  2. /files/etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml/profile/settings/param[17]/#attribute/value = 10.255.202.90
  3. augtool> set /files/etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml/profile/settings/param[16]/#attribute/value 10.0.0.2

Putting that into puppet however isn't that tvivial .

When you try to do this

  1. augeas{"sipxprofile" :
  2. changes => [
  3. "set /augeas/load/Xml/incl[last()+1] /etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml",
  4. "set /files/etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml/profile/settings/param[16]/#attribute/value 10.0.0.2",
  5. "set /files/etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml/profile/settings/param[17]/#attribute/value 10.0.0.2",
  6. ],
  7. }

Puppet really doesn't output what you want to do ... it only outputs the snippet you modify ..
It's the load statement above that is the really important piece but puppet can't directly work with that so you need to go around that using
The way to solve this is

  1. augeas{"sipxprofile" :
  2. lens => "Xml.lns",
  3. incl => "/etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml",
  4. context => "/files/etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml",
  5. changes => [
  6. "set profile/settings/param[16]/#attribute/value $ipaddress",
  7. "set profile/settings/param[17]/#attribute/value $ipaddress",
  8. ],
  9. onlyif => "get profile/settings/param[16]/#attribute/value != $ipaddress",
  10. }