↓ Archives ↓

Category → cfengine

Config Management & F***ing Shell Scripts

Joke tech projects pop up all the time. Some, like the lamentable titstare, should never have seen the light of day. Others, like Mark Zuckerberg’s Facemash start out as a joke and grow into something that is anything but. Most come and go without anyone really noticing. Whilst their lives are typically short we often learn more from people’s reactions to these projects than we do from their use of them. An excellent example of this popped up recently when a couple of frustrated Chef and Ansible users came up with a simple configuration management alternative, the regrettably named F***ing Shell Scripts (FSS). As they explain it, as in “Why can’t we just use f***ing shell scripts?”. Whilst usability issues with configuration management tools – particularly in the Enterprise – is a topic close to my heart, I would not normally have had much more than a passing interest in FSS. What grabbed my attention was the reaction it elicited when posted to Hacker News. With over 150 comments FSS...

FOSDEM 2014 is coming

and with that almost a full week of side events.
For those who don't know FOSDEM, (where have you been hiding for the past 13 years ? ) Fosdem is the annual Free and Open Source Developers European meeting. If you are into open source , you just can't mis this event where thousands of likeminded people will meet.

And if 2 days of FOSDEM madness isn't enough people organise events around it.

Last year I organised PuppetCamp in Gent, the days before Fosdem and a MonitoringLove Hackfest in our office the 2 days after FOSDEM This year another marathon is planned.

On Friday (31/1/2014) the CentOs community is hosting a Dojo in Brussels at the IBM Forum. (Free, but registration required by the venue)

After the success of PuppetCamp in Gent last year we decided to open up the discussion and get more Infrastructure as Code people involved in a CfgMgmtCamp.eu

The keynotes for CfgMgmtCamp will include the leaders of the 3 most popular tools around , both Mark Burgess, Luke Kanies and Adam Jacob will present at the event which will take place in Gent right after Fosdem. We expect people from all the major communities including, but not limited to , Ansible, Salt, Chef, Puppet, CFengine, Rudder, Foreman and Juju (Free but registration required for catering)

And because 3 events in one week isn't enough the RedHat Community is hosting their Infrastructure.next conference after CfgMgmtCamp at the same venue. (Free but registration required for catering)

cya in Belgium next year..

Infrastructure as Code – A comprehensive overview

I've been tracking infrastructure as code for a few years now. Over the years it has gotten closer to real code.

Close but no sigar yet.... We've come a long way but when you compare it to real languages it still feels in it's infancy. In this updated overview I gave at the ABUG, I went through:

  • the basic concepts of infrastructure as code
  • the differences/concepts in the languages (chef, puppet, ...)
  • the editors , syntax checkers, highlighting
  • integration with git version control
  • integration with CI systems
  • the different forms of testing (syntax, compile, unit, smoke testing)
  • using vagrant, veewee and the tools in that eco-system
  • debugging , profiling your code

This talk is probably the most comprehensive tool list that I've seen/made about the subject. But feel free to post and add your findings in the comments!

Note: that at the end of the presentation there are many extra links still to be sorted or slight outdated tools.

I've given previous versions of this talk at Devoxx 2012 and Jax2012. Enjoy the Jax2012 video here:

What if Config Management was created by Game Designers

Ever wondered what Config Management would look like if it was created by Game Designers?

Enjoy my Ignite session from Devopsdays London 2013

@Beaker on #Devops

Yesterday @beaker posted his ideas on the #devops movement ...

Apparently we haven't been stressing enough on the fact that it isn't just about Devs and Ops,
So let me repeat it's not just about Devs and Ops, it's about breaking silo's , about being good at our jobs, about getting conversation started, about talking to different stakeholders in the processes . We are absolutely trying to include all groups, not exclude some.

@beaker also seems to have seen many presentations where developers are shown to have evolved in practice and methodology, but operators (of all kinds) are described as being stuck in the dark ages. , is that a different point of view on another continent \, on this side of the Atlantic, it's mostly the Ops people that are already using agile methods spreading the word and it isn't about Devs talking about Deopvs yet. It's actually mostly the ops spreading the word because they feel most of the pain .

Hoff also wonders about routers switches firewall and all the other boxen where we aren't running puppet or chef on , the boxes that are left out of our fully automated environments .
Indeed, Puppetcamp Europe once again woke up the discussion on how to tackle these boxen, the lack of use of existing standards was covered .. and some mentioned that CIM and family are pretty much death or irrelevant for real life usage , both the Puppet and Chef communities are working on manifest, modules and recipes to solve the issues.

But the good thing is that we now have the security people involved too, maybe they'll figure out how to survive longer than 6 months in a CSO position if they talk to the others and come out of their Ivory towers :)

Trackback URL for this post:


Linux Open Administration Days 2010

So about 4 monts ago there was the crazy idea to start a new FOSS event in Belgium targeted at sysadmins.

What started out as an event for local people to meet local people with some local speakers actually ended up being a small local event with some top international speakers on onfiguration mananagement and system administration mixed with a bunch of good local ones !

I had the honour to open the conference with an extremely short version of the Devops talk I gave earlier last year.. extremely short as I knew that over the course of the weekend the topic would reoccur a lot.

We had the first european talk on Chef, by Joshua Timberman, and we had Puppet talks amongst by Dan Bode from Puppetlabs and CFengine talks , devops was a frequently dropped word,

We had a book raffle where we handed out O'Reilly's .. we had a great free pizza party (got the idea from the saturday pizza event at LCA 2005) , and we had some free beer. Sounds like a good combination for a geeky weekend.

Apart from the regular talks there were plenty of Open Spaces where interesting topics were discussed ... we had spaces on Open Source vs Open Core , strong voices were heard when we discussed what we should do with the Open Core companies that claim to value Open Source , some people think we should actually list the fauxpensource ones somewhere and make sure the world knows about them

We had an awesome configuration management discussion session discussing Chef vs Puppet vs CFengine . And much much more ...

Some people owe me plenty of Sushi as I had to do my MySQL HA talk before their Managing MySQL talk , but other than that .. things just went fine..

Trackback URL for this post:


Recipes for Automated installation of OS and beyond

Up until now, I've described the options to automate shell scripting, virtual machine creation and network provisioning. So now we can actually get started with automating the installation of the Operating System itself. This not surprisingly is were most sysadmins spent most of their time.

The good news is that we are slowly going a way from the custom scripting toward a reusable and shareable language. Very similar to programming design patterns. Over the years the methods have evolved towards configuration management, which is obviously a good thing.

This also helps in leaving the hero culture sysadmins have because they are the guys who now how to manage their systems.

Jump/Kick start

The first way of automating the OS installation came from using Kickstart or Jumpstart. In essence it is a config file that defines the information necessary to do an installation. It typically contains:

  • the machine's network configuration (interfaces, IP, Routes, DNS)
  • the disk layout setup
  • the packages to be installed
  • patches to be applied

When booting the kernel of an OS, there usually is a way to specify this file, allowing automated installation. Over time this file was put on a floppy, then on a CDROM, USB Drive. To automatically start the installation the install media needed to be modified to change the default kernel boot options. The syntax of these files is typically different per OS.

To create your first file, you would do a manual installation of the OS, and at the end it would create a file corresponding to the manual choices you made. After that, the file could be altered to include additional post-installation scripts or extra packages.

Automating the creation of this has never really been important, I was either using an Editor or using a GUI to generate them. Interestly I found Snake as an example to scripting the creation of the template.

So in other to automate the installation of software, sysadmin would encourage developers to create packages for their install, so that the installation could be easily automated.

An example of a Solaris Jumpstart:

# install_type MUST be first
install_type      initial_install
# start with the minimal required number of packages
cluster           SUNWCXall
cluster           SUNWCapache delete
cluster           SUNWCpcmc   delete
cluster           SUNWCpcmcx  delete
cluster           SUNWCthai   delete
cluster           SUNWClp     delete
cluster           SUNWCnis    delete
cluster           SUNWCppp    delete
# format the entire disk for Solaris
fdisk   all   solaris all
# define how the disk is partitioned
partitioning      explicit
filesys           rootdisk.s0 6144  /
filesys           rootdisk.s1 1024  swap
filesys           rootdisk.s7 free  /state/partition1
# install systems as standalone
system_type standalone
# specify patches to install
patch 119281-06 nfs
# specify packages to install
package SPROcc add nfs

System installation frameworks

Over time frameworks emerged to manage all these packages installation and managing :

Configuration Management

As System recipes and configuration management beautifully describes, managing systems is more then just managing and installing packages : sysadmins would have tons of custom scripts and HOWTO's to describe how to do additional things f.i. installing a complete Webserver according to company standards. All of this knowledge was usually put in specific custom scripts, without a good way of sharing this, . What we actually need is a formalized description of all the things that need to be present and configured on the machines in your setup.

Enter the world configuration management tools. These tools provide a ways of describing the recipes in a language that makes abstraction of the actual system its running on. F.i. adding a user or adding a package is abstracted in the language.

Examples are:

Puppet seems to have most of the market for now, but the Chef guys are catching up. Also a lot of the system management tools have puppet integration:

Example puppet recipe for ensuring the permission of a sudoers file

# /etc/puppet/manifests/classes/sudo.pp
class sudo {
    file { "/etc/sudoers":
        owner => "root",
        group => "root",
        mode  => 440,

Example for mysql node installation

class mysql-server {
     $password = "insert_password_here"
     package { "MySQL-client": ensure => installed }
     package { "MySQL-server": ensure => installed }
     package { "MySQL-shared": ensure => installed }

     exec { "Set MySQL server root password":
       subscribe => [ Package["MySQL-server"], Package["MySQL-client"], Package["MySQL-shared"] ],
       refreshonly => true,
       unless => "mysqladmin -uroot -p$password status",
       path => "/bin:/usr/bin",
       command => "mysqladmin -uroot password $password",

What's fascinating is the new set of tools that starts to emerge here and how it really starts to integrate in real programming languages such as ruby.

  • Moonshine an opensource configuration management and deployment system that follows the Rails way.
  • Gepetto A helper suite for Puppet projects to create, manage and help daily development
  • ShadowPuppet is a Ruby DSL for Puppet, extracted from Moonshine. ShadowPuppet provides a DSL for creating collections (manifests) of Puppet Resources in Ruby.
  • Carpet of the Agile Web Operations guys is an example on how you can mix capistrano with puppet.
  • Cft - Configuration File Tracking: lets you kinda record puppet recipes while entering commands.

For managing configuration files Augeas is doing a really nice job: it parses configuration files in their native formats and transforms them into a tree. Configuration changes are made by manipulating this tree and saving it back into native config files.

Just Enough Operating System

Now that sysadmins are spending more time with their configuration management, they slim down their initial kickstart installations by only using a minimal base install of the OS in their kickstart + a configuration management tool. And when this minimal install is done, they continue applying their recipes to the installation.

This minimal install is sometimes called a base install , an installation with the minimal set of packages installed. Some OS vendors are actively working on providing what is called a JeOS which stands for Just enough Operating System. This is an image that contains an already installed minimal install. They are also useful for creating appliances. Some example are:

Middleware automation

If you're installing a complete stack f.i. Oracle Application Server, Database Server, changes are that the initial package install does nothing. What you need is to create your own instances, pools and so on. What's important is that some of these include their own automation language. It is essential that they have a way of doing every GUI action from the commandline. Otherwise you end up writing scripts for managing their config files. This is a bad thing because this might change overtime or even worse impossible because they store things in binary format.

Examples of good API's are:

Example of asadmin scripting

   # create an cluster
   asadmin create-cluster --user admin --passwordfile adminpassword.txt --host hostname -port 4848 cluster1
   # create instance 1
   asadmin create-instance --user admin --passwordfile adminpassword.txt --host hostname -port 4848 --cluster cluster1 --nodeagent nodeagent1 --systemproperties "JMX_SYSTEM_CONNECTOR_PORT=8687:IIOP_LISTENER_PORT=3330:IIOP_SSL_LISTENER_PORT=4440:IIOP_SSL_ MUTUALAUTH_PORT=5550:HTTP_LISTENER_PORT=1110:HTTP_SSL_LISTENER_PORT=2220" instance1
   # create instance 2
   asadmin create-instance --user admin --passwordfile adminpassword.txt --host hostname -port 4848 --cluster cluster1 --nodeagent nodeagent1 --systemproperties "JMX_SYSTEM_CONNECTOR_PORT=8688:IIOP_LISTENER_PORT=3331:IIOP_SSL_LISTENER_PORT=4441:IIOP_SSL_ MUTUALAUTH_PORT=5551:HTTP_LISTENER_PORT=1111:HTTP_SSL_LISTENER_PORT=2221" instance2
   # start the cluster
   asadmin start-cluster --user admin --passwordfile adminpassword.txt --host hostname --port 4848 cluster1

Some tools also provide a 'silent install' feature. It will record the installation into a text file that you can later replay to have the same result. Using some sed/awk scripts you can probably automate some of this.