↓ Archives ↓

Category → jeos

Docker vs Reality , 0 – 1

(aka the opinionated summary of the #devopsdays London November OpenSpace on , Containers and the new flood of Image Sprawl)

There's a bunch of people out there that think I don't like docker, they are wrong.

I just never understood the hype about it since I didn't see, (and still don't) see it being used at large and people seem to understand that as being against it.

So let me put a couple of things straight :

There's absolutely nothing wrong with using a container based approach when deploying your infrastructure. If you remember my talks about the rise of Open Source Virtualization some years ago you've noticed that I've always mentioned OpenVZ and friends as good alternatives if you wanted to have a lot of isolated platforms on one machine. LXC and friends have grown .. they are even more usable these days. Years ago people bought bare metal and ran Hypervisors on it to isolate resources. These days people rent VM's and also want the same functionality so the use of the combination of Virtualization and Container based technologies is a very good match there.

There's also nothing wrong with using Infrastructure as Code tools to build an reproducable image you are going to deploy will provide you with a disposable image which allows you to quickly launch a reproducable and versionned platform for your application if that application is supposed to be shortlived. The tooling around today is not yet there to have these images long lived as you still need to manage the config inside the containers as your application will evolve, it will change, your environment will change (think even about changing to a different loghost..) , but when you don't have to keep state you can dispose the image and redeploy a new reproducable one.

In the embedded world, this kind of approach with multiple banks has been a round for a while , one image running, a second bank as a fallback, and when you upgrade the passive bank you can swap the roles and still have roll back.

There's is also nothing wrong on combining these to approaches and using tools such as Docker and Packer.

But there is lot wrong with building images that then start living their own life, tools like Veewee etc saw the light to create an easy way to make sure the JeOS image (Just Enough Operating System) we created was reproducable, not to ship around virtual appliances.

But, lets be realistic, the number of applications that are suitable for this kind of environment is small. Most applications these days are still very statefull, and when your application contains state you need to manage that
that state, you can't just dispose an image which has state. Specially in an Enterprise environment stateless, immutable applications are really the exception rather than the rule.

When your application maps with stateless and short lived, or a some people like to call it Immutable please do so.. but if it doesn't please remember that we started using configuration management tools like CFengine, Puppet and Chef to prevent Image Sprawl and Config Drift
There's proprietary businesses out there building tools to detect config drift and extort organisations to solve problems that shouldn't have existed in the first place.

Luckily the majority of smart people I've spoken to over the past couple of weeks pretty much confirmed this ...
Like one of the larger devops minded appliation hosting outsourcers in emea, I asked them how much % of their customer base they could all "Immutable" , exactly 0% was the answer.

Image Based Container solutions are definitely not a one size fits all solution, and we have along way to go before we get there if at all ..

Till then I like not to diffuse my attention to too many different types of deploying platforms, just not to make stuff more complex than it already is...as complexity is the enemy of reliability

Recipes for Automated installation of OS and beyond

Up until now, I've described the options to automate shell scripting, virtual machine creation and network provisioning. So now we can actually get started with automating the installation of the Operating System itself. This not surprisingly is were most sysadmins spent most of their time.

The good news is that we are slowly going a way from the custom scripting toward a reusable and shareable language. Very similar to programming design patterns. Over the years the methods have evolved towards configuration management, which is obviously a good thing.

This also helps in leaving the hero culture sysadmins have because they are the guys who now how to manage their systems.

Jump/Kick start

The first way of automating the OS installation came from using Kickstart or Jumpstart. In essence it is a config file that defines the information necessary to do an installation. It typically contains:

  • the machine's network configuration (interfaces, IP, Routes, DNS)
  • the disk layout setup
  • the packages to be installed
  • patches to be applied

When booting the kernel of an OS, there usually is a way to specify this file, allowing automated installation. Over time this file was put on a floppy, then on a CDROM, USB Drive. To automatically start the installation the install media needed to be modified to change the default kernel boot options. The syntax of these files is typically different per OS.

To create your first file, you would do a manual installation of the OS, and at the end it would create a file corresponding to the manual choices you made. After that, the file could be altered to include additional post-installation scripts or extra packages.

Automating the creation of this has never really been important, I was either using an Editor or using a GUI to generate them. Interestly I found Snake as an example to scripting the creation of the template.

So in other to automate the installation of software, sysadmin would encourage developers to create packages for their install, so that the installation could be easily automated.

An example of a Solaris Jumpstart:

# install_type MUST be first
install_type      initial_install
# start with the minimal required number of packages
cluster           SUNWCXall
cluster           SUNWCapache delete
cluster           SUNWCpcmc   delete
cluster           SUNWCpcmcx  delete
cluster           SUNWCthai   delete
cluster           SUNWClp     delete
cluster           SUNWCnis    delete
cluster           SUNWCppp    delete
# format the entire disk for Solaris
fdisk   all   solaris all
# define how the disk is partitioned
partitioning      explicit
filesys           rootdisk.s0 6144  /
filesys           rootdisk.s1 1024  swap
filesys           rootdisk.s7 free  /state/partition1
# install systems as standalone
system_type standalone
# specify patches to install
patch 119281-06 nfs
# specify packages to install
package SPROcc add nfs

System installation frameworks

Over time frameworks emerged to manage all these packages installation and managing :

Configuration Management

As System recipes and configuration management beautifully describes, managing systems is more then just managing and installing packages : sysadmins would have tons of custom scripts and HOWTO's to describe how to do additional things f.i. installing a complete Webserver according to company standards. All of this knowledge was usually put in specific custom scripts, without a good way of sharing this, . What we actually need is a formalized description of all the things that need to be present and configured on the machines in your setup.

Enter the world configuration management tools. These tools provide a ways of describing the recipes in a language that makes abstraction of the actual system its running on. F.i. adding a user or adding a package is abstracted in the language.

Examples are:

Puppet seems to have most of the market for now, but the Chef guys are catching up. Also a lot of the system management tools have puppet integration:

Example puppet recipe for ensuring the permission of a sudoers file

# /etc/puppet/manifests/classes/sudo.pp
class sudo {
    file { "/etc/sudoers":
        owner => "root",
        group => "root",
        mode  => 440,

Example for mysql node installation

class mysql-server {
     $password = "insert_password_here"
     package { "MySQL-client": ensure => installed }
     package { "MySQL-server": ensure => installed }
     package { "MySQL-shared": ensure => installed }

     exec { "Set MySQL server root password":
       subscribe => [ Package["MySQL-server"], Package["MySQL-client"], Package["MySQL-shared"] ],
       refreshonly => true,
       unless => "mysqladmin -uroot -p$password status",
       path => "/bin:/usr/bin",
       command => "mysqladmin -uroot password $password",

What's fascinating is the new set of tools that starts to emerge here and how it really starts to integrate in real programming languages such as ruby.

  • Moonshine an opensource configuration management and deployment system that follows the Rails way.
  • Gepetto A helper suite for Puppet projects to create, manage and help daily development
  • ShadowPuppet is a Ruby DSL for Puppet, extracted from Moonshine. ShadowPuppet provides a DSL for creating collections (manifests) of Puppet Resources in Ruby.
  • Carpet of the Agile Web Operations guys is an example on how you can mix capistrano with puppet.
  • Cft - Configuration File Tracking: lets you kinda record puppet recipes while entering commands.

For managing configuration files Augeas is doing a really nice job: it parses configuration files in their native formats and transforms them into a tree. Configuration changes are made by manipulating this tree and saving it back into native config files.

Just Enough Operating System

Now that sysadmins are spending more time with their configuration management, they slim down their initial kickstart installations by only using a minimal base install of the OS in their kickstart + a configuration management tool. And when this minimal install is done, they continue applying their recipes to the installation.

This minimal install is sometimes called a base install , an installation with the minimal set of packages installed. Some OS vendors are actively working on providing what is called a JeOS which stands for Just enough Operating System. This is an image that contains an already installed minimal install. They are also useful for creating appliances. Some example are:

Middleware automation

If you're installing a complete stack f.i. Oracle Application Server, Database Server, changes are that the initial package install does nothing. What you need is to create your own instances, pools and so on. What's important is that some of these include their own automation language. It is essential that they have a way of doing every GUI action from the commandline. Otherwise you end up writing scripts for managing their config files. This is a bad thing because this might change overtime or even worse impossible because they store things in binary format.

Examples of good API's are:

Example of asadmin scripting

   # create an cluster
   asadmin create-cluster --user admin --passwordfile adminpassword.txt --host hostname -port 4848 cluster1
   # create instance 1
   asadmin create-instance --user admin --passwordfile adminpassword.txt --host hostname -port 4848 --cluster cluster1 --nodeagent nodeagent1 --systemproperties "JMX_SYSTEM_CONNECTOR_PORT=8687:IIOP_LISTENER_PORT=3330:IIOP_SSL_LISTENER_PORT=4440:IIOP_SSL_ MUTUALAUTH_PORT=5550:HTTP_LISTENER_PORT=1110:HTTP_SSL_LISTENER_PORT=2220" instance1
   # create instance 2
   asadmin create-instance --user admin --passwordfile adminpassword.txt --host hostname -port 4848 --cluster cluster1 --nodeagent nodeagent1 --systemproperties "JMX_SYSTEM_CONNECTOR_PORT=8688:IIOP_LISTENER_PORT=3331:IIOP_SSL_LISTENER_PORT=4441:IIOP_SSL_ MUTUALAUTH_PORT=5551:HTTP_LISTENER_PORT=1111:HTTP_SSL_LISTENER_PORT=2221" instance2
   # start the cluster
   asadmin start-cluster --user admin --passwordfile adminpassword.txt --host hostname --port 4848 cluster1

Some tools also provide a 'silent install' feature. It will record the installation into a text file that you can later replay to have the same result. Using some sed/awk scripts you can probably automate some of this.