Category → testing
An Assimilation type schema in Neo4j
This week I want to talk about an aspect of the Assimilation database schema which is somewhat controversial, an aspect of the schema for which the jury is still out. I chose to represent the Assimilation node type hierarchy with relationships which currently serve no purpose other than to represent the types of nodes in the database. This post will talk about why I put the type hierarchy in, and why it might be a good idea, or maybe not.Devops in Munich
Devopsdays Mountainview sold out in a short 3 hours .. but there's other events that will breath devops this summer.
DrupalCon in Munich will be one of them ..
Some of you might have noticed that I`m cochairing the devops track for DrupalCon Munich,
The CFP is open till the 11th of this month and we are still actively looking for speakers.
We're trying to bridge the gap between drupal developers and the people that put their code to production, at scale.
But also enhancing the knowledge of infrastructure components Drupal developers depend on.
We're looking for talks both on culture (both success stories and failure) , automation,
specifically looking for people talking about drupal deployments , eg using tools like Capistrano, Chef, Puppet,
We want to hear where Continuous Integration fits in your deployment , do you do Continuous Delivery of a drupal environment.
And how do you test ... yes we like to hear a lot about testing , performance tests, security tests, application tests and so on.
... Or have you solved the content vs code vs config deployment problem yet ?
How are you measuring and monitoring these deployments and adding metrics to them so you can get good visibility on both
system and user actions of your platform. Have you build fancy dashboards showing your whole organisation the current state of your deployment ?
We're also looking for people talking about introducing different data backends, nosql, scaling different search backends , building your own cdn using smart filesystem setups.
Or making smart use of existing backends, such as tuning and scaling MySQL, memcached and others.
So lets make it clear to the community that drupal people do care about their code after they committed it in source control !
Please submit your talks here
Behavioral testing with Vagrant – Take 2
A big thanks to Atlassian for allowing me to post this series!!
Running tests from within the VM
After I covered Puppet Unit Testing, the logical step is writing about Behavioral testing.
While writing this , I can up with a good example of why BDD needs to complement your Unit tests:
I have installed the Apache Puppet Module, and all provision ran ok. I wasn't until I tested the webpage with lynx http://localhost that I understood I needed to create a default website. This is of course a trivial example, but I shows you that BDD can help you in testing logical errors.
When this topic arises, most people are familiar with Cucumber Nagios. It contains a series of Cucumber steps that allow you to test http request, amqp, dns, ssh, command.
From what I found, most people would execute these test on the VMs directly. This requires you to install cucumber and all of it's dependent gems in the VM. Gareth RushGrove wrote a great blogpost on packaging cucumber-nagios with fpm
Running tests from outside the VM - Take 1
In some situations, the required gems, libraries might lead to conflicts or introduce dependencies you would rather not have on your production machine. And they would become another point to maintenance in your production machines.
So in a previous blogpost Vagrant Testing,Testing One Two , I already described using modified Cucumber-Nagios steps that interact with Vagrant over ssh.
Running tests from outside the VM - Take 2
But I had a problem with the previous approach. Depending on the situation I would need to run the same tests via different connection methods: vagrant uses ssh, ec2 via fog, openvz via vzctl etc...
So I came up with a new flexible approach: use a configurable command to connect to a vm and have it execute the same steps.
With a little Aruba help
While Cucumber-Nagios slowly moves into Cuken, the SSH steps are getting converted Aruba steps for local exection. And in combination to the ssh-forever steps for ssh interaction.
The Aruba gem is a set of CLI Steps for Cucumber. You can use it to interactively interact with a process or just do a run. Example steps could look like:
Given I run "ssh localhost -p 2222" interactively
And I type "apache2ctl configtest"
And the exit status should be 0
Making it connection neutral
As you can see in the previous step, there is still the connection in the Feature. Not great if we want to run it local. I rephrased it to:
Feature: apache check
Scenario: see if the apache header is served
Given I execute `lynx http://localhost --dump` on a running system
Then the output should match /It works/
Then the exit status should be 0
Scenario: check if the apache config is valid
Given I execute `apache2ctl configtest` on a running system
Then the exit status should be 0
Writing the logic
Here is the logic to make this work (put it in features/support/step_definitions/remote_system_connect_steps.rb . It uses two environment variables:
SYSTEM_EXECUTE: the command to execute just one command
SYSTEM_CONNECT: the command to connect to the system
Example for vagrant would be:
SYSTEM_EXECUTE: "vagrant ssh_config | ssh -q -F /dev/stdin default"
SYSTEM_CONNECT: "vagrant ssh"
This can be also your favorite knife ssh, vzctl 33 enter, mc-ssh somehost
When /^I execute `([^`]*)` on a running system$/ do |cmd|
@execute_command=ENV['SYSTEM_EXECUTE']
@connect_failed=false
unless @execute_command.nil?
steps %Q{ When I run `#{@execute_command} "#{cmd}"` }
else
@execute_failed=true
raise "No SYSTEM_EXECUTE environment variable specified"
end
end
When /^I connect to a running system interactively$/ do
@connect_command=ENV['SYSTEM_CONNECT']
@connect_failed=false
unless @connect_command.nil?
steps %Q{
When I run `#{@connect_command}` interactively
}
else
@connect_failed=true
raise "No SYSTEM_COMMAND environment variable specified"
end
end
When /^I disconnect$/ do
steps %Q{ When I type "exit $?" }
end
Monkey Patching Aruba
By default, Aruba uses shellwords to parse the commandlines you pass, it seems to have an issue with "|" symbols. This is the patch I came up with: (in features/support/env.rb)
require 'aruba/cucumber'
require 'shellwords'
# Here we monkey patch Aruba to work with pipe commands
module Aruba
class Process
include Shellwords
def initialize(cmd, exit_timeout, io_wait)
@exit_timeout = exit_timeout
@io_wait = io_wait
@out = Tempfile.new("aruba-out")
@err = Tempfile.new("aruba-err")
@process = ChildProcess.build(cmd)
@process.io.stdout = @out
@process.io.stderr = @err
@process.duplex = true
end
end
end
After this a regular cucumber run, should work (Note: use a recent cucumber version 1.1.x)
Automating it with Rake
The last part is automating this for Vagrant. For this we create a little rake task:
require "cucumber/rake/task"
task :default => ["validate"]
# Usage rake validate
# - single vm: rake validate
# - multi vm: rake validate vm=logger
Cucumber::Rake::Task.new(:validate) do |task|
# VM needs to be running already
vm_name=ENV['vm'] || ""
ssh_name=ENV['vm'] || "default"
ENV['SYSTEM_CONNECT']="vagrant ssh #{vm_name}"
ENV['SYSTEM_EXECUTE']="vagrant ssh_config #{vm_name}| ssh -q -F /dev/stdin #{ssh_name}"
task.cucumber_opts = ["-s","-c", "features" ]
end
Final words
The solution allows you to reuse the command execution steps, for running them locally, over ssh, or some other connection command.
- This only works for commands that run over ssh, but I think it is already powerfull to do this. If would require amqp testing, you could probably find a command check as well.
- Shell escaping is not 100% correct, this needs more work to work with the special characters or quotes inside quotes.
- When testing, I sometimes miss the context of how a server is created (f.i. the params passed to the puppet manifest or the facts), maybe I could this in a puppet manifests. Not sure on this
- If there is an interest, I could turn this into a vagrant plugin, to make it really easy.
All code can be found at the demo project: https://github.com/jedi4ever/vagrant-guard-demo
Test Driven Infrastructure with Vagrant, Puppet and Guard
This is a repost of my SysAdvent blogpost. It's merely here for archival purposes, or for people who read my blog but didn't see the sysadvent blogpost.
Why
Lots has been written about Vagrant. It simply is a great tool: people use it as a sandbox environment to develop their Chef recipes or Puppet manifests in a safe environment.
The workflow usually looks like this:
- you create a vagrant vm
- share some puppet/chef files via a shared directory
- edit some files locally
- run a
vagrant provisionto see if this works - and if you are happy with it, commit it to your favorite version control repository
Specifically for puppet, thanks to the great work by Nikolay Sturm and Tim Sharpe, we can now also complement this with tests written in rspec-puppet and cucumber-puppet. You can find more info at Puppet unit testing like a pro.
So we got code, and we got tests, what else are we missing? Automation of this process: it's funny if you think of it that we automate the hell out of server installations, but haven't automated the previous described process.
The need to run vagrant provision or rake rspec actually breaks my development flow: I have to leave my editor to run a shell command and then come back to it depending on the output.
Would it not be great if we could automate this whole cycle? And have it run tests and provision whenever files change?
How
The first tool I came across is autotest: it allows one to automatically re-execute tests depending on filesystem changes. Downside is that it could either run cucumber tests or rspec tests.
So enter Guard; it describes itself as a command line tool to easily handle events on file system modifications (FSEvent / Inotify / Polling support). Just what we wanted!
Installing Guard is pretty easy, you require the following gems in your Gemfile
gem 'guard'
gem 'rb-inotify', :require => false
gem 'rb-fsevent', :require => false
gem 'rb-fchange', :require => false
gem 'growl', :require => false
gem 'libnotify', :require => false
As you can tell by the names, it uses different strategies to detect changes in your directories. It uses growl (if correctly setup) on Mac OS X and libnotify on Linux to notify you if your tests pass or fail.
Once installed you get a command guard.
Guard uses a configuration file Guardfile, which can be created by guard init. In this file you define different guards based on different helpers: for example there is guard-rspec, guard-cucumber and many more. There is even a guard-puppet(which we will not use because it works only for local provisioning)
To install one of these helpers you just include it in your Gemfile. We are using only two here:
gem 'guard-rspec'
gem 'guard-cucumber'
Each of these helpers has a similar way of configuring themselves inside a Guardfile. A vanilla guard for a ruby gem with rspec testing would look like this:
guard 'rspec' do
watch(%r{^spec/.+_spec\.rb$})
watch(%r{^lib/(.+)\.rb$}) { |m| "spec/lib/#{m[1]}_spec.rb" }
watch('spec/spec_helper.rb') { "spec" }
end
Whenever a file that matches a watch expression changes, it would run an rspec test. By default if no block is supplied, the file itself is run. You can alter the path in a block as in the example.
Once you have a Guardfile you simply run guard (or bundle exec guard) to have it watch changes. Simple hu?
What
Vagrant setup
Enter our sample puppet/vagrant project. You can find the full source at http://github.com/jedi4ever/vagrant-guard-demo It's a typical vagrant project with the following tree structure:(only 3 levels shown)
├── Gemfile
├── Gemfile.lock
├── Guardfile
├── README.markdown
├── Vagrantfile
├── definitions # Veewee definitions
│ └── lucid64
│ ├── definition.rb
│ ├── postinstall.sh
│ └── preseed.cfg
├── iso # Veewee iso
│ └── ubuntu-10.04.3-server-amd64.iso
└── vendor
└── ruby
└── 1.8
Puppet setup
The project follows Jordan Sissel's idea of puppet nodeless configuration. To specify the classes to apply to a host, we use a fact called: server_role. We read this from a file data/etc/server_tags via a custom fact (inspired by self-classifying puppet node).
This allows us to only require one file, site.pp. And we don't have to fiddle with our hostname to get the correct role. Also if we want to test multiple roles on this one test machine, just add another role to the data/etc/server_tags file.
├── data
│ └── etc
│ └── server_tags
$ cat data/etc/server_tags
role:webserver=true
The puppet modules and manifests can be found in puppet-repo. It has class role::webserver which includes class apache.
puppet-repo
├── features # This is where the cucucumber-puppet catalog policy feature lives
│ ├── catalog_policy.feature
│ ├── steps
│ │ ├── catalog_policy.rb
│ └── support
│ ├── hooks.rb
│ └── world.rb
├── manifests
│ └── site.pp #No nodes required
└── modules
├── apache
| <module content>
├── role
│ ├── manifests
│ │ └── webserver.pp # Corresponds with the role specified
│ └── rspec
│ ├── classes
│ └── spec_helper.rb
└── truth # Logic of puppet nodeless configuration
├── lib
│ ├── facter
│ └── puppet
└── manifests
└── enforcer.pp
Puppet - Vagrant setup
These are the settings we use in our Vagrant file to make puppet work:
config.vm.share_folder "v-data", "/data", File.join(File.dirname(__FILE__), "data")
# Enable provisioning with Puppet stand alone. Puppet manifests
# are contained in a directory path relative to this Vagrantfile.
config.vm.provision :puppet, :options => "--verbose" do |puppet|
puppet.module_path = ["puppet-repo/modules"]
puppet.manifests_path = "puppet-repo/manifests"
puppet.manifest_file = "site.pp"
end
Puppet tests setup
The cucumber-puppet tests will check if the catalog compiles for role role::webserver
Feature: Catalog policy
In order to ensure basic correctness
I want all catalogs to obey my policy
Scenario Outline: Generic policy for all server roles
Given a node with role "<server_role>"
When I compile its catalog
Then compilation should succeed
And all resource dependencies should resolve
Examples:
| server_role |
| role::webserver |
The rspec-puppet tests will check if the package http gets installed
require "#{File.join(File.dirname(__FILE__),'..','spec_helper')}"
describe 'role::webserver', :type => :class do
let(:facts) {{:server_tags => 'role:webserver=true',
:operatingsystem => 'Ubuntu'}}
it { should include_class('apache') }
it { should contain_package('httpd').with_ensure('present') }
end
Guard setup
To make Guard work with a setup like our puppet-repo directory we need to change some things.
This has mostly to do with conventions used in development projects where Guard is normally used.
Fixing Guard-Cucumber to read from puppetrepo/features
The first problem is that the Guard-Cucumber gem standard reads it's features from features directory.
This is actually hardcoded in the gem. But nothing a little monkey patching can't solve:
require 'guard/cucumber'
# Inline extending the ::Guard::Cucumber
# Because by default it only looks in the ['features'] directory
# We have it in ['puppet-repo/features']
module ::Guard
class ExtendedCucumber < ::Guard::Cucumber
def run_all
passed = Runner.run(['puppet-repo/features'], options.merge(options[:run_all] || { }).merge(:message => 'Running all features'))
if passed
@failed_paths = []
else
@failed_paths = read_failed_features if @options[:keep_failed]
end
@last_failed = !passed
throw :task_has_failed unless passed
end
end
end
# Monkey patching the Inspector class
# By default it checks if it starts with /feature/
# We tell it that whatever we pass is valid
module ::Guard
class Cucumber
module Inspector
class << self
def cucumber_folder?(path)
return true
end
end
end
end
end
Orchestration of guard runs
The second problem was to have Guard only execute the Vagrant provision when BOTH the cucumber and rspec tests would be OK.
Inspired by the comments of Netzpirat, I got it working so that the block vagrant provision would only execute on both tests being complete.
# This block simply calls vagrant provision via a shell
# And shows the output
def vagrant_provision
IO.popen("vagrant provision") do |output|
while line = output.gets do
puts line
end
end
end
# So determine if all tests (both rspec and cucumber have been passed)
# This is used to only invoke the vagrant_provision if all tests show green
def all_tests_pass
cucumber_guard = ::Guard.guards({ :name => 'extendedcucumber', :group => 'tests'}).first
cucumber_passed = cucumber_guard.instance_variable_get("@failed_paths").empty?
rspec_guard = ::Guard.guards({ :name => 'rspec', :group => 'tests'}).first
rspec_passed = rspec_guard.instance_variable_get("@failed_paths").empty?
return rspec_passed && cucumber_passed
end
Guard matchers
With all the correct guards and logic setup, it's time to specify the correct options to our Guards.
group :tests do
# Run rspec-puppet tests
# --format documentation : for better output
# :spec_paths to pass the correct path to look for features
guard :rspec, :version => 2, :cli => "--color --format documentation", :spec_paths => ["puppet-repo"] do
# Match any .pp file (but be carefull not to include any dot-temporary files)
watch(%r{^puppet-repo/.*/[^.]*\.pp$}) { "puppet-repo" }
# Match any .rb file (but be carefull not to include any dot-temporary files)
watch(%r{^puppet-repo/.*/[^.]*\.rb$}) { "puppet-repo" }
# Match any _rspec.rb file (but be carefull not to include any dot-temporary files)
watch(%r{^puppet-repo/.*/[^.]*_rspec.rb})
end
# Run cucumber puppet tests
# This uses our extended cucumber guard, as by default it only looks in the features directory
# --strict : because otherwise cucumber would exit with 0 when there are pending steps
# --format pretty : to get readable output, default is null output
guard :extendedcucumber, :cli => "--require puppet-repo/features --strict --format pretty" do
# Match any .pp file (but be carefull not to include any dot-temporary files)
watch(%r{^puppet-repo/[^.]*\.pp$}) { "puppet-repo/features" }
# Match any .rb file (but be carefull not to include any dot-temporary files)
watch(%r{^puppet-repo/[^.]*\.rb$}) { "puppet-repo/features" }
# Feature files are monitored as well
watch(%r{^puppet-repo/features/[^.]*.feature})
# This is only invoked on changes, not at initial startup
callback(:start_end) do
vagrant_provision if all_tests_pass
end
callback(:run_on_change_end) do
vagrant_provision if all_tests_pass
end
end
end
The full Guardfile is on github
Run it
From within the top directory of the project type
$ guard
Now open a second terminal and change some of the files and watch the magic happen.
Final remarks
The setup described is an idea I only recently started exploring. I'll probably enhance this in the future or may experience other problems.
For the demo project, I only call vagrant provision, but this can of course be extended easily. Some ideas:
- Inspired by Oliver Hookins - How we use Vagrant as a throwaway testing environment:
- use sahara to create a snapshot just before the provisioning
- have it start from a clean machine when all tests pass
- Turn this into a guard-vagrant gem, to monitor files and tests
Test Driven Infrastructure with Vagrant, Puppet and Guard
This is a repost of my SysAdvent blogpost. It's merely here for archival purposes, or for people who read my blog but didn't see the sysadvent blogpost.
Why
Lots has been written about Vagrant. It simply is a great tool: people use it as a sandbox environment to develop their Chef recipes or Puppet manifests in a safe environment.
The workflow usually looks like this:
- you create a vagrant vm
- share some puppet/chef files via a shared directory
- edit some files locally
- run a
vagrant provisionto see if this works - and if you are happy with it, commit it to your favorite version control repository
Specifically for puppet, thanks to the great work by Nikolay Sturm and Tim Sharpe, we can now also complement this with tests written in rspec-puppet and cucumber-puppet. You can find more info at Puppet unit testing like a pro.
So we got code, and we got tests, what else are we missing? Automation of this process: it's funny if you think of it that we automate the hell out of server installations, but haven't automated the previous described process.
The need to run vagrant provision or rake rspec actually breaks my development flow: I have to leave my editor to run a shell command and then come back to it depending on the output.
Would it not be great if we could automate this whole cycle? And have it run tests and provision whenever files change?
How
The first tool I came across is autotest: it allows one to automatically re-execute tests depending on filesystem changes. Downside is that it could either run cucumber tests or rspec tests.
So enter Guard; it describes itself as a command line tool to easily handle events on file system modifications (FSEvent / Inotify / Polling support). Just what we wanted!
Installing Guard is pretty easy, you require the following gems in your Gemfile
gem 'guard'
gem 'rb-inotify', :require => false
gem 'rb-fsevent', :require => false
gem 'rb-fchange', :require => false
gem 'growl', :require => false
gem 'libnotify', :require => false
As you can tell by the names, it uses different strategies to detect changes in your directories. It uses growl (if correctly setup) on Mac OS X and libnotify on Linux to notify you if your tests pass or fail.
Once installed you get a command guard.
Guard uses a configuration file Guardfile, which can be created by guard init. In this file you define different guards based on different helpers: for example there is guard-rspec, guard-cucumber and many more. There is even a guard-puppet(which we will not use because it works only for local provisioning)
To install one of these helpers you just include it in your Gemfile. We are using only two here:
gem 'guard-rspec'
gem 'guard-cucumber'
Each of these helpers has a similar way of configuring themselves inside a Guardfile. A vanilla guard for a ruby gem with rspec testing would look like this:
guard 'rspec' do
watch(%r{^spec/.+_spec\.rb$})
watch(%r{^lib/(.+)\.rb$}) { |m| "spec/lib/#{m[1]}_spec.rb" }
watch('spec/spec_helper.rb') { "spec" }
end
Whenever a file that matches a watch expression changes, it would run an rspec test. By default if no block is supplied, the file itself is run. You can alter the path in a block as in the example.
Once you have a Guardfile you simply run guard (or bundle exec guard) to have it watch changes. Simple hu?
What
Vagrant setup
Enter our sample puppet/vagrant project. You can find the full source at http://github.com/jedi4ever/vagrant-guard-demo It's a typical vagrant project with the following tree structure:(only 3 levels shown)
├── Gemfile
├── Gemfile.lock
├── Guardfile
├── README.markdown
├── Vagrantfile
├── definitions # Veewee definitions
│ └── lucid64
│ ├── definition.rb
│ ├── postinstall.sh
│ └── preseed.cfg
├── iso # Veewee iso
│ └── ubuntu-10.04.3-server-amd64.iso
└── vendor
└── ruby
└── 1.8
Puppet setup
The project follows Jordan Sissel's idea of puppet nodeless configuration. To specify the classes to apply to a host, we use a fact called: server_role. We read this from a file data/etc/server_tags via a custom fact (inspired by self-classifying puppet node).
This allows us to only require one file, site.pp. And we don't have to fiddle with our hostname to get the correct role. Also if we want to test multiple roles on this one test machine, just add another role to the data/etc/server_tags file.
├── data
│ └── etc
│ └── server_tags
$ cat data/etc/server_tags
role:webserver=true
The puppet modules and manifests can be found in puppet-repo. It has class role::webserver which includes class apache.
puppet-repo
├── features # This is where the cucucumber-puppet catalog policy feature lives
│ ├── catalog_policy.feature
│ ├── steps
│ │ ├── catalog_policy.rb
│ └── support
│ ├── hooks.rb
│ └── world.rb
├── manifests
│ └── site.pp #No nodes required
└── modules
├── apache
| <module content>
├── role
│ ├── manifests
│ │ └── webserver.pp # Corresponds with the role specified
│ └── rspec
│ ├── classes
│ └── spec_helper.rb
└── truth # Logic of puppet nodeless configuration
├── lib
│ ├── facter
│ └── puppet
└── manifests
└── enforcer.pp
Puppet - Vagrant setup
These are the settings we use in our Vagrant file to make puppet work:
config.vm.share_folder "v-data", "/data", File.join(File.dirname(__FILE__), "data")
# Enable provisioning with Puppet stand alone. Puppet manifests
# are contained in a directory path relative to this Vagrantfile.
config.vm.provision :puppet, :options => "--verbose" do |puppet|
puppet.module_path = ["puppet-repo/modules"]
puppet.manifests_path = "puppet-repo/manifests"
puppet.manifest_file = "site.pp"
end
Puppet tests setup
The cucumber-puppet tests will check if the catalog compiles for role role::webserver
Feature: Catalog policy
In order to ensure basic correctness
I want all catalogs to obey my policy
Scenario Outline: Generic policy for all server roles
Given a node with role "<server_role>"
When I compile its catalog
Then compilation should succeed
And all resource dependencies should resolve
Examples:
| server_role |
| role::webserver |
The rspec-puppet tests will check if the package http gets installed
require "#{File.join(File.dirname(__FILE__),'..','spec_helper')}"
describe 'role::webserver', :type => :class do
let(:facts) {{:server_tags => 'role:webserver=true',
:operatingsystem => 'Ubuntu'}}
it { should include_class('apache') }
it { should contain_package('httpd').with_ensure('present') }
end
Guard setup
To make Guard work with a setup like our puppet-repo directory we need to change some things.
This has mostly to do with conventions used in development projects where Guard is normally used.
Fixing Guard-Cucumber to read from puppetrepo/features
The first problem is that the Guard-Cucumber gem standard reads it's features from features directory.
This is actually hardcoded in the gem. But nothing a little monkey patching can't solve:
require 'guard/cucumber'
# Inline extending the ::Guard::Cucumber
# Because by default it only looks in the ['features'] directory
# We have it in ['puppet-repo/features']
module ::Guard
class ExtendedCucumber < ::Guard::Cucumber
def run_all
passed = Runner.run(['puppet-repo/features'], options.merge(options[:run_all] || { }).merge(:message => 'Running all features'))
if passed
@failed_paths = []
else
@failed_paths = read_failed_features if @options[:keep_failed]
end
@last_failed = !passed
throw :task_has_failed unless passed
end
end
end
# Monkey patching the Inspector class
# By default it checks if it starts with /feature/
# We tell it that whatever we pass is valid
module ::Guard
class Cucumber
module Inspector
class << self
def cucumber_folder?(path)
return true
end
end
end
end
end
Orchestration of guard runs
The second problem was to have Guard only execute the Vagrant provision when BOTH the cucumber and rspec tests would be OK.
Inspired by the comments of Netzpirat, I got it working so that the block vagrant provision would only execute on both tests being complete.
# This block simply calls vagrant provision via a shell
# And shows the output
def vagrant_provision
IO.popen("vagrant provision") do |output|
while line = output.gets do
puts line
end
end
end
# So determine if all tests (both rspec and cucumber have been passed)
# This is used to only invoke the vagrant_provision if all tests show green
def all_tests_pass
cucumber_guard = ::Guard.guards({ :name => 'extendedcucumber', :group => 'tests'}).first
cucumber_passed = cucumber_guard.instance_variable_get("@failed_paths").empty?
rspec_guard = ::Guard.guards({ :name => 'rspec', :group => 'tests'}).first
rspec_passed = rspec_guard.instance_variable_get("@failed_paths").empty?
return rspec_passed && cucumber_passed
end
Guard matchers
With all the correct guards and logic setup, it's time to specify the correct options to our Guards.
group :tests do
# Run rspec-puppet tests
# --format documentation : for better output
# :spec_paths to pass the correct path to look for features
guard :rspec, :version => 2, :cli => "--color --format documentation", :spec_paths => ["puppet-repo"] do
# Match any .pp file (but be carefull not to include any dot-temporary files)
watch(%r{^puppet-repo/.*/[^.]*\.pp$}) { "puppet-repo" }
# Match any .rb file (but be carefull not to include any dot-temporary files)
watch(%r{^puppet-repo/.*/[^.]*\.rb$}) { "puppet-repo" }
# Match any _rspec.rb file (but be carefull not to include any dot-temporary files)
watch(%r{^puppet-repo/.*/[^.]*_rspec.rb})
end
# Run cucumber puppet tests
# This uses our extended cucumber guard, as by default it only looks in the features directory
# --strict : because otherwise cucumber would exit with 0 when there are pending steps
# --format pretty : to get readable output, default is null output
guard :extendedcucumber, :cli => "--require puppet-repo/features --strict --format pretty" do
# Match any .pp file (but be carefull not to include any dot-temporary files)
watch(%r{^puppet-repo/[^.]*\.pp$}) { "puppet-repo/features" }
# Match any .rb file (but be carefull not to include any dot-temporary files)
watch(%r{^puppet-repo/[^.]*\.rb$}) { "puppet-repo/features" }
# Feature files are monitored as well
watch(%r{^puppet-repo/features/[^.]*.feature})
# This is only invoked on changes, not at initial startup
callback(:start_end) do
vagrant_provision if all_tests_pass
end
callback(:run_on_change_end) do
vagrant_provision if all_tests_pass
end
end
end
The full Guardfile is on github
Run it
From within the top directory of the project type
$ guard
Now open a second terminal and change some of the files and watch the magic happen.
Final remarks
The setup described is an idea I only recently started exploring. I'll probably enhance this in the future or may experience other problems.
For the demo project, I only call vagrant provision, but this can of course be extended easily. Some ideas:
- Inspired by Oliver Hookins - How we use Vagrant as a throwaway testing environment:
- use sahara to create a snapshot just before the provisioning
- have it start from a clean machine when all tests pass
- Turn this into a guard-vagrant gem, to monitor files and tests
Puppet unit testing like a pro
A big thanks to Atlassian for allowing me to post this series!!
In our previous blogpost on Puppet Versioning, we described the most basic check to see if a puppet manifest was valid. We used the parseonly function to see if it would compile.
Until know this means we have only have if the compiler is happy, not that it performs the function it needs to do. In 2009 after the first devopsdays I wrote a collection of Test Driven Infrastructure Links . This was obviously inspired by Lindsay Holmwood's talk on cucumber-nagios.
On the Opscode chef front, Stephen Nelson-Smith wrote a great book Test-driven Infrastructure with Chef on how to do this. Also see the cuken project where re-usable cucumber steps are grouped.
Because we are using Puppet here at Atlassian, I was out to understand the current state of puppet testing. A lot can already be found at http://puppetlabs.com/blog/testing-modules-in-the-puppet-forge/
Note that I've purposely named this blog 'Puppet unit testing', as the tests I'm describing now, don't run against an actual system. Therefore it's hard to test the actual behavior.
Tip 1: cucumber-puppet
Inspired by Lindsay Holmwood's talk on cucumber-nagios and Ohad Levy's manitest Nikolay Sturm created cucumber-puppet
In his post on Thoughts on testing puppet manifests he explains that the idea of writing tests is NOT about duplicating the code, and he identified the most common problems he was facing are:
- catalog does not compile: syntax errors, missing template files, ..
- catalog does compile, but cannot be applied: unreachable or non-existent resources, missing file resources in repo
- catalog does applies, but is faulty: faulty files, due to empty manifests variables or wrong values, missing dependencies (wrong order ...), files are installed without ensuring a directory ...
An important advice is:
Resource specifications can be useful for documentation purposes or refactorings. However, there is a risk of reimplementing your Puppet manifest, so be wary.
$ cd puppet-mymodule
$ gem install cucumber-puppet
Write features per module, this is the structure we are aiming at:
module
+-- manifests
+-- lib
+-- features
+-- support
| +-- hooks.rb
| +-- world.rb
+-- catalog
+-- feature..
Generate a cucumber-puppet world:
$ cucumber-puppet-gen world
Generating with world generator:
[ADDED] features/support/hooks.rb
[ADDED] features/support/world.rb
[ADDED] features/steps
# Adjust the paths to your modules and manifests
$ cat features/support/hooks.rb
Before do
# adjust local configuration like this
# @puppetcfg['confdir'] = File.join(File.dirname(__FILE__), '..', '..')
# @puppetcfg['manifest'] = File.join(@puppetcfg['confdir'], 'manifests', 'site.pp')
# @puppetcfg['modulepath'] = "/srv/puppet/modules:/srv/puppet/site-modules"
# adjust facts like this
@facts['architecture'] = "i386"
end
# Nothing exciting here
$ cat features/support/world.rb
require 'cucumber-puppet/puppet'
require 'cucumber-puppet/steps'
World do
CucumberPuppet.new
end
Generating a policy feature:
$ cucumber-puppet-gen policy
Generating with policy generator:
[ADDED] features/catalog
# Notice the <hostname>.example.com.yaml
# These files contain the facts to test your catalog against
#
$ cat features/catalog/policy.feature
Feature: General policy for all catalogs
In order to ensure applicability of a host's catalog
As a manifest developer
I want all catalogs to obey some general rules
Scenario Outline: Compile and verify catalog
Given a node specified by "features/yaml/<hostname>.example.com.yaml"
When I compile its catalog
Then compilation should succeed
And all resource dependencies should resolve
Examples:
| hostname |
| localhost |
To do an actual run:
$ cucumber-puppet features/catalog/policy.feature
Feature: General policy for all catalogs
In order to ensure applicability of a host's catalog
As a manifest developer
I want all catalogs to obey some general rules
Scenario Outline: Compile and verify catalog # features/catalog/policy.feature:6
Given a node specified by "features/yaml/<hostname>.example.com.yaml" # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:1
When I compile its catalog # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:14
Then compilation should succeed # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:48
And all resource dependencies should resolve # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:28
Examples:
| hostname |
| localhost |
Cannot find node facts features/yaml/localhost.example.com.yaml. (RuntimeError)
features/catalog/policy.feature:7:in `Given a node specified by "features/yaml/<hostname>.example.com.yaml"'
Failing Scenarios:
cucumber features/catalog/policy.feature:6 # Scenario: Compile and verify catalog
1 scenario (1 failed)
4 steps (1 failed, 3 skipped)
0m0.006s
List of commands:
Generators for cucumber-puppet
Available generators
feature Generate a cucumber feature
policy Generate a catalog policy
testcase Generate a test case for the test suite
testsuite Generate a test suite for puppet features
world Generate cucumber step and support files
General options:
-p, --pretend Run, but do not make any changes.
-f, --force Overwrite files that already exist.
-s, --skip Skip files that already exist.
-d, --delete Delete files that have previously been generated with this generator.
--no-color Don't colorize the output
-h, --help Show this message
--debug Do not catch errors
He has also added support for testing exported-resources.
- I found it not very clear from the documentation, I better understood the basics while watching a good video tutorial on cucumber-puppet was given by Tom Sulston.
- General information on cucumber can be found in the Pragmatic Programmers's Cucumber book
- Or at the the cucumber main site http://cukes.info/
And for a more practical explanation, see how Oliver Hookins describes the way Nokia uses cucumber-puppet
Scenario: Proxy host and port have sensible defaults
Given a node of class "mymodule::myapp"
And we have loaded "test" settings
And we have unset the fact "proxy_host"
And we have unset the fact "proxy_port"
When I compile the catalog
Then there should be a file "/etc/myapp/config.properties"
And the file should contain "proxy.port=-1"
And the file should contain /proxy\.host=$/
----
Then /^the file should contain "(.*)"$/ do |text|
fail "File parameter 'content' was not specified" if @resource["content"].nil?
fail "Text content [#{text}] was not found" unless @resource["content"].include?(text)
end
Then /^the file should contain \/([^\"].*)\/$/ do |regex|
fail "File parameter 'content' was not specified" if @resource["content"].nil?
fail "Text regex [/#{regex}/] did not match" unless @resource["content"] =~ /#{regex}/
end
Tip 2: rspec-puppet
While the idea on using specs and puppet is not new (https://github.com/jes5199/puppet_spec), the new tool on the block is rspec-puppet brought to us by Tim Sharpe. The same person who gave us vim-puppet and puppet-lint
Like the cucumber-puppet structure, the idea is to have specs directory close to your module:
module
+-- manifests
+-- lib
+-- spec
+-- spec_helper.rb
+-- classes
| +-- <class_name>_spec.rb
+-- defines
| +-- <define_name>_spec.rb
+-- functions
+-- <function_name>_spec.rb
I found it useful to change the default spec_helper.rb as the default
require 'rspec-puppet'
RSpec.configure do |c|
c.module_path = File.expand_path(File.join(File.dirname(__FILE__), '..', '..'))
c.manifest_dir = File.expand_path(File.join(File.dirname(__FILE__), '..', '..','..','manifests'))
end
desc "Run specs check on puppet manifests"
RSpec::Core::RakeTask.new(:spec) do |t|
t.pattern = './demo-puppet/modules/**/*_spec.rb' # don't need this, it's default
t.verbose = true
t.rspec_opts = "--format documentation --color"
# Put spec opts in a file named .rspec in root
end
Here is a quick example for checking if the class apache installs a package httpd when on a Debian system
require "#{File.join(File.dirname(__FILE__),'..','spec_helper')}"
describe 'apache', :type => :class do
let (:title { 'basic' })
let(:params) { { } }
let(:facts) { {:operatingsystem => 'Debian', :kernel => 'Linux'} }
it { should contain_package('httpd').with_ensure('installed') }
end
A more detailed description can be found at
For more generic information on rspec:
- The main website to find all the expectations, Mocks etc.. http://rspec.info/documentation/
- The book on Rspec by Pragmatic programmers- http://pragprog.com/book/achbd/the-rspec-book
Conclusion cucumber-puppet vs rspec-puppet
I think you can write your tests in both to do the same. Currently they both support 2.6 and 2.7
I found the rspec-puppet a bit simpler to juggle with providing params like :name or :facts. The yaml file didn't feel to flexible to me. Also cucumber seems to install more dependent gems, that might inflict with other projects.
But as Nikolay already said:
"don't duplicate your manifests in your tests" Focus on the catalog problems he described earlier and test your logic. Don't test if puppet is doing it's job, test that your logic it's doing it's job.
This is why I called them unit-tests, they don't test the real functionality. (That's for the next blogpost)
Tip 3: puppet-lint
To check you files against programming style you can use https://github.com/rodjek/puppet-lint. It will check for Rules on Spacing, Identation & Whitespace , Quoting, Resources, Conditionals, Classes
An easy way to integrate it in your Rakefile is:
require 'puppet-lint'
desc "Run lint check on puppet manifests"
task :lint do
linter = PuppetLint.new
Dir.glob('./demo-puppet/modules//**/*.pp').each do |puppet_file|
puts "Evaluating #{puppet_file}"
linter.file = puppet_file
linter.run
end
fail if linter.errors?
end
Now you can simply run:
$ rake lint
Tip 4: go wild and build your own test/catalog logic
After having a look at the rspec-puppet logic, I looked deeper in the way to walk trough the catalog object. This is pretty much work in progress, but the idea is find a way to look at changes in the catalog.
The following is a list of useful examples on understanding on how to work with puppet in ruby code:
The first list of links are some fun tools written by Dean Wilson of www.puppetcookbook.com fame:
- http://www.unixdaemon.net/tools/puppet/puppet-cucumber-providers.html
- http://www.unixdaemon.net/tools/puppet/listing-puppet-managed-files.html
- https://github.com/deanwilson/puppet-scripts/blob/master/puppet-ls
- https://github.com/deanwilson/puppet-scripts/blob/master/puppet-pkg
- https://github.com/deanwilson/puppet-scripts/blob/master/pm-grep
R.I. Pienaar of Mcollective Fame shows a way to create diff on a catalog. this can be useful to understand what tests to run in between changes:
- http://www.devco.net/archives/2010/11/14/getting_diffs_for_puppet_catelogs.php
Dramatically better CLI tools for interacting with Puppet: https://github.com/lak/puppet-interfaces
And another fun tool is puppet-growl that will run a puppet file syntax check everytime a file changes in a directory.
This final gist shows how to walk through the catalog and check the classes and resources available:
Beyond Configuration Mgmt
(This post has been sitting in the drafts folder for way to long, I decided to push the publish button anyhow .. some people might get ideas from it..)
We've all run in to the problem, you've puppetized, or euh .. cooked , about every part of your infrastructure and then there's this one service which has no config files, a broken api that doesn't allow you to configure antyhing, but a magnificent web gui to configure all aspects of the service. Magnificent for the eye , full of AJAX and other fancy stuff which wget isn't really keen on. Off course before it even starts working you need to set it's password , from that webgui.
Sometimes when you are lucky they store al their config in a database, which you can dump, parse and replace all the host specific parameters for other deployments, but is that an approach you like ? As for each new version you'll need to reanalyze the db layout. But no matter how you look at it ,dumping the DB and restoring it is an ugly hack you don't want.
Other alternatives like sniffing the traffic and replaying the POSTS etc were considered ... but fancy AJAX stuff and SSL make that less trivial than it seems
Wo while discussing with an upstream project they proposed to actually screenscrape their config webgui .
So screenscraping the config gui it is .. but how ... I started looking at tools that are typically used for testing rather than for automation, with the purpose of replaying the scenarios one needs to configure the services.
My first attempt was Selenium, it plugs into a browser , so it's easy to acraully record what it has to do, and it saves it's scenarios in a somewhat readable/ editable format.
Having found the export to perl function it alll looked promising. However the export to perl isn't really an export to perl as I epxected .. I assumed it would just generate the perl code to run the same scneario which would be awesome .. it however generates a perl script that instructs a selenium server to run the script.
One of the annoyancies I ran into with Selenium is that a browser
doesn't accept self signed certificates , and one can't preprovision a browser easyily with those freshly created certificates. (Yes Karl I already read about certutil ... )
I had heard good things about Cucumber so I was pretty eager to start testing it ... In short Cucumber lack documentation ,
I tried a couple of things but I couldn't get beyond testing if a certain string was on a page.. couldn't figure out how to fill in a form etc ...
Maybe if anyone could point me to some great documentation on how you should write recipe's here ... I didn't find the documentation all to easy to find ..
Bummer as it really really looks promisiung .. specially since it is so lightweight ..
IP played with JMeter and Sahi too .. but still
So apart from filing bugs to the upstream project/product and hoping they understand your problem and are willing to oopen up their API , what other options do you folks suggest ?
I gave a short talk about this at Puppetcamp in Amsterdam and the audience came up with a bunch of other potential projects to look at .
The main problem still is that all these are tools to automate testing , they don't provide you with a general purpose approach to solve the configuration mgmt problem, each time the upstream vendor modifies the layout of his page you hav e to do the work again and that .. really doesn't sound promising ..
Using Interviews to Improve Software Quality
A real-life story of one way to introduce stunning quality into a product at the end of the development process using interviews - without adding staff.Using Interviews to Improve Software Quality
Stunning Quality
Back in the 90s I was involved with about 100 other people in a project to develop a new voice mail system – software, hardware and firmware. The hardware was 100% new design, and the software was about 70% new. Along the way we stumbled into something that improved our end quality in a way that can reasonably be described as stunning.
A few months away from our first controlled introduction (CI) or beta site – and things didn't really feel right – lots of the system was best described as squishy – that is, not really solid. They passed unit tests, and the system tests worked continually for days or weeks without failures, but then something would fail. These squishy components included key subsystems like voice record/play – vital to the system. For a voice mail system, this is having a database for critical data that usually stores the data. Not good enough.
About this time, our second level manager (2LM) declared that everyone will work 60 hours a week on site until our CI. As it turns out, my job had been to lead the device driver team (we had about 18 new UNIX device drivers – and I was the only experienced UNIX device driver writer) – and all of these were quite solid – for reasons worthy of another article. But returning to this article... As was my personal style, I had been winding down and celebrating no longer being able to break anything no matter how hard I tried by spending time in what is sometimes called Beneficial Scholastic (BS) discussions – avoiding anything that looked like work for a few days. So, when our 2LM declared that we were all to work 60 hours a week, I wondered what I could possibly do with 60 hours a week. So I asked her what we were supposed to do with those 60 hours. Her reply? Do the same as you are doing – only more so. Some of you may know me, and know I have a certain fondness for BS discussions – but I was certain that not even I could spend 60 hours a week in BS – not to count that it would be a bit on the unproductive side.
Squishy Code
I knew there was a good bit of squishy code out there – you could feel it in the air – in the hallway and water-cooler talk. If I was going to spend 60 hours a week on-site I wanted it to be for something that mattered. I wanted to get the voice play/record code fixed – now that would matter. But it wasn't my code – and deciding to start fixing or writing tests for someone else's code is definitely not the kind of thing to endear you to others. I also knew that this wasn't the only squishy code out there. So I decided to try another approach.
In parts of this company (like many) you can occasionally run into a shoot-the-messenger mentality hiding not very far down. I had an idea on how to redirect our effort in a much more profitable way – but I didn't want to get shot as the bearer of bad news. So I asked our 2LM - “If I knew how we should be spending our time most effectively, would you want to know?”. To her credit, she replied “Of course!”.
What Needs Doing?
So I went off to carry out my idea. My idea was simple – I interviewed most of the people in the organization and set the emotional stage before asking them a few questions, wrote down the answers and put them into a short (5-page) report.
For the purposes of this posting, imagine March 23 was the cut date for our furst customer. My interviews with the project engineers went something like this:
Imagine it's the morning of March 23. You come in to work. Most of the offices are dark. It's very quiet. Those people who are here, just came to print copies of their resumes. We failed. Why did we fail? Then I asked – What should we do to prevent this? Followed by the important question – Can I use your name?
Interestingly enough every single person said yes to the last question. By far the most common answers to the first question were of the form “My software caused us to fail”. The most common answers to the second question were of the form “We need to test it unreasonably – beat the excrement out of it”.
This was an interesting set of results – indicative, in my opinion of a great organization. People understood the gravity of what they were trying to do, they took responsibility for their own bugs, and they knew what to do to fix it, and very much wanted to.
I wrote up the memo, and added a graph to it – since my management (at all levels) loved graphs. Now, it isn't a graph that any mathematician would love – it was more of an illustration – but this is OK I was showing it to management, not to mathematicians. The original graph is lost to antiquity – but I've drawn an approximation to it below.
This graph represents the universal truth that “some things are better than other things” - and also the common truth that some things are good enough, but some things are not. There are basically two lines on the graph separated by a small margin. This would be the result of applying uniform effort across the project. Note that few things rise up to cross the “good enough” line. My proposal then, was to apply our efforts only in the areas that were below the “good enough” line, and fill in only the areas below the line – indicated by light blue in the illustration. Seems perfectly obvious. In fact, it is perfectly obvious. What wasn't obvious was which things weren't good enough.
Since I was still skeptical that I wouldn't be a victim of “kill the messenger”, I went back to my 2LM and asked her if she still wanted to know what we should be doing – she said “Of course”. Since I was still concerned about shoot-the-messenger, I handed her my memo, and went home to hide where I would be hard to find. By the next day, I was dying of curiosity and went to see how my memo was taken. I looked and looked but could find no managers at all. I finally got up the nerve to ask my 2LM's assistant where everyone was – she said they were all off-site discussing the “Robertson memo”. I was completely floored. This was a very surprising result to me. To my shock and the everlasting credit of my 2LM, she reorganized the entire project around the recommendations I had collected in my memo.
If you take staff off the good enough things, and put them on the things that aren't good enough, then lots of normal rules on who does what get broken down. People are by definition, going to be working on areas that aren't “theirs”. And because it's being done over the whole project, it's by definition not a condemnation of anyone in particular, and is thereby socially acceptable.
More Testing – The Right Kind in the Right Places
Not all the suggested things were testing, but most were. The project had good unit tests. It had good system tests. Many of the recommendations for new tests turned out to be automated and merciless subsystem tests – which were subsequently nicknamed Bamm-Bamm tests. Since I had done the interviews and written the memo, I claimed first choice on which area to work in. I chose to work on the voice play/record testing. I wrote a set of unreasonable, merciless automated tests, which set up the hardware in cross-connected mode to allow for monitoring, grabbed the voice APIs at a low level and exercised them randomly. Play, skip ahead, skip back, speed up, slow down, record, play a touch tone, etc – all randomly, and without any kind of rhyme, rhythm, or restraint. This setup also could use more voice channels than were visible on the real system – and it didn't have to be connected to a switch – meaning it could run without tying up so much expensive test hardware, and it could produce more load than could ever possibly exist in the real system. The first bug it found reproduced itself reliably in 5 minutes – which took at least a week to reproduce itself in the system test environment.
Once the voice subsystem could run these tests for an hour without a failure, no one could ever find any more bugs. This from a subsystem where previously it had taken weeks to reproduce a problem – even once. Like the first bug, these subsequent bugs found reproduced themselves in a few minutes. This is a huge difference. You could reproduce a problem a few times, create a fix, and try it out all in a morning – instead of a few months.
This kind of result was certainly dramatic, but there were many others performing similar work on other weak subsystems. This all sounds wonderful, but what was the actual result? Our 2LM decided to delay our CI by a week, to let us finish our testing and fixing. Our CI site was already a heavy user of our voice mail systems – and we were going to replace their current system with the new one – which would not look any different to them. So, they already had a culture of significant use of voice mail, and they would hit the system hard on the first Monday after the cutover.
And Then A Miracle Occurs...
We installed the system and migrated their data over to it, and then everyone sat around and watched – waiting to respond to that first crash – which we assumed would come by 10:30 their local time. But that crash didn't come – and it didn't come – and it still didn't come. We were all pretty shocked. We assumed that they must have had a company holiday – so we pulled the traffic logs and compared them to their previous weeks of traffic – they were doing what they had always done – and it was just working. It was fully 6 weeks later before this heavy user of voice mail had their first complaint of any kind. And it was 6 months before our first crash in the field. Given how much new hardware and software there was, this was more than a little surprising. This reliability continued on and certainly appeared to be better than any product put out by this company (one known for reliable products), but more interesting confirmation of this came years later.
For a variety of internal political reasons, there were really only two major feature releases of this product – and after that it went mostly into maintenance mode – except for recording new languages and getting certified in new countries. However, a number of years later, some of the chips on the board weren't going to be available any more. Given the political issues surrounding the project, I assumed that would be the end of it – after all, the corporate hierarchy had tried to kill it numerous times, and it was a niche product in a small part of our portfolio. As it turns out, the product was incredibly profitable – and in spite of the efforts to minimize it and kill it, had become responsible for a very significant portion of the profit of our division – and politics or no, somehow no one was willing to leave all that money on the table. Of course, because of the political issues, all the development staff had fled to places that weren't going to kill their careers.
The Results Over Time
To keep this revenue stream going, the company had to gather up some of the original staff to create an updated version of the product. We had a new VP who didn't know about all the political issues of the past, and got together the new team in a meeting to learn about this project. He asked “Who all is working on the product now?” One person raised his hand, then someone said, “Yeah, but he's so good, he only works on it half time”. Of course, everyone laughed. But it was the truth – this 100-person software/hardware/firmware project which was out there in the field making tons of money had been maintained completely by one person half-time for the last 5 years. It simply worked - all the time – almost without fail. The only known bug was due to a hardware design issue relating to where some signal paths ran. About once a year one particular DSP would crash because of crosstalk in this set of signal paths. No one would let him redesign the board to fix it – so he put in a simple work around to catch it quickly and restart the DSP. This was really the only outstanding issue of any substance in the product. Of course, the redesign also performed more cost reductions as a result of 10 years of hardware evolution, so that it became even more profitable than it had been before.
When I saw the stunned look on that VP's face when he realized that despite the laughter this wasn't a larger in-joke – I really realized the magnitude of what we had accomplished. It was written all over his face that he really didn't believe that it was possible for a popular and profitable product of this size to work so well that it almost never needed fixing – for more than five years. It isn't exactly a common occurrence – “unheard of” probably isn't too strong a term.
Questions and Reflections
A series of interesting questions come out of this – Which things that we did made the difference? Why did they make such a difference? Why were they necessary? If this was a great organization, why didn't everyone already know all these things and act accordingly? How much of the answers to the “what to do about it” were influenced by my personal assumptions?
I don't have definitive answers to these questions – but I have thought about them a lot since this happened – since I was as stunned by the outcome as anyone. Although we all recognized how good the quality seemed to be at this first release, the political implications of it all for our own personal careers took first priority – and most of us forgot about it and got caught up in surviving and starting on our next things.
There are several things you can observe about this which seem interesting – first of all, this process helped identify what really needed doing – and reminded people of what they mostly already knew – which things worked, and which ones didn't.
Secondly, because the information came from the staff, it has inherent credibility – when someone says their own software isn't too good, it has more credibility than if someone is complaining about someone else's software. These complaints came from expert sources.
Thirdly, each of the areas has information from experts on what they thought would most likely result in success.
Fourthly, the resulting plan to take all these things seriously, resulted in a radical reallocation of staff – breaking down normal social barriers like “this is my code, stay out”, and replacing them with more of a community approach – where the areas with troubles got more help. This last point is worth elaborating some more. A normal project profile looks a bit like a normal distribution – with a small amount of staff at the beginning, a large amount at the middle, with the staff tapering off at the end. However, the truth is, that at the beginning and ends of those curves, that there is a good bit of underutilized staff – that is people who don't really have enough to do. I was personally a case in point – I had finished my code, but I was still on the project – part of the staffing curve – but not being utilized. Naturally, I wasn't about to tell anyone – I had put in incredible effort and hours over a period of years to get to this point. One of the things this technique does is locate that underutilized staff and put them to work where their help is most needed – in a way that causes minimal turf issues.
To be honest, I won't say there were no turf issues – but they were minimal and quickly overcome because the entire management structure supported these changes. The person I gave my testing tool (and bug reports to) wasn't completely happy that I'd found these issues – but he understood the benefits and after a bit, was OK with all this – especially since in the process I'd given him a tool to reproduce and fix problems very quickly – in the end making his life much easier.
Where Did The Staff (Money) Come From?
One of the more interesting realizations about this technique is that it cost very little additional money over what was already being spent for the project. How can that be, when clearly a lot more testing was getting done?
The answer is pretty simple, we redirected staff from those parts of the project that were above the line to places where things were below the line. Note that these people weren't writing code in those other areas, they were writing test code, which requires a good bit less knowledge of the particular subsystem under test. The staff already knew the project, how it worked, and had a basic understanding of most or all the subsystems – so they became productive quickly.
The fact that there was unused staff was an interesting thing – and I suspect is reasonably common towards the end of a project. Most of the people on my subteam were in a similar situation to mine – they had worked hard and gotten their code done right, and were now waiting for someone to bring them more bugs. I'm sure we weren't alone – people whose subsystems were above the imaginary “good enough” line were often in this situation. Below is a graph which I believe illustrates this situation reasonably well.
This represents the basic idea that staff needs rarely match staff availability. This particular graph represents the happy situation that eventually peak staff needs are met. The key realization is that in the latter phases of the project, there comes a situation where staff available to the project actually exceeds current needs. When individuals find themselves in this situation, they rarely call attention to it, but just try and keep busy and hope no one notices. In the case of our project, we chose to redeploy this staff to create new test mechanisms and solve other problems as identified by the informal survey I took.
Trying This Out For Yourself
If one were to try and reproduce this as an experiment, here are the things which seem to me to be most relevant to such an attempt:
-
Wait until the project is near enough completion, so that there is the possibility of having some underutilized staff that can be redirected, and a probability that the development staff has a good idea where the important problem areas are.
-
Perform interviews asking people what they think needs to happen. The person doing these interviews needs to understand development and be seen as an honest broker – either an outside development consultant or a respected developer with a reputation for honesty and respect of their peers. The emotional content of the method I used may have helped break people out of normal ways of thinking, but the exact formulation is probably not that important. Although having someone who can provide guidance in providing solutions to problems is not ideal interview technique, but it may be desirable in terms of getting good results.
-
Follow the recommendations that are gathered – trying to take them seriously. In my experience, reorganizing the project to move staff around will likely be necessary.
Needless to say, if you decide to repeat this experiment, I'd be interested in hearing about it.