↓ Archives ↓

Category → productivity

December 5th, 2011

Puppet versioning like a pro

A big thanks to Atlassian for allowing me to post this series!!

There is NO reason, not to use a version control system while developing puppet manifest/modules. Stating that should be an open door. It allows you to go back in time, share things more easily and track your changes. There is a lot of information out there on how to work with git or any other system. But here a few tips that might help you developing modules:

Tip 1 : give each module it's own repo and use a superproject to join

In a lot of blogposts and even in the excellent Pro Puppet book I see people checking in their entire environment directory into version control.

I'm all for version control but if you manage your modules dir as one flat repository, you loose the way to easily update and share modules from the forge. In essence you are doing a copy that starts living it's own life.

The idea goes like this:

/etc/puppet/environments/development/modules (super-project repo)
  -- puppet-apache (sub-project repo)
  -- puppet-mysql (sub-project repo)
  ...

The super-project repo will contain links to the submodules it uses. This allows the reuse of the sub-project repos in different super-projects. F.i. puppet-modules-team1, puppet-modules-team2 could be superprojects and use different sub-modules.

Using Git

Git has the concept of submodules that allows you to link a parent repository with subprojects. Further detailed documentation can be found at http://book.git-scm.com/5_submodules.html

Using this approach with puppet is nicely described at https://we.riseup.net/riseup+tech/puppet-git-submodules, with some tips at https://labs.riseup.net/code/documents/7

I was always scared away of using submodules, because things like checking in the superproject first and forgetting the submodule, makes part of it unusable. Let alone adding the submodule directoy with a 'slash' has bitten me a few times.

It's a good approach but it requires being awake :)

Using Mercurial

At Atlassian we heavily use Mercurial version control. So I learned the power of Mercurial .

To create a development repo with a specific apache submodule

$ cd /etc/puppet/environments/
$ hg init development
$ cd development/
$ mkdir modules
$ echo 'modules/apache = [git]git://github.com/puppet-modules/puppet-apache.git' > .hgsub
$ hg add .hgsub
$ git clone git://github.com/puppet-modules/puppet-apache.git modules/apache
$ hg ci -mapache

Now if we are checking this out to our test environment, we'll see that it automatically checks out the submodules.

$ cd /etc/puppet/environments
$ hg clone development/ test
updating to branch default
resolving manifests
getting .hgsub
getting .hgsubstate
cloning subrepo modules/apache from git://github.com/puppet-modules/puppet-apache.git
remote: Counting objects: 177, done.
remote: Compressing objects: 100% (94/94), done.
remote: Total 177 (delta 59), reused 168 (delta 52)
Receiving objects: 100% (177/177), 22.97 KiB, done.
Resolving deltas: 100% (59/59), done.
2 files updated, 0 files merged, 0 files removed, 0 files unresolved

I found this workflow:

  • less scary: no way to wrongly add directories
  • it's handy that you can add both svn, git and hg subrepositories
  • submodules are checked out by default (no git submodules init, update)

Note:

Tip 2: Think about your git workflow

In many of the examples you'll see people just commit to the 'master' branch. This works of course but if you are working on different modules/features, it's best to think about your git workflow. There are a lot of blogpost describing how to work in feature branches. Still I found it hard to remember the commands for branching, deleting old branches and so on.

Thanks to Tim Dysinger I learned about Gitflow.

It's basicly git helpers that take away a lot of the pain for working on hotfixes, features, releases. The usage is pretty easy:

# Create the master/developer/release/hotfix branches
$ git flow init

# Start working on a feature (branched from develop)
$ git flow feature start feature1
... do some work
$ git add ...somework...
$ git commit -m "somework feature1"

# This will merge feature1 back to develop
$ git flow feature finish feature1

# Now lets start a release
$ git flow release start release1
... do some work
$ git add ...somework...
$ git commit -m "release1"

# This will merge release1 into master
$ git flow release finish release1

The whole idea is described in details at http://nvie.com/posts/a-successful-git-branching-model/ . And the following video will show you how it works.

More information can also be found at:

Tip 3: Use pre/post-commit hooks

Even with the awesome editor support we previously described, it's still easy to miss a semi-column, or have an incorrect puppet syntax.

Using pre-commits

It's good practice to verify the syntax of your puppet manifests before comitting them to version control. This is well described on the puppetlabs version control page.

In essence before you commit, it will execute 

puppet --parseonly
to check if the syntax is correct.

Using post-commits

A lesser used technique is to run the same check on post-commit:

Instead of running your master repository out of /etc/puppet/environment [PROD], you checkin to an intermediate repository [CHECK].

developer -> LOCAL REPO (pre-commit) -> push -> CHECK REPO -> (post-commit) -> PROD REPO

In the post commit you can:

  • also verify the syntax in case someone didn't check before commiting.
  • if all successfull push the repo to the PROD directories

This helps overcome the problem that your puppetmaster does a run while the repo is an incorrect state.

More details are described at Git workflow and Puppet Environments

More to come

The next post will be about different ways to test your puppet manifests

By JEDI - Just Enough Developed Infrastructure • Posted in git, mercurial, productivity, puppet • Comments Off
December 4th, 2011

Puppet editing like a pro

I've spent some time recently on setting up my environment to work more productively on writing puppet manifests. This blogpost highlights some of the findings to get me more productive on editing puppet files and modules. Some older information can be found at Editor Tips on the puppetlabs website.

Tip 1: Syntax highlighting,snippet completion

Puppet syntax is very specific, it's important to get clues about missing curly braces, semi-colums, etc .. as fast as possible. There is support for this in the most common editors:

Using Textmate:

@Masterzen has created a textmate bundle for use with puppet. You can find it at https://github.com/masterzen/puppet-textmate-bundle.

Michael Halligan describes how to install it from the commandline

mkdir -p /Library/Application\ Support/TextMate/Bundles
cd /Library/Application\ Support/TextMate/Bundles
git clone git://gitorious.org/git-tmbundle/mainline.git Git.tmbundle
git clone http://git.gitorious.org/git-tmbundle/mainline.git Git.tmbundle
git clone https://github.com/masterzen/puppet-textmate-bundle.git Puppet.tmbundle
git clone https://github.com/drnic/Chef.tmbundle.git Chef.tmbundle
osascript -e 'tell app "TextMate" to reload bundles'

Using VIM:

If textmate is not your thing, here is how you can pimp up your vim:

When you look around for puppet/vim integration there seem to have been some re-incarnations:

The most advanced vim-puppet integration I could currently(Dec/2011) find is : - the vim-puppet Tim Sharpe created. - https://github.com/rodjek/vim-puppet

To use the vim-puppet plugin, you're best to use pathogen written by Tim Pope. I've followed the instructions at http://tammersaleh.com/posts/the-modern-vim-config-with-pathogen.

I've enabled the following plugins in my update_bundles script

git_bundles = [
  "git://github.com/astashov/vim-ruby-debugger.git",
  "git://github.com/ervandew/supertab.git",
  "git://github.com/godlygeek/tabular.git",
  "git://github.com/hallison/vim-rdoc.git",
  "git://github.com/msanders/snipmate.vim.git",
  "git://github.com/pangloss/vim-javascript.git",
  "git://github.com/scrooloose/nerdtree.git",
  "git://github.com/timcharper/textile.vim.git",
  "git://github.com/tpope/vim-cucumber.git",
  "git://github.com/tpope/vim-fugitive.git",
  "git://github.com/tpope/vim-git.git",
  "git://github.com/tpope/vim-haml.git",
  "git://github.com/tpope/vim-markdown.git",
  "git://github.com/tpope/vim-rails.git",
  "git://github.com/tpope/vim-repeat.git",
  "git://github.com/tpope/vim-surround.git",
  "git://github.com/tpope/vim-vividchalk.git",
  "git://github.com/tsaleh/taskpaper.vim.git",
  "git://github.com/tsaleh/vim-matchit.git",
  "git://github.com/tsaleh/vim-shoulda.git",
  "git://github.com/tsaleh/vim-tcomment.git",
  "git://github.com/tsaleh/vim-tmux.git",
  "git://github.com/vim-ruby/vim-ruby.git",
  "git://github.com/vim-scripts/Gist.vim.git",
  "git://github.com/scrooloose/syntastic",
  "git://github.com/rodjek/vim-puppet.git",
  "git://github.com/vim-scripts/Specky.git"
]

Most notable plugins:

  • Tabular gives you automatic => alignment
  • Syntastic gives you syntax feedback while you edit files
  • Snipmate gives you the snippets on tab expansion
  • Specky gives you functionality for rspec files
  • vim-ruby gives you extra functionality for ruby files
  • vim-cucumber gives you functionality for cucumber files

For more information on the vim-puppet project go to:

https://github.com/rodjek/vim-puppet/

The snippets that are expanded in the vim-puppet plugin can be found at:

https://github.com/rodjek/vim-puppet/blob/master/snippets/puppet.snippets

Tip 2: don't create modules structure by hand

I keep forgetting the correct structure, files etc.. when I create a new module. Luckily there is an easy way to generate a puppet module structure using the puppet-module gem

$ gem install puppet-module
$ puppet-module
Tasks:
  puppet-module build [PATH_TO_MODULE]                 # Build a module for release
  puppet-module changelog                              # Display the changelog for this tool
  puppet-module changes [PATH_TO_MODULE]               # Show modified files in an installed module
  puppet-module clean                                  # Clears module cache for all repositories
  puppet-module generate USERNAME-MODNAME              # Generate boilerplate for a new module
  puppet-module help [TASK]                            # Describe available tasks or one specific task
  puppet-module install MODULE_NAME_OR_FILE [OPTIONS]  # Install a module (eg, 'user-modname') from a repositor...
  puppet-module repository                             # Show currently configured repository
  puppet-module search TERM                            # Search the module repository for a module matching TERM
  puppet-module usage                                  # Display detailed usage documentation for this tool
  puppet-module version                                # Show the version information for this tool

Options:
  -c, [--config=CONFIG]  # Configuration file
$ puppet-module generate puppetmodule-apache
=========================================================================================
Generating module at /Users/patrick/demo-puppet/modules/puppetmodule-apache
-----------------------------------------------------------------------------------------
puppetmodule-apache
puppetmodule-apache/tests
puppetmodule-apache/tests/init.pp
puppetmodule-apache/spec
puppetmodule-apache/spec/spec_helper.rb
puppetmodule-apache/spec/spec.opts
puppetmodule-apache/README
puppetmodule-apache/Modulefile
puppetmodule-apache/metadata.json
puppetmodule-apache/manifests
puppetmodule-apache/manifests/init.pp

Tip 3 - Geppetto: a Puppet IDE

The folks from cloudsmith have created an Eclipse based editor called Geppetto. It integrates the syntax highlighting, module creation etc... and has nice integration with the Puppet forge

Note: this NOT related with the Gepetto (one P) project by Alban Peignier

James Turnbull was so kind to make a quick screencast on how it works:

But remember it's Java based, so it might take a while to fire it up :)

By JEDI - Just Enough Developed Infrastructure • Posted in productivity, puppet, textmate, vim • Comments Off