Category → Code
Rapid Puppet runs with MCollective
The typical Puppet use case is to run the daemon every 30 minutes or so and just let it manage your machines. Sometimes though you want to be able to run it on all your machines as quick as your puppet master can handle.
This is tricky as you generally do not have a way to cap the concurrency and it’s hard to orchestrate that. I’ve extended the MCollective Puppet Agent to do this for you so you can do a rapid run at roll out time and then go back to the more conservative slow pace once your window is over.
The basic logic I implemented is this:
- Discover all nodes, sort them alphabetically
- Count how many nodes are active now, wait till it’s below threshold
- Run a node by just starting a –onetime background run
- Sleep a second
This should churn through your nodes very quickly without overwhelming the resources of your master. You can see it in action here, you can see it started 3 nodes and once it got to the 4th 3 were already running and it waited for one of them to finish:
% mc-puppetd -W /dev_server/ runall 2 Thu Aug 05 17:47:21 +0100 2010> Running all machines with a concurrency of 2 Thu Aug 05 17:47:21 +0100 2010> Discovering hosts to run Thu Aug 05 17:47:23 +0100 2010> Found 4 hosts Thu Aug 05 17:47:24 +0100 2010> Running dev1.one.net, concurrency is 0 Thu Aug 05 17:47:26 +0100 2010> dev1.one.net schedule status: OK Thu Aug 05 17:47:28 +0100 2010> Running dev1.two.net, concurrency is 1 Thu Aug 05 17:47:30 +0100 2010> dev1.two.net schedule status: OK Thu Aug 05 17:47:32 +0100 2010> Running dev2.two.net, concurrency is 2 Thu Aug 05 17:47:34 +0100 2010> dev2.two.net schedule status: OK Thu Aug 05 17:47:35 +0100 2010> Currently 3 nodes running, waiting Thu Aug 05 17:48:00 +0100 2010> Running dev3.two.net, concurrency is 2 Thu Aug 05 17:48:05 +0100 2010> dev3.two.net schedule status: OK
This is integrated into the existing mc-puppetd client script you don’t need to roll out anything new to your servers just the client side.
Using this to run each of 47 machines with a concurrency of just 4 I was able to complete a cycle in 8 minutes. Doesn’t sound too impressive but my average run time is around 40 seconds on every node with some being 90 to 150 seconds. My puppetmaster server that usually sits at a steady 0.2mbit out were serving a constant 2mbit/sec for the duration of this run.
Monitoring ActiveMQ
I have a number of ActiveMQ servers, 7 in total, 3 in a network of brokers the rest standalone. For MCollective I use topics extensively so don’t really need to monitoring them much other than for availability. I also though do a lot of Queued work where lots of machines put data in a queue and others process the data.
In the Queue scenario you absolutely need to monitor queue sizes, memory usage and such. You also need to graph things like rates of messages, consumer counts and memory use. I am busy writing a number of Nagios and Cacti plugins to help with this, you can find them on Github.
To use these you need to have the ActiveMQ Statistics Plugin enabled.
First we need to monitor queue sizes:
$ check_activemq_queue.rb --host localhost --user nagios --password passw0rd --queue exim.stats --queue-warn 1000 --queue-crit 2000 OK: ActiveMQ exim.stats has 1 messages
This will connect to localhost monitoring a queue exim.stats warning you when it’s got 1000 messages and critical at 2000.
I need to add to this the ability to monitor memory usage, this will come over the next few days.
I also have a plugin for Cacti it can output stats for the broker as a whole and also for a specific queue. First the whole broker:
$ activemq-cacti-plugin.rb --host localhost --user nagios --password passw0rd --report broker stomp+ssl:stomp+ssl storePercentUsage:81 size:5597 ssl:ssl vm:vm://web3 dataDirectory:/var/log/activemq/activemq-data dispatchCount:169533 brokerName:web3 openwire:tcp://web3:6166 storeUsage:869933776 memoryUsage:1564 tempUsage:0 averageEnqueueTime:1623.90502285799 enqueueCount:174080 minEnqueueTime:0.0 producerCount:0 memoryPercentUsage:0 tempLimit:104857600 messagesCached:0 consumerCount:2 memoryLimit:20971520 storeLimit:1073741824 inflightCount:9 dequeueCount:169525 brokerId:ID:web3-44651-1280002111036-0:0 tempPercentUsage:0 stomp:stomp://web3:6163 maxEnqueueTime:328585.0 expiredCount:0
Now a specific queue:
$ activemq-cacti-plugin.rb --host localhost --user nagios --password passw0rd --report exim.stats size:0 dispatchCount:168951 memoryUsage:0 averageEnqueueTime:1629.42897052992 enqueueCount:168951 minEnqueueTime:0.0 consumerCount:1 producerCount:0 memoryPercentUsage:0 destinationName:queue://exim.stats messagesCached:0 memoryLimit:20971520 inflightCount:0 dequeueCount:168951 expiredCount:0 maxEnqueueTime:328585.0
Grab the code on GitHub and follow there, I expect a few updates in the next few weeks.
Bootstrapping Puppet on EC2 with MCollective
The problem of getting EC2 images to do what you want is quite significant, mostly I find the whole thing a bit flakey and with too many moving parts.
- When and what AMI to start
- Once started how to do you configure it from base to functional. Especially in a way that doesn’t become a vendor lock.
- How do you manage the massive sprawl of instances, inventory them and track your assets
- Monitoring and general life cycle management
- When and how do you shut them, and what cleanup is needed. Being billed by the hour means this has to be a consideration
These are significant problems and just a tip of the ice berg. All of the traditional aspects of infrastructure management – like Asset Management, Monitoring, Procurement – are totally useless in the face of the cloud.
A lot of work is being done in this space by tools like Pool Party, Fog, Opscode and many other players like the countless companies launching control panels, clouds overlaying other clouds and so forth. As a keen believer in Open Source many of these options are not appealing.
I want to focus on the 2nd step above here today and show how I pulled together a number of my Open Source projects to automate that. I built a generic provisioner that hopefully is expandable and usable in your own environments. The provisioner deals with all the interactions between Puppet on nodes, the Puppet Master, the Puppet CA and the administrators.
<rant> Sadly the activity in the Puppet space is a bit lacking in the area of making it really easy to get going on a cloud. There are suggestions on the level of monitoring syslog files from a cronjob and signing certificates based on that. Really. It’s a pretty sad state of affairs when that’s the state of the art.
Compare the ease of using Chef’s Knife with a lot of the suggestions currently out there for using Puppet in EC2 like these: 1, 2, 3 and 4.
Not trying to have a general Puppet Bashing session here but I think it’s quite defining of the 2 user bases that Cloud readiness is such an after thought so far in Puppet and its community. </rant>
My basic needs are that instances all start in the same state, I just want 1 base AMI that I massage into the desired final state. Most of this work has to be done by Puppet so it’s repeatable. Driving this process will be done by MCollective.
I bootstrap the EC2 instances using my EC2 Bootstrap Helper and I use that to install MCollective with just a provision agent. It configures it and hook it into my collective.
From there I have the following steps that need to be done:
- Pick a nearby Puppet Master, perhaps using EC2 Region or country as guides
- Set up the host – perhaps using /etc/hosts – to talk to the right master
- Revoke and clean any old certs for this hostname on all masters
- Instruct the node to create a new CSR and send it to its master
- Sign the certificate
- Run my initial bootstrap Puppet environment, this sets up some hard to do things like facts my full build needs
- Run the final Puppet run in my normal production environment.
- Notify me using XMPP, Twitter, Google Calendar, Email, Boxcar and whatever else I want of the new node
This is a lot of work to be done on every node. And more importantly it’s a task that involves many other nodes like puppet masters, notifiers and so forth. It has to adapt dynamically to your environment and not need reconfiguring when you get new Puppet Masters. It has to deal with new data centers, regions and countries without needing any configuration or even a restart. It has to happen automatically without any user interaction so that your auto scaling infrastructure can take care of booting new instances even while you sleep.
The provisioning system I wrote does just this. It follows the above logic for any new node and is configurable for which facts to use to pick a master and how to notify you of new systems. It adapts automatically to your ever changing environments thanks to discovery of resources. The actions to perform on the node are easily pluggable by just creating an agent that complies to the published DDL like the sample agent.
You can see it in action in the video below. I am using Amazon’s console to start the instance, you’d absolutely want to automate that for your needs. You can also see it direct on blip.tv here. For best effect – and to be able to read the text – please fullscreen.
In case the text is unreadable in the video a log file similar to the one in the video can be seen here and an example config here
Past this point my Puppet runs are managed by my MCollective Puppet Scheduler.
While this is all done using EC2 nothing prevents you from applying these same techniques to your own data center or non cloud environment.
Hopefully this shows that you can wrap all the logic needed to do very complex interactions with systems that are perhaps not known for their good reusable API’s in simple to understand wrappers with MCollective, exposing those systems to the network at large with APIs that can be used to reach your goals.
The various bits of open source I used here are:
- MCollective
- EC2 Bootstrap helper on CentOS 5.5
- The MCollective Server Provisioner
- The sample provisioner agent
- My Nagger notification framework with it’s XMPP plugin
- The Naggernotify MCollective Agent
- The Puppet CA Mcollective Agent
- Puppet and Facter
Aggregating Nagios Checks With MCollective
A very typical scenario I come across on many sites is the requirement to monitor something like Puppet across 100s or 1000s of machines.
The typical approaches are to add perhaps a central check on your puppet master or to check using NRPE or NSCA on every node. For this example the option exist to easily check on the master and get one check but that isn’t always easily achievable.
Think for example about monitoring mail queues on all your machines to make sure things like root mail isn’t getting stuck. In those cases you are forced to do per node checks which inevitably result in huge notification storms in the event that your mail server was down and not receiving the mail from the many nodes.
MCollective has had a plugin that can run NRPE commands for a long time, I’ve now added a nagios plugin using this agent to combine results from many hosts.
Sticking with the Puppet example, here are my needs:
- I want to know if anywhere some puppet machine isn’t successfully doing runs.
- I want to be able to do puppetd –disable and not get alerts for those machines.
- I do not want to change any configs when I am adding new machines, it should just work.
- I want the ability to do monitoring on subsets of machines on different probes
This is a pretty painful set of requirements for nagios on its own to achieve. Easy with the help of MCollective.
Ultimately, I just want this:
OK: 42 WARNING: 0 CRITICAL: 0 UNKNOWN: 0
Meaning 42 machines – only ones currently enabled – are all running happily.
The NRPE Check
We put the NRPE logic on every node. A simple check command in /etc/nagios/nrpe.d/check_puppet_run.cfg:
command[check_puppet_run]=/usr/lib/nagios/plugins/check_file_age -f /var/lib/puppet/state/state.yaml -w 5400 -c 7200
In my case I just want to know there are successful runs happening, if I wanted to know the code is actually compiling correctly I’d monitor the local cache age and size.
Determining if Puppet is enabled or not
Currently this is a bit hacky, I’ve filed tickets with Puppet Labs to improve this. The way to determine if puppet is disabled is to check if the lock file exist and if its 0 bytes. If it’s not zero bytes it means a puppetd is currently doing a run – there will be a pid in it. Or the puppetd crashed and there’s a stale pid preventing other runs.
To automate this and integrate into MCollective I’ve made a fact puppet_enabled. We’ll use this in MCollective discovery to only monitor machines that are enabled. Get this onto all your nodes perhaps using Plugins in Modules.
The MCollective Agent
You want to deploy the MCollective NRPE Agent to all your nodes, once you’ve got it right you can test it easily using something like this:
% mc-nrpe -W puppet_enabled=1 check_puppet_run
* [ ============================================================> ] 47 / 47
Finished processing 47 / 47 hosts in 395.51 ms
OK: 47
WARNING: 0
CRITICAL: 0
UNKNOWN: 0Note we’re restricting the run to only enabled hosts.
Integrating into Nagios
The last step is to add this to nagios. I create SSL certs and a specific client configuration for Nagios and put these in it’s home directory.
The check-mc-nrpe plugin works best with Nagios 3 as it will return subsequent lines of output indicating which machines are in what state so you get the details hidden behind the aggregation in alerts. It also outputs performance data for total node, each status and also how long it took to do the check.
The nagios command would be something like this:
define command{
command_name check_mc_nrpe
command_line /usr/sbin/check-mc-nrpe --config /var/log/nagios/.mcollective/client.cfg -W $ARG1$ $ARG2$
}And finally we need to make a service:
define service{
host_name monitor1
service_description mc_puppet-run
use generic-service
check_command check_mc_nrpe!puppet_enabled=1!check_puppet_run
notification_period awakehours
contact_groups sysadmin
}Here are a few other command examples I use:
All machines with my Puppet class “pki”, check the age of certs:
check_command check_mc_nrpe!pki!check_pki
All machines with my Puppet class “bacula::node”, make sure the FD is running:
check_command check_mc_nrpe!bacula::node!check_fd
…and that they were backed up:
check_command check_mc_nrpe!bacula::node!check_bacula_main
Using this I removed 100s of checks from my monitoring platform, saving on resources and making sure I can do my critical monitor tasks better.
Depending on the quality of your monitoring system you might even get a graph showing the details hidden behind the aggregation:
The above is a graph showing a series of servers where the backup ran later than usual, I had 2 alerts only, would have had more than 30 before aggregation.
Restrictions for Probes
The last remaining requirement I had was to be able to do checks on different probes and restrict them. My Collective is one big one spread all over the world which means sometimes things are a bit slow discovery wise.
So I have many nagios servers doing local checks. Using MCollective discovery I can now easily restrict checks, for example If I only wanted to check machines in the USA and I had a fact country I only have to change my command line in the service declaration:
check_command check_mc_nrpe!puppet_enabled=1 country=us!check_puppet_run
This will then via MCollective discovery just monitor machines in the US.
What to monitor this way
As this style of monitoring is done using Discovery you would need to think carefully about what you monitor this way. It’s totally conceivable that if a node is under high CPU load that it wont respond to discovery commands in time, and so wont get monitored!
You would then for example not want to monitor things like load averages or really critical services this way, but we all have a lot of peripheral things like zombie process counts and a lot of other places where aggregation makes a lot of sense, in those cases by all means consider this approach.
Tutorial: Writing MCollective Agents
I’ve recorded a screencast that walks you through the process of developing a SimpleRPC Agent, give it a DDL and also a simple client to communicate with it.
The tutorial creates a small echo agent that takes input and return it unmodified. It validates that you are sending a string and has a sample of dealing with intermittent failure.
Once you’ve watched this, or even during, you can use the following links are reference material: Writing Agents, Data Definition Language and Writing Clients.
You can view it directly on blip.tv which will hopefully be better quality.
I used a few VIM Snippets during the demo to boilerplate the agent and DDL, you’ll find these in the tarball for the upcoming 0.4.7 release in the ext/vim directory, they are already on GitHub too.
Recent MCollective releases and roadmap.
I’ve had two successive Marionette Collective releases recently, I was hoping to have one big one but I was waiting for the Stomp maintainers to do a release and it was taking a while.
These two releases are both major feature releases covering major feature sets. See lower down for a breakdown of it all.
We’re nearing feature completeness for the SimpleRPC layer as I am adding a number of features of interest to Enterprise and Large users especially around security and web UIs.
Once we’re at the end of this cycle I’ll do a 1.0.0 release and then from there move onto the next major feature cycle. The next cycle will focus on queuing long running tasks, background scheduling, future scheduling of tasks and a lot of related work. I posted some detail about these plans to the list recently.
Over the new few days or weeks I’ll do a number of Screencasts exploring some of these new features in depth, for now the list of what’s new:
Security
- New SSL based security system
- New Authorization system [blog post] and sample plugins, allowing for fine grained control over every request
Connectivity
We can use Ruby Gem Stomp 1.1.6 which brings a lot of enhancements:
- Connection pools for failover between multiple ActiveMQs
- Lots of tunables about the connection pools such as retry frequencies etc
- SSL TLS between node and ActiveMQ
Writing Web and Dynamic UIs
- A DDL that describes agents, inputs and outputs:
- Creates auto generated documentation
- Can be used to auto generate user interfaces
- The client library will only make requests that validate against the DDL
- In future input validations will move into the DDL and will be done automatically for you
- Web UI’s can bypass or do their own discovery and use the DDL to auto generate user interfaces
Usability
- Fire-and-Forget style requests, for when you just want something done but do not care about results, these requests are very quick as they do not do any discovery.
- Agents can now be reloaded without restarting the daemon
- A new mc-inventory tool that can be used to view facts, agents and classes for a node
- Many UI enhancements to the CLI tools
Puppet Module Repository isn’t just for modules
You can store more than just your modules at the Forge. 
I just added my types and providers to my collection of modules at the new Puppet Module Forge. I’d love to all those people maintaining types and providers, functions, and facts add theirs to the Forge also. It’s a cool way to share your code (and the site allows you to provide links back to your code repository and ticketing system so user’s can report bugs). In time I hope most people’s environments will consist of the core types and providers bundled with Puppet and a selection of cool code generated by the community and sourced from the Puppet Forge.
Puppet Forge in beta!
The Puppet Forge AKA the Puppet Module Repository is live and operational. It’s a store of Puppet modules (and types and providers) that allows you to share your awesome code and modules with others.
It also comes with the puppet-module tool that allows you to build modules for, manage and install modules from the forge. You can install puppet-module via a gem:
$ sudo gem install puppet-module
Both the site and tool are in public beta right now so hammer away at it and tell us what you think!
Xen Live Migration with MCollective
I retweeted this on twitter, but it’s just too good to not show. Over at rottenbytes.com Nicolas is showing some proof of concept code he wrote with MCollective that monitors the load on his dom0 machines and initiate live migrations of virtual machines to less loaded servers.
This is the kind of crazy functionality I wanted to enable with MCollective and it makes me very glad to see this kind of thing. The server side and client code combined is only 230 lines – very very impressive.
This is a part of what VMWare DRS does Nico has some ideas to add other sexy features as well as this was just a proof of concept. The logic for what to base migrations on will be driven by a small DSL for example.
I asked him how long it took to knock this together: time taken to get acquainted with MCollective combined with time to write the agent and client was only 2 days, that’s very impressive. He already knew Ruby well though 
And has a Ruby gem to integrate with Xen.
I’m copying the output from his code below, but absolutely head over to his blog to check it out he has the source up there too:
[mordor:~] ./mc-xen-balancer [+] hypervisor2 : 0.0 load and 0 slice(s) running [+] init/reset load counter for hypervisor2 [+] hypervisor2 has no slices consuming CPU time [+] hypervisor3 : 1.11 load and 3 slice(s) running [+] added test1 on hypervisor3 with 0 CPU time (registered 18.4 as a reference) [+] added test2 on hypervisor3 with 0 CPU time (registered 19.4 as a reference) [+] added test3 on hypervisor3 with 0 CPU time (registered 18.3 as a reference) [+] sleeping for 30 seconds [+] hypervisor2 : 0.0 load and 0 slice(s) running [+] init/reset load counter for hypervisor2 [+] hypervisor2 has no slices consuming CPU time [+] hypervisor3 : 1.33 load and 3 slice(s) running [+] updated test1 on hypervisor3 with 0.0 CPU time eaten (registered 18.4 as a reference) [+] updated test2 on hypervisor3 with 0.0 CPU time eaten (registered 19.4 as a reference) [+] updated test3 on hypervisor3 with 1.5 CPU time eaten (registered 19.8 as a reference) [+] sleeping for 30 seconds [+] hypervisor2 : 0.16 load and 0 slice(s) running [+] init/reset load counter for hypervisor2 [+] hypervisor2 has no slices consuming CPU time [+] hypervisor3 : 1.33 load and 3 slice(s) running [+] updated test1 on hypervisor3 with 0.0 CPU time eaten (registered 18.4 as a reference) [+] updated test2 on hypervisor3 with 0.0 CPU time eaten (registered 19.4 as a reference) [+] updated test3 on hypervisor3 with 1.7 CPU time eaten (registered 21.5 as a reference) [+] hypervisor3 has 3 threshold overload [+] Time to see if we can migrate a VM from hypervisor3 [+] VM key : hypervisor3-test3 [+] Time consumed in a run (interval is 30s) : 1.7 [+] hypervisor2 is a candidate for being a host (step 1 : max VMs) [+] hypervisor2 is a candidate for being a host (step 2 : max load) trying to migrate test3 from hypervisor3 to hypervisor2 (10.0.0.2) Successfully migrated test3 !
Authorization plugins for MCollective SimpleRPC
Till now The Marionette Collective has relied on your middleware to provide all authorization and authentication for requests. You’re able to restrict certain middleware users from certain agents, but nothing more fine grained.
In many cases you want to provide much finer grain control over who can do what, some cases could be:
- A certain user can only request service restarts on machines with a fact customer=acme
- A user can do any service restart but only on machines that has a certain configuration management class
- You want to deny all users except root from being able to stop services, others can still restart and start them
This kind of thing is required for large infrastructures with lots of admins all working in their own group of machines but perhaps a central NOC need to be able to work on all the machines, you need fine grain control over who can do what and we did not have this will now. It would also be needed if you wanted to give clients control over their own servers but not others.
Version 0.4.5 will have support for this kind of scheme for SimpleRPC agents. We wont provide a authorization plugin out of the box with the core distribution but I’ve made one which will be available as a plugin.
So how would you write an auth plugin, first a typical agent would be:
module MCollective module Agent class Service<RPC::Agent authorized_by :action_policy # .... end end end
The new authorized_by keyword tells MCollective to use the class MCollective::Util::ActionPolicy to do any authorization on this agent.
The ActionPolicy class can be pretty simple, if it raises any kind of exception the action will be denied.
module MCollective module Util class ActionPolicy def self.authorize(request) unless request.caller == "uid=500" raise("You are not allow access to #{request.agent}::#{request.action}") end end end end end
This simple check will deny all requests from anyone but Unix user id 500.
It’s pretty simple to come up with your own schemes, I wrote one that allows you to make policy files like the one below for the service agent:
policy default deny allow uid=500 * * * allow uid=502 status * * allow uid=600 * customer=acme acme::devserver
This will allow user 500 to do everything with the service agent. User 502 can get the status of any service on any node. User 600 will be able to do any actions on machines with the fact customer=acme that also has the configuration management class acme::devserver on them. Everything else will be denied.
You can do multiple facts and multiple classes in a simple space separated list. The entire plugin to implement such policy controls was only 120 – heavy commented – lines of code.
I think this is a elegant and easy to use layer that provides a lot of functionality. We might in future pass more information about the caller to the nodes. There’s some limitations, specifically about the source of the caller information being essentially user provided so you need to keep that mind.
As mentioned this will be in MCollective 0.4.5.