Solaris packages, DevOps, Bass and other things of interest...

Tiller v0.0.7 Now Supports Environment Variables

| Comments

Just a quick update : Tiller has just been updated to v0.0.7. I’ve just added a new EnvironmentDataSource which is super-simple, but very convenient. It makes all your environment variables accessible to templates (as global values) by converting them to lower-case, and prefixing them with env_.

To use it, just add it to your datasources in /etc/tiller/common.yaml :

  - file
  - environment

And then you can do things like this in your templates :

Hello, world!
My auto-generated hostname is <%= env_hostname %>

Check it out at GitHub or install the Ruby gem.

Tiller Project and Docker Container Configuration

| Comments

After several days of hacking on my Runner.rb project, I’m pleased to say that I’ve completed a much more polished and complete solution to shipping multiple configuration files in a single Docker container. The project is now known as Tiller and has been published as a Ruby gem so you can just run gem install tiller to install it. You can find a good overview of it, and how it works over at the Github, but it’s still essentially the same approach :

  • Provide templates of your configuration files as ERB templates
  • Provide a source for each “environment” containing the values that should be placed in those templates, and where you want them installed. This is usually done with a YAML configuration file, but you can now use other sources for this information (see later in this blog post).

The first change (apart from the name) is that there’s no longer a nasty config hash to use inside your templates; you can simply declare a value in an environment file:

  target: /var/www/html/example.html
    welcome_text: 'Hello, world!'

And then reference it straight inside the template :

<h1>This is generated from example.erb</h1>
<%= welcome_text %>

However, a much bigger change is that I have abstracted out the data generation and sources of templates. I’ve bundled two providers (FileDataSource and FileTemplateSource) that simply read the contents of ERB files under /etc/tiller/templates, and use YAML files under /etc/tiller/environments so that it mimics the old Runner.rb.

This means you can now write your own plugins (and I’ll also work on some additional ones to ship with later releases) to do things like pull templates from a remote HTTP server, populate the values with custom/dynamic variables such as FQDN, IP address of the host, or even fetch them from a LDAP server instead of pulling them off a YAML file on disk.

As a very simple example of a custom “global data source”, suppose your Docker containers all use the FQDN structure (e.g., and you wanted to extract the site part to use in a configuration file template. You could write a file called /usr/local/lib/tiller/data/example.rb :

class ExampleDataSource < Tiller::DataSource

  def global_values
    # site: Just the second part of the FQDN
    # This assumes DNS is working correctly!
    { 'site' =>  Socket.gethostbyname(Socket.gethostname).first.split('.')[1] }


And then load it in /etc/tiller/common.yaml along with the default file data source :

  - file
  - example

And now all your templates can use this by referencing <%= site %>.

There’s much more you can do with this, including defining values for specific templates, and writing TemplateSources to provide the templates themselves, but that’s a bit too much detail for this introductory blog post. Go and check out the documentation at the Github page, browse through some of the examples, and check this blog for more updates & examples.

Dynamic Configuration Files With Docker Containers

| Comments

I’ve been using Docker containers on Linux systems for a while now, and have recently developed a neat solution to what I imagine is a fairly common problem.

I had a number of Docker containers that I wanted to launch from a common image, but with a slightly different configuration depending on the environment in which I was launching them. For example, a web application container might connect to a different database in a staging environment, or a MongoDB replica set name might be different. This meant my options basically looked like:

  • Maintain multiple containers / Dockerfiles.
  • Maintain the configuration in separate data volumes and use --volumes-from to pull the relevant container in.
  • Bundle the configuration files into one container, and manually specify the CMD or ENTRYPOINT values to pick this up.

None of those really appealed due to needless duplication, or the complexity of an approach that would necessitate really long docker run commands. So I knocked up a quick & simple Ruby script (UPDATE: This project is now called tiller and has changed significantly) that I could use across all my containers, which does the following :

  • Generates configuration files from ERB templates
  • Uses values provided in YAML files, one per environment
  • Copies the generated templates to the correct location and specifies permissions
  • Executes a replacement process once it’s finished (e.g. mongod, nginx, supervisord, etc.)

This way I can keep all my configuration together in the container, and just tell Docker which environment to use when I start it. As a simple example, here’s what it looks like when you run it :

# docker run -t -i -e environment=staging markround/demo_container:latest
Runner.rb v0.0.1
Using runner configuration from /etc/runner
Using environment staging
Parsing /etc/runner/templates/mongodb.conf.erb
Setting ownerships and privileges on /etc/mongodb.conf
Template generation completed, about to exec replacement process.
Calling /usr/bin/supervisord...

You can check it out, along with the documentation at it’s Github repository.

Getting Started With Sensu on Solaris

| Comments

Sensu is a monitoring framework written in Ruby. It’s small and very easy to extend, as well as being extremely scalable. It uses a message bus to communicate with it’s different components (clients, servers, api hosts) as well as external systems such as Graphite, which can be used to produce graphs and store metrics.

It uses a subscription-based model, where clients register with the server, instead of the traditional Nagios-like model where the server requires a long list of clients, checks and other configuration. This makes Sensu particularly well-suited for a fast-changing and dynamic cloud infrastructure, where hosts may be added or removed on a regular basis.

In this tutorial, I’ll show how to install the various components that make up a basic Sensu installation, and the Uchiwa dashboard. In future articles I’ll show how to add checks, manage alerts, and configure a range 3rd party components such as the previously mentioned Graphite.

While the package management details, paths and so on are all Solaris-specific and use my packages, the steps and general principles behind this article should be applicable to other systems. You could also use a different source of packages (Solaris 11.2 will come with a bundled RabbitMQ server for example), or even compile your own manually.

For the sake of this tutorial, I’ll set up all the Sensu server components on a single machine, although there’s no reason why you couldn’t split them up over multiple hosts.

Install RabbitMQ and Redis

Firstly, we need to install two pre-requisites for Sensu – Redis and RabbitMQ. Redis is used by Sensu as a datastore, and RabbitMQ is used to pass messages (check commands, and the results of those checks) between the various components of Sensu and it’s clients.

To get started, add one of my package repositories to your system. You can pick either the /dev or /stable branches, depending on your requirements. See the documentation for more information on how to do this, and the difference between the repository branches.

Once you’ve done that, an install of redis and rabbitmq is simple :

$ sudo pkg install redis rabbitmq
           Packages to install:  4
       Create boot environment: No
Create backup boot environment: No
            Services to change:  1

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                                4/4     5189/5189    52.2/52.2    0B/s

PHASE                                          ITEMS
Installing new actions                     5690/5690
Updating package state database                 Done
Updating image state                            Done
Creating fast lookup database                   Done

You should now see both services come online :

$ svcs -a | egrep "(rabbit|redis)"
online         13:01:22 svc:/network/redis:default
online         13:01:24 svc:/network/rabbitmq:default

Testing the services

Redis is very easy to test with the “ping” command. If all is working properly, it should reply with “PONG” :

$ redis-cli> ping
PONG> exit

Your RabbitMQ installation can be verified by browsing to the login screen on port 15672 (e.g. http://address_of_your_server:15672). You should see the following :

RabbitqMQ login page

RabbitMQ creates a default user called “guest”, with the password also set to “guest”. However, it initially restricts access to this account from the local machine. As you’re most likely going to want to administer the RabbitMQ server from another machine (e.g. one with a GUI desktop and web browser), you’ll need to set up access control permitting this.

Following the example in the [Rabbit MQ manual] (, run the following to create a new, blank rabbitmq.config file, and unset the loopback_users configuration item so that all users are permitted to log in remotely :

$ echo "[{rabbit, [{loopback_users, []}]}]." | sudo tee /opt/mar/etc/rabbitmq/rabbitmq.config
$ sudo /sbin/svcadm restart rabbitmq

And after logging in as the default “guest” account, you’ll see the following page :

RabbitMQ main page

Configuring RabbitMQ

We now need to create a new “vhost” and user for Sensu to use. You can easily do this through the GUI, although I’ll show how to do it through the command line, so you can just copy & paste. First, create the vhost :

$ sudo rabbitmqctl add_vhost /sensu

Then add the user sensu with the password password123 (obviously, in a production environment, you’d change this to something more secure!) :

$ sudo rabbitmqctl add_user sensu password123

Finally, grant that user full permissions on the sensu vhost :

$ sudo rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"

If you want to be able to use that account to login to the web interface, you will also need to apply the administrator tag to the account :

$ sudo rabbitmqctl set_user_tags sensu administrator

Although doing this is discouraged for security reasons. If you want to remove the administrator tag after you’ve confirmed you can log in, simply set an empty list of tags :

$ sudo rabbitmqctl set_user_tags sensu

For more information on all these commands, see the rabbitmqctl man page

Install Sensu

Install the server and verify connections

Now we have the basic requirements setup, we can start to install the different components that make up a Sensu installation. Start by installing my sensu-server meta-package, which will pull in Ruby, various libraries, configuration files and a SMF manifest :

$ sudo pkg install sensu-server

You should then see the sensu-server service come online :

$ svcs sensu-server
STATE          STIME    FMRI
online         15:08:32 svc:/network/sensu-server:default

A quick look at the files under /opt/mar/etc/sensu/conf.d will show the reason: The configuration files telling Sensu where and how to connect to Redis and RabbitMQ all have the defaults set to the examples used above (e.g. localhost, with the default password of ‘password123’) :

$ cat /opt/mar/etc/sensu/conf.d/rabbitmq.json
  "rabbitmq": {
    "user": "sensu",
    "port": 5672,
    "vhost": "/sensu",
    "host": "localhost",
    "password": "password123"

You’ll obviously want to change these credentials in a production setting! You can also verify that the server has started correctly by looking at the RabbitMQ admin console. If you click on the “connections” tab, you’ll see a new connection ftom the sensu server :

RabbiqMQ connections

Install the API

The next component to install is the API service, which lets other tools communicate with the Sensu server. As before, this is a simple “pkg install” away :

$ sudo pkg install sensu-api

And verify :

$ svcs sensu-api
STATE          STIME    FMRI
online         15:21:49 svc:/network/sensu-api:default

You’ll also see another connection showing up in the RabbitMQ console.

Install the client

We’re installing a client on the same machine to demonstrate this process. Again, one more package to install :

$ sudo pkg install sensu-client

This however will fail to start :

$ svcs -xv
svc:/network/sensu-client:default (?)
 State: maintenance since Thu Jun 12 15:25:25 2014
Reason: Start method failed repeatedly, last exited with status 2.
   See: /var/svc/log/network-sensu-client:default.log
Impact: This service is not running.

The reason is simple – it needs a configuration to be present before it will start. If you look at the sample configuration provided (/opt/mar/etc/sensu/conf.d/client.json.example), you’ll see a few things that need to be changed. Copy this file and edit it :

 $ sudo cp /opt/mar/etc/sensu/conf.d/client.json.example /opt/mar/etc/sensu/conf.d/client.json
 $ sudo vi /opt/mar/etc/sensu/conf.d/client.json

And modify the address and hostname fields accordingly. Also, change the “subscriptions” to an empty array (we’ll cover that later). For example, on my host “” with the IP address of, my configuration looks like :

  "client": {
    "address": "",
    "safe_mode": false,
    "name": "",
    "subscriptions": []

Save this file, and then clear the service – you should see it come online immediately :

$ sudo /sbin/svcadm clear sensu-client
$ svcs sensu-client
STATE          STIME    FMRI
online         15:31:15 svc:/network/sensu-client:default

You’ll also see one more connection in the RabbitMQ console, and you’ll also see messages being passed between the components every few seconds :

RabbiqMQ connections


You can now use the “sensu-cli” tool (a 3rd party component, but one I’ve bundled with my sensu-common package) to further verify Sensu is up and running. On the first run, it will create a configuration file in your home directory, so you’ll need to enter the same command twice :

$ sensu-cli client list
We created the configuration file for you at /home/mark/.sensu/settings.rb.  
You can also place this in /etc/sensu/sensu-cli. Edit the settings as needed.

$ sensu-cli client list
safe_mode:  false
subscriptions:  []
timestamp:  1402583695
1 total items

We’ll now also install the Uchiwa dashboard, to provide a graphical view into Sensu. This is a 3rd party tool written in Node.js and is a big improvement over the default sensu-dashboard tool. Installation is straightforward from my package repositories :

$ sudo pkg install uchiwa
$ svcs uchiwa
online         13:51:22 svc:/network/uchiwa:default

And you can now browse to http://address_of_your_server:3000, and login with the username “admin” and password “secret” (change these in /opt/mar/etc/uchiwa/config.js). You’ll see the following screen :

Uchiwa dashboard

And there you have it! A very basic Sensu installation. Of course, it doesn’t actually monitor anything yet, but I’ll cover that in a future document. Any questions of feedback, just use the comments in this blog article. See you next time!

Ruby Gems Update

| Comments

I had a bash at finishing off the Ruby gem dependencies for Sensu on Solaris 11 over the last few days (and a bunch of other stuff,but I’ll write about that a bit later).

These have been pushed out to the /dev branch of my repositories, and should be available via a simple “pkg” install command. I’m going to build various meta-packages that pull all this in automatically, but for now if you want to give them a try, just use something like :

$ sudo pkg install 'pkg://markround/mar/libraries/rubygem-*'
           Packages to install:  52
            Packages to update:   1
       Create boot environment:  No
Create backup boot environment: Yes

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                              53/53   13711/13711    30.4/30.4    0B/s

PHASE                                          ITEMS
Installing new actions                   16437/16437
Updating modified actions                        1/1
Updating package state database                 Done
Updating package cache                           1/1
Updating image state                            Done
Creating fast lookup database                   Done

These are again Solaris 11 x86-only. Some of these will have to remain that way – in particular, the libv8 and rubyracer gems require a x86 architecture as the V8 Javascript Engine doesn’t support SPARC. However, these are only used by the sensu-dashboard gem, so as long as you don’t require the GUI components you should be OK. The whole Sensu architecture is distributed though, so you can always run the dashboard elsewhere on a supported platform.

Anyway, after installing you should end up with the following gems ready to use :

$ gem list

*** LOCAL GEMS ***

addressable (2.3.6)
amq-client (1.0.2)
amq-protocol (1.2.0)
amqp (1.0.0)
async_sinatra (1.0.0)
bigdecimal (1.2.0)
builder (3.2.2)
bundler (1.6.2)
carrot-top (0.0.7)
coffee-script (2.2.0)
coffee-script-source (1.7.0)
commonjs (0.2.7)
cookiejar (0.3.0)
daemons (1.1.9)
em-http-request (1.0.3)
em-redis-unified (0.4.2)
em-socksify (0.3.0)
em-worker (0.0.2)
eventmachine (1.0.3)
execjs (2.0.2)
handlebars_assets (0.15)
hike (1.2.3)
hirb (0.7.1)
http_parser.rb (0.6.0)
io-console (0.4.2)
ipaddress (0.8.0)
json (1.7.7)
less (2.5.0)
libv8 ( x86_64-solaris-2.11)
minitest (4.3.2)
mixlib-cli (1.4.0)
mixlib-config (2.1.0)
mixlib-log (1.6.0)
mixlib-shellout (1.3.0)
multi_json (1.9.2)
ohai (6.16.0)
oj (2.0.9)
psych (2.0.0)
rack (1.5.2)
rack-protection (1.5.2)
rainbow (2.0.0)
rake (0.9.6)
rdoc (4.0.0)
ref (1.0.5)
sass (3.3.4)
sinatra (1.3.5)
slim (2.0.2)
sprockets (2.12.0)
systemu (2.6.4, 2.5.2)
temple (0.6.7)
test-unit (
therubyracer (0.12.1)
thin (1.5.0)
tilt (1.4.1)
trollop (2.0)
yajl-ruby (1.2.0)
yui-compressor (0.12.0)

Ruby Solaris Packages and Gems

| Comments

My work on packaging up Sensu for Solaris 11 continues, and I’ve just tackled the Ruby 2.0.0 package and some associated Ruby Gems. These are all pre-compiled x86 Solaris 11 IPS packages and are currently available in the /dev branch of my repositories. To install everything in one go, just do the following :

$ pkg install pkg:/mar/runtime/ruby pkg:/mar/libraries/rubygem-*

               Packages to install: 17
       Create boot environment: No
Create backup boot environment: No

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                              17/17   20907/20907    19.6/19.6  315k/s

PHASE                                          ITEMS
Installing new actions                   23951/23951
Updating package state database                 Done
Updating image state                            Done

And then you should be all set up :

$ ruby -v
ruby 2.0.0p481 (2014-05-08 revision 45883) [i386-solaris2.11]

$ gem list
*** LOCAL GEMS ***

amq-protocol (1.9.2)
amqp (1.3.0)
async_sinatra (1.0.0)
bigdecimal (1.2.0)
builder (3.2.2)
bundler (1.6.2)
daemons (1.1.9)
em-redis-unified (0.4.2)
em-worker (0.0.2)
eventmachine (1.0.3)
io-console (0.4.2)
json (1.7.7)
minitest (4.3.2)
oj (2.0.9)
psych (2.0.0)
rack (1.5.2)
rack-protection (1.5.3)
rake (0.9.6)
rdoc (4.0.0)
sensu-transport (0.0.1)
sinatra (1.3.5)
test-unit (
thin (1.5.0)
tilt (1.4.1)

See the docs for more information on how to get started and switch repositories if needed. Note that while these packages are still a “work in progress”, I’ve found them perfectly stable so far for my requirements. But obviously, the ususal disclaimer applies, and these packages may crash your system, delete data, kill your cat, steal your food and so on…

Redis Solaris Packages

| Comments

As part of my work to package up the fantastic Sensu monitoring framework for Solaris 11, I have just uploaded a complete package of Redis 2.8.9 to my IPS package repositories (x86 only at the moment – see the docs linked below). This also includes a SMF manifest, so a pkg install redis should provide with all you need to get going straight away:

$ pkg install redis
           Packages to install:  1
       Create boot environment: No
Create backup boot environment: No
            Services to change:  1

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                                1/1           9/9      0.9/0.9  880k/s

PHASE                                          ITEMS
Installing new actions                         22/22
Updating package state database                 Done
Updating image state                            Done
Creating fast lookup database                   Done

$ /opt/mar/bin/redis-cli> ping

It’s currently available in both the /dev and /stable branches – for more information and a quick overview on how to use my repositories, see the documentation and user guide. I aim to get the rest of the components such as RabbitMQ and so on sorted over the next few weeks, so very soon you should hopefully be able to do a simple pkg install sensu-server or pkg install sensu-client on Solaris 11 x86.

Solaris 11.2 Beta

| Comments

Yesterday, Oracle announced Solaris 11.2 which includes a lot of interesting new features; not least of which is a full OpenStack distribution. There’s a lot of other improvements as well to all areas of the OS, from ZFS administration to the IPS system and Automated Installer. It also looks like Puppet is now included for systems management. All in all, a good release although it still pains me to see the “Oracle” logo slapped over everything, as well as the general lack of interest in anything Solaris-related these days. Hopefully this can re-kindle some excitement in what is after all, a ground-breaking and incredibly solid platform.

Anyway, one thing that was missing from the announcement was the simple set of steps needed to update a current 11.1 system to the new beta. The beta documentation assumes that 11.2 will be in a production IPS repository, so instead you’ll simply need to switch to the new Beta repository and set it as your “solaris” publisher, upgrade and reboot :

root@solaris:~# pkg set-publisher -O solaris
root@solaris:~# pkg update --accept

A short download and update will then happen…

            Packages to remove:   5
           Packages to install:  94
            Packages to update: 478
           Mediators to change:   2
       Create boot environment: Yes
Create backup boot environment:  No

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                            577/577   36168/36168  574.0/574.0  386k/s

PHASE                                          ITEMS
Removing old actions                       7121/7121
Installing new actions                   26702/26702
Updating modified actions                26093/26283
Updating modified actions                26283/26283
Updating package state database                 Done
Updating package cache                       483/483
Updating image state                            Done
Creating fast lookup database                   Done

A clone of solaris exists and has been updated and activated.
On the next boot the Boot Environment solaris-1 will be
mounted on '/'.  Reboot when ready to switch to this updated BE.

Note the last part – one of the things I love about IPS and Solaris 11 is the creation of boot environments. I can see that my system is ready to reboot into Solaris 11.2 :

root@solaris:~# beadm list
BE        Active Mountpoint Space Policy Created
--        ------ ---------- ----- ------ -------
solaris   N      /          8.26M static 2014-04-28 10:41
solaris-1 R      -          5.81G static 2014-04-30 11:52

So a quick reboot follows :

root@solaris:~# shutdown -i6 -g0 -y

And there it is…

Last login: Wed Apr 30 11:38:32 2014 from xxyyzz
Oracle Corporation      SunOS 5.11      11.2    April 2014
mark@solaris:~$ uname -a
SunOS solaris 5.11 11.2 i86pc i386 i86pc
mark@solaris:~$ head /etc/release
                             Oracle Solaris 11.2 X86
  Copyright (c) 1983, 2014, Oracle and/or its affiliates.  All rights reserved.
                             Assembled 23 April 2014

And, thanks to the magic of ZFS and Boot Environments, I can quickly switch back to 11.1 if there’s a problem :

mark@solaris:~$ beadm list
BE        Active Mountpoint Space Policy Created
--        ------ ---------- ----- ------ -------
solaris   -      -          9.12M static 2014-04-28 10:41
solaris-1 NR     /          6.03G static 2014-04-30 11:52

mark@solaris:~$ pfexec beadm activate solaris
mark@solaris:~$ beadm list

BE        Active Mountpoint Space Policy Created
--        ------ ---------- ----- ------ -------
solaris   R      -          4.23G static 2014-04-28 10:41
solaris-1 N      /          2.40G static 2014-04-30 11:52

mark@solaris:~$ pfexec init 6

Reboot, and I’m back to where I started. Much smoother than some upgrades I’ve had in the past :) Here’s to the full 11.2 release!

Behringer DEQ2496 Review for Bassists

| Comments

I’ve just posted a little review/guide to the Behringer DEQ2496 unit, and how it works in a bass player’s rack. I made this video because I’ve seen lots of clips and reviews of the DEQ2496, but none focusing on using it in a rack setting for a gigging bassist. It offers a lot of useful features such as a graphical/parametric EQ, compressor, limiter, feedback suppressor and more. In this video I introduce the unit, cover basic setup and look at the EQ modules.

Any questions, please leave a comment and I’ll do my best to answer! Part 2 coming soon…


| Comments

Well, it was about time. This blog has been stagnating for a long time, partly due to the clunky PHP-based system that used to run it. Needless to say, although I used to do a lot with PHP, I’m now old enough to know better. I’ve therefore done a “rip & replace” update on this site, switching instead to the wonderful, Ruby-based Octopress – a static blogging system built on top of Jekyll.

Besides the warm and fuzzy feeling of booting PHP out and using a lovely Ruby codebase, this also brings other benefits such as a massively reduced amount of effort required to manage this site and create content. Plus, it’s one less bit of software I have to worry about patching and securing.

I’ve ported most of the old site’s content over, although comments haven’t made the transition. I’ve added Disqus comments as a replacement, and will also look at extracting the old comments from the database so no information will be lost. You’ll also have to excuse any graphical glitches; the article’s HTML should be displayed correctly, but I’ve already had to make a couple of manual tweaks here and there, and will fix up things as I go along. Anyway, onwards and upwards!