markround.com

DevOps, Sound Engineering and other things of interest...

Three Years of Tiller

| Comments

On July 18th, 2014 – 3 years to the day of writing this post – I pushed the first public code of Tiller to Github. Back then, it was just a simple little tool that I wrote (mostly as a learning exercise), found useful and thought others may like to use.

Since then, there have been:

  • Nearly 60,000 downloads from rubygems.org
  • 575 commits
  • 244 stars on Github
  • 54 releases
  • 52 issues closed
  • 16 new plugins created
  • 11 pull requests from other developers
  • Over 2,000 lines of documentation written

And countless emails, comments and discussions on the chatroom.

I’d be the first to admit that it’s certainly not the best code you’ll ever see, and in terms of project size it’s tiny compared to some of the other things you’ll see hosted on Github. But something that was once solely mine has clearly resonated with others (The 1.1.0 release actually contained more of other people’s code than my own changes!), spawned a community and has found a place as a building block in projects I’d never have imagined, from people I’ve never even met.

It’s a feeling I didn’t expect to experience outside of my musical endeavours. As when I’ve slaved over a mixing desk for days or spent hours trying to perfect a bass riff, it’s been a labour of love. For me, it’s the same nerve-racking sensation publishing a mastered musical track, or pushing a new code release to rubygems.org. I find myself second-guessing my decisions every time, and – despite working on these things because it’s like an itch I have to scratch – I can’t help but wonder whether anyone will like what I’ve just created.

Just like making music, I strongly believe that code can be an art form. It’s something that ends up reflecting a little of yourself – and whatever your endeavours, you’ll only ever get better at as you continue, receive helpful criticism and words of encouragement.

So here we are, 3 years later from that first tentative code push. It’s immensely satisfying to see a community spring up around this little tool, and I want to take a moment to thank everyone who has ever emailed me, submitted a bug report, come up with suggestions, submitted code or otherwise joined in helping this project fill a niche. You’re my encouragement to keep up with this project, learn more, grow as a developer and above all – keep having fun!

Recent updates

Anyway, enough of my meandering musings.. It’s been quite a while since I last posted about Tiller, but that doesn’t mean things have been quiet! On the contrary, development is busier than ever and there have been a lot of new features and improvements added over the last year. There is of course the Changelog where you can see all these updates, but I thought I’d take the opportunity to provide a quick round-up of some of the highlights.

0.9.0

Tiller 0.9.0 was released on the 10th of August 2016, and the 0.9 series included a lot of new features and some nice improvements:

Precedence changes

In previous versions of Tiller, global values were merged together in the order that plugins were loaded. Then, the same was done for template values. Finally, template values were merged over the top of global values. As had been discussed in the Gitter chatroom many times, this led to some counter-intuitive behaviour! Starting with Tiller 0.9.0, the behaviour has now been greatly simplified.

We now go through the plugins in order, and for each one we merge template values over global values, then proceed onto the next plugin. To put it simply: A template value will take priority over a global value, and any value from a plugin loaded later will take priority over any previously loaded plugins.

Vaults!

The 0.9.x series also saw two “Vault” plugins. The first supports Hashicorp’s Vault product and was provided in an awesome Pull Request including test cases and documentation from the fantastic liquid-sky.

The second plugin adds support for Ansible Vault – a simple way of encrypting a YAML file and passing it into a Docker container. I’m particularly fond of this plugin as it means it’s trivial to ship a safely encrypted set of YAML values or credentials inside your container, and unlock it at run-time. Even if you’re not using Ansible for orchestration or configuration management (and you should!), Ansible vault is simple to use:

1
2
$ ansible-vault create my_secret_vars.yaml
Vault password: <enter password here>

And that’s it! You can then edit the file with ansible-vault edit and bundle it into your container. See the plugin documentation for more examples.

Config.d

Tiller 0.9.5 added support for merging configuration in a config.d directory structure (and 0.9.6 improved it). This was another simple change suggested by a user (thanks, rafik777!) but means it’s easier now to create layered containers and split configuration out over multiple files.

1.0.0

The big 1.0.0 release! This marked the start of using semantic versioning, and as always, a big priority is never to break existing installs. I now strongly recommend a modern Ruby installation (see the requirements page for more details) but will continue to test on older installations.

Apart from some nice new features such as exec on write and dynamic values, the big highlight for me of this series so far is the 1.1.0 release. As I mentioned above, for the first time since I started this little project, I made a release with more of other people’s changes in it than my own code!

The future

There’s several new features coming in future releases, and I’ve already started to lay the foundations for plugin versioning. This means I (and other plugin authors) can experiment with new features or radically alter the internal workings of Tiller without breaking compatibility with existing installations. The first thing that will use this will be a new feature to specify multiple “targets” per template. So you’ll be able to do something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
vhost_template.erb:
  - target: /etc/httpd/sites_available/my_site.conf
    user: root
    group: httpd
    perms: 0644
    values:
      site_name: "My site"
      site_fqdn: www.mysite.example.com
  - target: /etc/httpd/sites_available/other_site.conf
    user: root
    group: httpd
    perms: 0644
    values:
      site_name: "My other site"
      site_fqdn: www.myothersite.example.com

… and so on. As always, feel free to submit an issue, join the chatroom or send me an email with any suggestions or improvements you’d like to see. And keep having fun!

Tiller 0.8.0 Released!

| Comments

Tiller has just seen its 0.8.0 release, and as you’d expect from a major version bump, it’s a big one. There’s some breaking changes if you write your own plugins – nothing too major, a quick search & replace should fix things for you. But, more importantly, there’s a major new feature which brings two big improvements.

I’ve added a “helper modules” feature, which lets you group together custom utility functions in Ruby that can be called from within templates. And I’ve included a built-in function called Tiller::render that lets you include and parse nested sub-templates from another template.

Sub-Templates

Now you can include other templates in your templates by using the built-in Tiller::render helper function. This may be useful if you have a large configuration file to generate: you can now split it up into separate files for each section and combine them together at run-time. Or, you may need to generate a lot of similar configuration blocks (for example, Nginx virtual hosts). You can now iterate over a list and render a single sub-template for each block.

For example, if you have a template called main.erb, you can include another template called sub.erb by calling this function inside main.erb:

main.erb
1
2
3
This is the main.erb template. 
This will include the sub.erb template below this line:
<%= Tiller::render('sub.erb') -%>

You can nest sub-templates as deeply as you wish, so you can have sub-templates including another sub-template and so on.

Helper modules

The sub-template system builds on top of a new “helper module” feature. A helper module is intended to group together small blocks of code that you can include in your templates to perform mark-up or other utility functions. They aren’t intended to replace the existing Data- and Template-source plugins; if you need to get some values into your templates, or hook up to some external service, these are probably still the best way to go about it.

But you can see how if you had a more complicated transformation to do (e.g. convert markdown text into HTML) or needed to include some logic in a function, this would help clean up your templates, as well as keep a clean separation of code and configuration.

As an example, this is how you’d add a Lorem Ipsum generator for filling in place-holder text. We’ll simply wrap the excellent forgery gem, so first make sure you have it installed:

1
2
3
4
5
$ gem install forgery
Successfully installed forgery-0.6.0
Parsing documentation for forgery-0.6.0
Done installing documentation for forgery after 0 seconds
1 gem installed

I’ll also assume you are using the default directory for custom plugins; if you want to change this, use the -l/--lib-dir option to Tiller to point to somewhere else.

First, create a file named /usr/local/lib/tiller/helper/lorem_ipsum.rb , and put the following code inside:

lorem_ipsum.rb
1
2
3
4
5
6
7
require 'forgery'

module Tiller::LoremIpsum
  def self.words(num)
    Forgery(:lorem_ipsum).words(num)
  end
end

Note that I defined this inside a custom namespace so it doesn’t clash with anything else. You can then load this module by adding the following to the top-level of your common.yaml:

1
helpers: [ "lorem_ipsum" ]

Now, in your templates you can call this function like so:

1
This is some place-holder content : <%= Tiller::LoremIpsum.words(10) %>

When you run Tiller with the -v (verbose) flag, you’ll see Helper modules loaded ["lorem_ipsum"] amongst the output, and your template will contain the following text when generated :

1
This is some place-holder content : lorem ipsum dolor sit amet consectetuer adipiscing elit proin risus

Tiller 0.8 Changes for Custom Plugins

| Comments

Posted this in the Gitter chat but just to spread it to a wider audience : Tiller 0.8.x will be coming in a little bit; the reason for the 0.8 version bump is that there are some internal changes which will break any custom plugins you may have written. The relevant commit is here: markround/tiller@c2f6a4f.

In short, I’m moving from per-instance @log and @config vars, and instead moving to a singleton pattern so there are now single Tiller::log and Tiller::config module variables. A simple search-and-replace (e.g. s/\@log/Tiller::log/g and s/\@config/Tiller::config/g) on your custom plugins should be all you need once 0.8.x is released.

The Pleiades

| Comments

And now a diversion from most of my geeky posts! I’ve just finished (well, as “finished” as most of my musical projects get) my latest track: “The Pleiades”, featuring the talents of my sister, psy-trance producer Spinney Lainey on flute. I’ve still got a long way to go on my journey through the world of music production, but this is the first thing I’ve felt more or less happy with and wanted to share it with the world. Hope you enjoy!

New Consul Plugin for Tiller

| Comments

It’s only a minor version number bump, but Tiller 0.7.8 now brings a major new plugin which I’m really excited about : It now supports the awesome Consul system from Hashicorp. Consul is a distributed key/value store and service discovery mechanism – the basic idea is that you have a Consul cluster consisting of multiple nodes for high availability and performance. You then place all your configuration values in it, and also register your services (like web server backends, databases, cache nodes and so on) with it. This means you can have a dynamic environment where components discover their configuration and other nodes in your infrastructure on-the-fly: no more hard-coding database URIs or load balancer pools!

This makes it an ideal fit for a “cloud” Docker environment using something like Swarm, Kubernetes or Mesosphere/Marathon. Your containers can run on any host, advertise whatever ports they like, and Consul will make sure that everything can find what it needs to. The only sticking point is how to get your configuration to your applications.

If you’re writing your own microservices from scratch, you can of course talk directly to the Consul API, but for other things that require configuration files (Nginx, Apache, Rails applications and so on) you need a tool to talk to Consul and generate the files for you. Hashicorp (the company behind Consul) do have a tool called consul-template which does this, but Tiller has (to my admittedly biased point of view!) several advantages, not least the ability to use straight-forward Ruby ERB templates and embedded Ruby code instead of Go template syntax, and the ability to load other data source plugins.

So you although you can fetch everything from Consul, Tiller lets you do things like store templates in files or retrieve them from a HTTP web service, and then “chain” key/value stores : Provide values from defaults files first, then check Consul, and finally over-ride some settings from environment variables at run-time.

If you’re new to Tiller, I recommend having a quick look at the documentation, and following some of my Tiller blog posts, in particular this article which walks through a practical example of using Tiller inside Docker.

That said, here follows a quick example of using Tiller to fetch data from Consul. In the examples below, I’m just generating some demonstration template files for simplicity. In a real-world situation, these template files would be application configuration files like nginx.conf, mongod.conf and so on.

Getting started

The Tiller Consul plugin requires the diplomat Ruby gem to be installed, so assuming you have a working Ruby environment, this should be all you need:

1
2
3
4
$ gem install diplomat tiller
Successfully installed diplomat-0.17.0
Successfully installed tiller-0.7.8
2 gems installed

Start a Consul server

Go to the Consul downloads page, and grab both the Web UI and binary download for your platform, and unzip them in the same location. I’ll do this with shell commands below, for the Mac OS platform (replace “darwin” with “linux” if you are using a Linux system):

1
2
3
4
5
$ mkdir consul
$ cd consul
$ wget https://releases.hashicorp.com/consul/0.6.4/consul_0.6.4_web_ui.zip
$ wget https://releases.hashicorp.com/consul/0.6.4/consul_0.6.4_darwin_amd64.zip
$ for z in *.zip; do unzip $z; done

Now, start up Consul in stand-alone mode :

1
2
3
$ ./consul agent -server -bootstrap \
  -client=0.0.0.0 -data-dir=./data \
  -ui -ui-dir=. -advertise=127.0.0.1

You’ll see some startup messages :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
==> Starting Consul agent...
==> Starting Consul agent RPC...
==> Consul agent running!
         Node name: 'mdr.local'
        Datacenter: 'dc1'
            Server: true (bootstrap: true)
       Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
      Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
    Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
             Atlas: <disabled>

==> Log data will now stream in as it occurs:

    2016/05/12 19:56:14 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state

Leave this process running in your shell, and with a browser check your server is up and running by visiting [http://localhost:8500/ui]. You should see a screen similar to the following :

Default consul screen

We’ll now populate it with some test data. I have a script to do this, so download and run it, passing the base URL to your Consul server as the only argument :

1
2
3
4
$ wget https://raw.githubusercontent.com/markround/tiller/master/features/support/consul_test_data.rb
$ chmod +x consul_test_data.rb
$ ./consul_test_data.rb http://localhost:8500
Populating Consul at http://localhost:8500 with test data

Now, if you visit your Consul page and click on the “Key / Value” link at the top, you’ll see a bunch of data under the /tiller path. Here’s the view of the global values :

Populated consul screen

And if you click around further, you can find the template definitions also stored in Consul :

Template view inside consul

Incidentally, this is the same data that is used in my automated tests that check all the features of Tiller are working correctly whenever I make any changes. You can see the results of these tests over at Travis CI, or run them yourself if you clone the Git source and run bundle exec rake features

Tiller setup

Now your Consul server is ready to go, so here’s how to hook Tiller up to it. Just create your common.yaml file with the following contents (just running “true” after we’ve finished for demonstration purposes – in a real Docker container, this would be your application or webserver binary etc.):

common.yml
1
2
3
4
5
6
7
8
9
---
exec: ["true"]
data_sources: ["consul"]
template_sources: [ "consul" ]

consul:
  url: "http://127.0.0.1:8500"
  register_services: true
  register_nodes: true

And run Tiller against it, using the “development” environment :

1
2
3
4
5
6
7
8
9
10
11
12
$ tiller -b . -v -e development
tiller v0.7.8 (https://github.com/markround/tiller) <github@markround.com>
Using configuration from .
Using plugins from /usr/local/lib/tiller
Using environment development
Template sources loaded [ConsulTemplateSource]
Data sources loaded [ConsulDataSource]
Available templates : ["template1.erb", "template2.erb"]
...
...
Child process exited with status 0
Child process finished, Tiller is stopping.

And there you have it. You’ll have ended up with a couple of files : template1.txt and template2.txt in your current directory, which have been entirely populated with templates and values all served from Consul:

1
2
3
4
5
6
7
8
9
10
11
$ cat template1.txt

This is template1.
This is a value from Consul : development value from consul for template1.erb
This is a global value from Consul : consul global value
This is a per-environment global : This is over-written for template1 in development
If we have enabled node and service registration, these follow.
Nodes : {"mdr.local"=>"127.0.0.1"}
Services : {"consul"=>[#<OpenStruct Node="mdr.local", Address="127.0.0.1",
ServiceID="consul", ServiceName="consul", ServiceTags=[], ServiceAddress="",
ServicePort=8300, ServiceEnableTagOverride=false, CreateIndex=3, ModifyIndex=4>]}

Have a try at running Tiller with the “production” environment and see what changes. You can also try changing the values or even the templates themselves inside Consul to see the changes reflected whenever you run Tiller.

What next ?

So, assuming you’re using Tiller as a Docker CMD or ENTRYPOINT to generate your configuration files before handing over to a replacement process, you can now create dynamic containers that are populated with data entirely from a Consul cluster. Or you can chain one of the many other plugins (environment, environment_json, defaults, file and so on) together, so your container can get core values from variety of sources.

You can also make use of Consul’s service registration system (perhaps by using the fantastic Registrator) and populate your configuration files dynamically with auto-discovered microservice backends and much more.

Check out the full Consul plugin documentation for more information, read the rest of the docs or drop by the new Gitter chatroom for help and advice.

As always, all feedback and comments welcome!

Tiller 0.7.7 and File Globals

| Comments

Thanks to a nice suggestion by kydorn in issue #18, the file datasource supports a global_values: block, so you can now create defaults per environment (and use the defaults datasource to provide a default for all environments). This is available in Tiller 0.7.7, which was just released.

This means if you have a common value that you want available to all templates (but may be different in each environment), you don’t have to repeat the definition for every template. In conjunction with the defaults datasource, it lets you do things like :

  • Use the defaults datasource to specify default values for all environments
  • Use global_values: for defaults specific to each environment
  • Optionally over-write them with local config: values on templates

There’s an example now in the README.md which demonstrates this :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
data_sources: [ 'defaults','file','environment' ]
template_sources: [ 'file' ]

defaults:
  global:
    per_env: 'This is the default across all environments'

environments:

  development:
    global_values:
      per_env: 'This has been overwritten for the development environment'

    test.erb:
      target: test.txt
      config:
        per_env: 'This has again been overwritten by the local value just for this template'

  production:

    # This will get the value from the defaults module, as we don't specify a
    # per-environment or any per-template value overwriting it.
    test.erb:
      target: test.txt

Hopefully that will cut down the size of some config files, and also introduce some more flexibility to your Tiller setup. Thanks again, Kydorn, for the suggestion!

As always, please feel free to send any other ideas/suggestions/bugs my way – either through the GitHub issue tracker, email, or now through the new Gitter.im chatroom.

Tiller 0.7.6 and Environment JSON V2

| Comments

I recently released Tiller 0.7.6 with a new feature for the environment_json plugin. This can be used to provide dynamic configuration for Docker containers, using JSON in an environment variable. I previously posted a complete example of how you’d make use of this. However, most Tiller plugins provide values as “global” sources, which will be over-ridden by local, template-specific values. The documentation makes note of this in the “Gotchas” section, but there’s now another alternative to using global plugins all the way through.

If your JSON structure contains a "_version" : 2 key:value pair, Tiller will source global values from a global key, and treat other keys as being template-specific. For example :

tiller_json
1
2
3
4
5
6
7
8
9
{
  "_version" : 2,
  "global" : {
    "global_value" : "This is a global value available to all templates"
  },
  "template.erb" : {
    "local_value" : "This will create or over-ride the 'local_value' only on template.erb"
  }
}

Only a minor update, but hopefully it should add some flexibility around value precedence :)

Tiller 0.7.0 and Simpler Configuration Files

| Comments

Yesterday, I released Tiller 0.7.0, and as you might imagine with a version “bump” from the 0.6.x series, there’s a few changes. Some of it is internal stuff that you’ll probably only notice if you’re building your own plugins for Tiller, but there is a fairly big new feature for end users: Single-file configuration.

Recently, when I’ve written some documentation or examples on how to use Tiller, I’ve been struck at how I had to create several files in different directories just to illustrate a simple point. This approach has some benefits when dealing with larger configurations or inherited Dockerfiles, but for simple jobs it’s a little unwieldy.

Take for instance my JSON walkthrough example as a case-in-point: You have to create 3 different files (4 if you include the template) before you can get started. Instead, with the new format common.yaml, we can put everything apart from the template content in a single file:

common.yaml
1
2
3
4
5
6
7
8
9
10
11
data_sources: [ 'defaults', 'file', 'environment_json' ]
template_sources: [ 'file' ]

defaults:
    global:
        default_value: 'This is a default global value that may be replaced'

environments:
    development:
        json_template.erb:
            target: 'parsed_template.txt'

I’m sure you’ll agree that makes things much easier to read and follow!

Admittedly, for complex environments you may well want to keep the “one file per environment” approach – particularly if you have a base Docker container with a standard common.yaml that all your other containers inherit. For that reason, Tiller will continue to support both methods and I have no intention of removing support for “old style” configuration. So don’t think of the way things used to be done as deprecated in anyway, it’s just that there’s now a new approach that may make your life easier :)

As always, I welcome all feedback, bug reports & feature requests – just leave a comment on this blog, email me, or open a ticket on the GitHub issues page.

Centralized Docker Container Configuration With Tiller

| Comments

If you’ve been using Docker for a while, you probably know that you can use Tiller to generate configuration files inside your containers. This means you can provide a single container for running in a variety of different environments (think of a web application that needs different database URIs and credentials depending on where it is run).

You can also use it to provide ‘parameterized’ containers – where you allow end-users to provide some aspects of the configuration. For example, the team over at StackStorm are using Tiller and the environment plugin to let users easily configure their containers. I’ve provided a variety of plugins to help with this, and have written about it previously.

There was always a catch, though. Until recently, Tiller obtained configuration from a ‘local’ source – either configuration files bundled in a container, or from environment variables passed in at container run time. This is probably fine for most situations, but did prevent a lot of interesting use-cases. However, the 0.6.x releases now support fetching configuration and templates from a variety of network sources. Currently (as of 0.6.1), you can retrieve data from either a ZooKeeper cluster or a HTTP server, with more services to be supported in later releases.

Update: Tiller now supports Consul!

This is a massive change as it now means you can have a central store for all your container configuration, and opens the door for a whole bunch of really cool possibilities. You could, for example, have a service that allows users to edit configuration files or set parameters through a web interface. Or you could plug Tiller into your automation stack so you can alter container configuration on the fly. You could even use Tiller “on the metal” to configure physical/VM hosts when they boot.

Of course, this does introduce an external dependancy on launching your containers, so you should ensure that before you implement this in a production setting you have carefully considered all your points of failure!

Enabling these plugins is straightforward – simply add them to common.yaml, e.g.

common.yaml
1
2
3
4
5
6
7
data_sources:
  - file
  - http

template_sources:
  - http
  - file

As with all Tiller plugins, the ordering is significant. In the above example, values from the HTTP data source will take precedence over YAML files, but templates loaded from files will take precedence over templates stored in HTTP. You should tweak this as appropriate for your environment.

You also do not need to enable both plugins; for example you may just want to retrieve values for your templates from a web server, but continue to use files to store your actual template content.

As the HTTP plugins can fetch everything (a list of templates, contents and values) from a webservice, if you accept the defaults your environment configuration blocks can literally now be reduced to :

environment.yaml
1
2
3
common:
  http:
    uri: 'http://tiller.example.com'

You can check out the documentation for these new plugins over at the Github project page. If you have any suggestions for improvements or feature requests, please feel free to open an issue or leave a comment below!