Mark's blog

Xenserver snapshot and template based backup script

We have recently started using Citrix Xenserver in production at work (fantastic product, I must do a write-up soon!) and needed a simple backup solution. Our VMs run from an iSCSI SAN and are backed up daily through various methods - e.g. Bacula for the Unix/Linux systems. However, we wanted the ability to quickly roll back to a previous VM snapshot, and get up and running quickly if our SAN failed for whatever reason. Our solution was to create a large shared NFS storage repository, and periodically snapshot VMs and copy the templates over to this SR. Doing this means that if the SAN fails, we can create a new VM quickly from this NFS store (using the Xenserver's local disks, or even the NFS SR itself as storage). Once up and running, we can bring VMs back up to date by restoring the latest backup to them.

In order to automate this, I wrote a quick script which I thought may prove useful to someone else, so decided to post it here : snapback.sh.

It is very simple, and although it may serve well as your only backup solution, it's really intended as an image-level compliment to your primary file-system based backup system such as Bacula, Amanda, Netbackup etc. It also has not had much testing, and I fully appreciate the scripting is pretty rudimentary and could do with some optimisation - there's no error checking, for instance. I kept it pretty verbose on purpose though, so you can get a good idea of exactly what it's doing at each step; it may be better to think of this as a template you can base your own scripts off!

Overview

The script creates a snapshot of a running VM on a configurable schedule, and then creates a template from this snapshot. It will copy all these backup templates over to a configurable storage repository, and then clean up any old backups according to a specified retention policy. These backups are full backups, so if you have a 10GB VM and keep 7 previous copies you will need a total of 80GB disk space on your backup VM. Non-running VMs, and those not configured (as detailed below) will be skipped.

Installation and usage

First, copy the script to your Xenserver pool master, and make it executable. A good location for this is /usr/local/bin/snapback.sh.

Next, create a cron entry for the script - to make it run daily just after 1AM, you'd create /etc/cron.d/backup with the following contents :

2 1 * * * root /usr/local/bin/snapback.sh > /var/log/snapback.log 2>&1

This will also record a log of it's actions to /var/log/snapback.log. You now need to edit the script and change the DEST_SR variable to the UUID of your backup storage repository. You can find this value by clicking on the SR in Xencenter; the UUID will be displayed as a value like "2c01dc26-f525-70d6-dedf-00baaec76645".

Lastly, you need to configure your backup and retention policy for your VMs. In Xencenter, right click your VM, and select "Properties". Click on "Custom Fields", and then "Edit Custom Fields". You should add two text fields :

backup : Can be one of "daily", "weekly", or "monthly". If it is set to weekly, it will by default run on a Sunday, and if it set to monthly, it will run on the first Sunday of the month. This day can be changed at the top of the script - see the WEEKLY_ON and MONTHLY_ON variables.
retain : How many previous backups (in addition to the currently running backup) to keep. So, setting this to a value of "2" would mean that after a backup has run, you would end up with 3 backups in total.

Adding a custom field

The script will look for these fields when it is run, and will skip any VM that doesn't have them set. You can also see them in the Xencenter summary and properties for the VM :

VM summary showing the custom fields

You can now either run the script manually, or wait until the cron job kicks off. It will produce a detailed log to the console (or log file if run through cron), and when it's finished, you'll see your template backup VMs listed in Xencenter, similar to this :

Backups listed in Xencenter

If you find that this clutters up the Xencenter view a little, you can always hide them (View->Server View->Custom Templates). To restore a VM from a backup, just right click, and choose "New template from backup". Anyway, I hope this helps someone else!

Linux, Solaris and FreeBSD iostat monitoring with Cacti

I've been looking for ages for a tool to parse the output from "iostat" on Linux, and graph it in Cacti. I found a few scripts and templates that did some of what I was looking for (disk I/O etc.), but nothing that gave me the full set of statistics such as queue length, utilisation, service time etc. I finally got round to writing my own set of templates and a data gathering script to provide this information, and it seems to work very well. So that others can benefit, I've posted the package archive and a brief description over on the Cacti forums (click Continue Reading for a download link to an updated version - the one on the Cacti forums has a bug so that it won't work with all versions of sysstat). Below are a couple of sample graphs to give you an idea of what it can do - there's also a few more samples posted in the Cacti forums thread :

Displaying iostat's kB/s data source.

Timing information from iostat (average wait and service time)

Installation is a simple matter of creating a cron job to gather iostat data, extending your snmpd.conf to call the included iostat.pl script, and then importing the templates. Full instructions are included in the README within the archive (click the Continue Reading link to see them), but if you have any comments, suggestions or problems please let me know!

Continue reading "Linux, Solaris and FreeBSD iostat monitoring with Cacti"

The setting sun

Well, that's that, then. Solaris as we knew it is pretty much dead. I've suspected for a while now that Oracle's intentions regarding Solaris were not what the community, or us "old-school" Solaris sysadmins wanted or had hoped for.

In the last few months, Oracle have completely alienated and scared off the community around OpenSolaris, killed any lines of communication by clamping down on employee blogs and have ignored open letters from highly influential and important community members begging for any kind of information. Theyve forbidden Sun/Oracle employees from heading up the Solaris user groups and booted the meetings out of their buildings; turned Solaris 10 into a 90-day trial, and pushed back the 2010.x release of OpenSolaris with no word as to its planned release date, or even if it is being continued as a product. And now, in a final act of desperation, the OGB has essentially threatened to "shoot itself in the head".

Even ignoring the OpenSolaris project, It's not at all rosy in "real" Solaris land, either. Requests for information and clarification are going unanswered, and I know of several managers who have had hardware quotes and support tickets ignored - there's a near total blackout of information from Oracle. People are fleeing Solaris in droves, and migrating to anything they can: Linux, FreeBSD (Dtrace and ZFS), AIX - hell, even HP-UX looks like a safer bet at the moment. And I never thought I'd find myself saying that!

It certainly appears that Oracle are doing a superb job of killing Solaris. But why would they do this, having paid all that money for Sun and announcing that they will increase spending on Solaris development ?

Well, this post on Slashdot (allegedly from a Sun/Oracle employee) confirms my suspicions as to why they may be doing this. Oracle just really doesn't care about Solaris as a general purpose data centre OS any more. There's just no money in it, and although I personally find it tragic it does make sense. It's probably also why they're killing all their OEM deals - why help a competitor sell hardware, when all you'll see from it is a possible support contract for the OS ? Oracle's overall aim is to have Solaris relegated to the role of running as the bottom layer in an Oracle "database machine", Java appserver bundle or inside a "Fishworks" storage appliance.

It excels at these tasks, and it would obviously fit into Oracle's stated goal of being a one stop shop, where if you want to run Oracle, they'll sell you the bundle - hardware, storage, OS and software. If they no longer want Solaris to be a dominant general purpose OS, then their approach makes sense. They don't need a "community" around the product, they don't need open source developers porting applications to it, and they certainly don't need the overhead of running and managing a community portal any more. Unless you are running (and paying for) Oracle applications on Solaris, you're probably more of an annoyance to them at the moment and I get the very strong idea that they'd rather you just quietly went elsewhere.

I just wish that if this was their plan, they'd make some sort of statement about it; rather than ignoring the Solaris community in the hopes that they'll eventually get frustrated and leave without Oracle having to spell it out for them. I think the way they are going about it reprehensible and it's a tragic end for such a historic and innovative OS. Sadly though, Larry is all about the bottom line and the old, altruistic Sun approach just wasn't bringing in the big bucks. As the Slashdot poster said : "Profit is king here. Anything else is overhead, and overhead eats into Larry's yacht fund."

Edit: Now it's official : http://www.theregister.co.uk/2010/08/13/opensolaris_is_dead/

Leaving the iPhone for the Palm Pre

While I've been a long-time Apple user and fan of their hardware, I have to say that I've become a little disillusioned of late. Mostly, this has been down to the iPhone: While it's undeniably been a game-changer and raised everyone's expectations of what a mobile phone could do, there's just been an uncomfortable, claustrophobic feel around the whole thing. From the start, there were ridiculous NDAs preventing developers from even discussing programming the device; an insane culture of overbearing secrecy and control; not to mention Apple pulling applications from the app store or rejecting them seemingly on a whim, censoring content and being generally hacker-unfriendly.

The final straw however was the news that they are suing HTC over patent violation. Software patents are a terrible thing for this industry, and it appears that instead of competing on their own merits, Apple has chosen to resort to being a patent troll - unsurprising perhaps, given their past behaviour.

All this neatly coincided with my contract with O2 coming up for renewal, so I could get a new phone. I'd done some research and had narrowed down my iPhone alternatives to just 2 options - Palm Pre, or some Android-powered device. Android would have required me to leave O2, so that really just left the Palm Pre (and Windows Mobile or a Blackberry were right out). All the reviews I'd read were pretty good for the Pre but I wanted to try WebOS out first. Fortunately, this was easy as the Palm SDK includes an emulator. This got me hooked - not only is the emulator based on VirtualBox (which is massively cool), but all the tools such as "WebOS Doctor" are available for Windows, MacOS AND Linux. I even installed it with no problems on the latest Ubuntu Lucid Alpha.

Playing with WebOS blew me away. I won't go into too much detail, as all the review sites have covered the WebOS experience in far greater depth. I should point out though that since most of the reviews were written, there have been numerous software updates which have brought a great deal of new features and improvements - the last 1.4 update for instance added video recording support! I will however simply say this : It actually makes the iPhone feel clunky in comparison. The multitasking card system combined with the finger gestures is amazingly fluid and intuitive. Little touches throughout such as the notification system, "synergy" contacts system and email handling put this streets ahead of the iPhone.

I was also pleased to see that there is a thriving "homebrew" community producing some awesome applications, games and patches to modify the behaviour of the Pre. Case in point - by default, the calendar application doesn't open in month view by default, nor does it show all-day appointments in that view. A few quick patches later, and I had customised it to my liking. These patches cover a huge range of tweaks, even including an on-screen keyboard.

The open Linux-based nature of this phone makes it a real hackers playground, yet despite all that geekiness it actually works brilliantly as a phone and communications device. I have all my contacts, Facebook friends, home IMAP mail, work Exchange mail (which works with EAS and remote wipe) and Google calendar syncing flawlessly. As an experiment, after the initial (short) learning curve, I swapped my SIM back out and into my old iPhone. I lasted a few hours before I'd gone back to the Pre in sheer frustration.

Not everything is quite so rosy, though. I have to say that the hardware build quality is a bit of a let-down. While it feels amazing in your hand (there's something about the form factor, dimensions and curves that make it feel comfortable) it does feel a bit.... flimsy. Replacing the back with the awesome Touchstone kit helps a great deal, as does adding a case and screen protector.

There's also the continued doom-and-gloom from the Tech pundits casting a cloud over Palm's future; but with news of a full App Market coming at the end of this month to UK users; an updated Facebook application; the PDK and Flash coming; and even QT being ported I'd say things are looking up.

Bye bye, Apple...