Overview

Installation and usage

2 1 * * * root /usr/local/bin/snapback.sh > /var/log/snapback.log 2>&1

This will also record a log of it's actions to /var/log/snapback.log. You now need to edit the script and change the DEST_SR variable to the UUID of your backup storage repository. You can find this value by clicking on the SR in Xencenter; the UUID will be displayed as a value like "2c01dc26-f525-70d6-dedf-00baaec76645".

Lastly, you need to configure your backup and retention policy for your VMs. In Xencenter, right click your VM, and select "Properties". Click on "Custom Fields", and then "Edit Custom Fields". You should add two text fields :

• backup : Can be one of "daily", "weekly", or "monthly". If it is set to weekly, it will be default run on a Sunday, and if it set to monthly, it will run on the first Sunday of the month. This day can be changed at the top of the script - see the WEEKLY_ON and MONTHLY_ON variables.
• retain : How many previous backups (in addition to the currently running backup) to keep. So, setting this to a value of "2" would mean that after a backup has run, you would end up with 3 backups in total.

Adding a custom field

The script will look for these fields when it is run, and will skip any VM that doesn't have them set. You can also see them in the Xencenter summary and properties for the VM :

VM summary showing the custom fields

You can now either run the script manually, or wait until the cron job kicks off. It will produce a detailed log to the console (or log file if run through cron), and when it's finished, you'll see your template backup VMs listed in Xencenter, similar to this :

Backups listed in Xencenter

I've been looking for ages for a tool to parse the output from "iostat" on Linux, and graph it in Cacti. I found a few scripts and templates that did some of what I was looking for (disk I/O etc.), but nothing that gave me the full set of statistics such as queue length, utilisation, service time etc. I finally got round to writing my own set of templates and a data gathering script to provide this information, and it seems to work very well. So that others can benefit, I've posted the package archive and a brief description over on the Cacti forums (click Continue Reading for a download link to an updated version - the one on the Cacti forums has a bug so that it won't work with all versions of sysstat). Below are a couple of sample graphs to give you an idea of what it can do - there's also a few more samples posted in the Cacti forums thread :

Displaying iostat's kB/s data source.

Timing information from iostat (average wait and service time)

Installation is a simple matter of creating a cron job to gather iostat data, extending your snmpd.conf to call the included iostat.pl script, and then importing the templates. Full instructions are included in the README within the archive (click the Continue Reading link to see them), but if you have any comments, suggestions or problems please let me know!

Well, that's that, then. Solaris as we knew it is pretty much dead. I've suspected for a while now that Oracle's intentions regarding Solaris were not what the community, or us "old-school" Solaris sysadmins wanted or had hoped for.

In the last few months, Oracle have completely alienated and scared off the community around OpenSolaris, killed any lines of communication by clamping down on employee blogs and have ignored open letters from highly influential and important community members begging for any kind of information. They’ve forbidden Sun/Oracle employees from heading up the Solaris user groups and booted the meetings out of their buildings; turned Solaris 10 into a 90-day trial, and pushed back the 2010.x release of OpenSolaris with no word as to it’s planned release date, or even if it is being continued as a product. And now, in a final act of desperation, the OGB has essentially threatened to "shoot itself in the head".

Warning: Rant ahead. I spent the best part of yesterday fighting to get a commercial bugtracking solution installed. It will remain nameless for now - although if you know the market, it shouldn't be too hard to guess which one it is. The installation routine in question is without doubt one of the rudest I've ever encountered, stopping just short of the current number one spot held by Cisco's VPN install on Mac OS X, due to the latters' spectacular star turn of defecating all over your /opt directory.

Firstly, it's a PHP product (although it actually started life as an ASP product on Windows - a quick peek at the source code reveals it has been run through an asp-to-php converter in order to provide a rough-n-ready Unix solution. Ugh!). It blows away your current PEAR install, and then tries to download four additional packages. Strike one - I don't particularly take too kindly to software changing parts of my filesystem without a warning, particularly when it could just ask you to upgrade yourself. It also doesn't take into account any webservers which may not have a direct path to the outside world - if you're locking things down, blocking processes from initiating outbound connections to the internet is a very good way of foiling many script-kiddy attacks.

After I allowed it to talk to the outside world, it mysteriously failed halfway through and didn't manage to download those additional modules. I then tried to download those modules manually and install them myself, only to run into another little present the installer had left for me - as a nice treat, it also renames your "pear" command to "pear_old" before it quits, which quite obviously leads to various things breaking all over the shop. Strike two - hey guys, if your installer is going to crap all over my system, it better clean up after itself and leave things in the state it found them. I have no idea why it chose to do this, but after I renamed the command back things seemed to work fine.

At least, until I go to the next part of the install. It wants some truly brain-damaged settings configured in php.ini and my.cnf. 50Mb post_max_size ? 600 seconds max_input_time ? Sure, no problem! What's the big deal with DoS attacks, anyway ? Suspiciously, it also wants MySQL to accept maximum packets of 50Mb. Again, this makes me feel uneasy, not just because there is no way on Earth it should normally be set that high; I've got the impression that the software is going to store file attachments in a database, which just makes me start foaming at the mouth just thinking about it. So, any thoughts I had about running this on a shared server went right out the window. Strike three - I'm going to have to try and isolate it's bad design decisions as much as possible from the rest of the applications on the server - including bunging it, a separate PHP install and MySQL server into it's own little ghetto.

But then fun doesn't stop there! It also includes a shared library extension that must be loaded into PHP. One problem - it's been linked against libraries that aren't even part of the stock Solaris install. Yup :

warning: ldd: stupid_library.so : is not executable
libstdc++.so.2.10.0 => (file not found)
libm.so.1 => /usr/lib/libm.so.1
libc.so.1 => /usr/lib/libc.so.1
libdl.so.1 => /usr/lib/libdl.so.1
/usr/platform/SUNW,UltraAX-i2/lib/libc_psr.so.1

Oh, JOY. After hunting this down, it seems as if it was linked against libstdc++ library in the SFWgcmn package, which is part of the optional freeware CD for Solaris. I downloaded this, installed it and tried again. No joy. The stupid, brain-dead library hadn't been compiled properly with correct -L and -R flags passed to the linker - it had no idea how to find this library. I've run out of strikes now - by this point, I'd reached the descision that this installer needed to be dragged out round the back of the barn and shot in the head.

I then had to implement a LD_LIBRARY_PATH wrapper around my whole Apache process, and anything else that needed to run with this extension. This is is A Bad Idea(tm), but the alternative of dinking with CRLE, and thereby polluting the rest of my system with /opt/sfw/lib seemed far worse. So after all that, was it worth it ? Well, it's a little difficult to say, given how dog-slow the product is. This is on a server that is running around 90-97% idle for most of the time, but for some reason pages take a good 4-5 seconds to load in a web browser. When they say you need a PHP opcode cache, they aren't joking. This also isn't without it's problems, as the available PHP opcode caches out there are either seemingly abandoned, forks that are starting over from scratch, commercial and wildly expensive, or just crash a lot on Solaris. Actually, pretty much all of them do that last one - but that's a rant for another day. Installing APC helps a lot, but is still a little unstable. Time will tell if this has been worth all the pain, but first impressions and all that...

Thanks to the awesome work of Boogie Shafer, there is now a FreeBSD port of my iostat scripts and templates for Cacti. I have included the modified tarball that was sent to me, this is inside the archive as "cacti-iostat-1.x-boogie_freebsd_linux_changes.tar.gz".

FreeBSD users should unpack this archive and follow the instructions inside. I have not had time to go through and merge these changes into one unified distribution yet, but as people were asking for the FreeBSD port, here it is! The next release of these scripts should see the FreeBSD scripts and templates etc. merged in, much the same as the Solaris modifications by Marwan Shaher and Eric Schoeller.

Follow the link to the original post to find the download link.

• No built in graphing (seriously, Dell - WTF?), but you can do it from the CLI - see here.
  
• Can't resize or change the I/O profile of a virtual disk once it's setup. This is a real pain, so make sure you set things up correctly the first time! You can however change the RAID level of a disk group once it's been created.
  
• You need a Windows or RHEL box to run the administration GUI on - I'm sure you can probably hack a way to get the CLI running under Debian, but I haven't tried. You're probably straight out of luck if you want to run it on anything else like Solaris.
• Can't mix SAS and SATA in the same enclosure. The controllers do support SATA as well as SAS, although SATA drives don't show up as options in the Dell pricing configuration thingy. Our account manager advised us that although technically you can mix SAS and SATA in the same enclosure, they'd experienced a higher than average number of disk failures in that configuration, due to the vibration patterns created by disks spinning at different rates (15K SAS and 7.2K SATA). If you need to mix the two types, your only real option is to attach a MD1000 array to the back (you can add up to two of these) and have each chassis filled with just one type of drive.
  

devices {
        device {
                vendor "DELL"
                product "MD3000i"
                product_blacklist "Universal Xport"
                path_grouping_policy group_by_prio
                getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
                path_checker rdac
                prio_callout "/sbin/mpath_prio_rdac /dev/%n"
                hardware_handler "1 rdac"
                failback immediate
        }
}

360026b90002ab6f40000056a4aa9e87b dm-12 DELL,MD3000i
[size=409G][features=0][hwhandler=1 rdac][rw]
_ round-robin 0 [prio=200][active]
 _ 21:0:0:1  sdi 8:128 [active][ready]
 _ 22:0:0:1  sdj 8:144 [active][ready]
_ round-robin 0 [prio=0][enabled]
 _ 20:0:0:1  sdg 8:96  [active][ghost]
 _ 23:0:0:1  sdh 8:112 [active][ghost] 

Update: It looks like the admin tool and SMcli are just shell script wrappers that run Java apps. I tried a quick'n'dirty hack of installing everything under RHEL, tarring up /opt/dell and /var/opt/SM and then transferring them over to a Debian Lenny host. All I had to change was the #!/bin/sh to #!/bin/bash at the top of the SMcli and SMclient wrappers, and they seem to work. I haven't put them through any serious testing though...

Part 1 - Overview, network layout and DRBD installation
Part 2 - DRBD and LVM
Part 3 - Heartbeat and automated failover
Part 4 - iSCSI and IP failover
Part 5 - Multipathing and client configuration
Part 6 - Anything left over!

In this part of the series, we'll configure an iSCSI client ("initiator"), connect it to the storage servers and set up multipathing. Note : Since Debian Lenny has been released since this series of articles started, that's the version we'll use for the client.

If you refer back to part one to refresh your memory of the network layout, you can see that the storage client ("badger" in that diagram) should have 3 network interfaces :

• eth0 : 172.16.7.x for the management interface, this is what you'll use to SSH into it.

And two storage interfaces. As the storage servers ("targets") are using 192.168.x.1 and 2, I've given this client the following addresses :

• eth1: 192.168.1.10
• eth2: 192.168.2.10

Starting at .10 on each range keeps things clear - I've found it can help to have a policy of servers being in a range of, say, 1 to 10, and clients being above this. Before we continue, make sure that these interfaces are configured, and you can ping the storage server over both interfaces, e.g. try pinging 192.168.1.1 and 192.168.2.1.

Assuming the underlying networking is configured and working, the first thing we need to do is install open-iscsi (which is the "initiator" - the iSCSI client). This is done by a simple :

# aptitude install open-iscsi

You should see the package get installed, and the service started :

Setting up open-iscsi (2.0.870~rc3-0.4) ...
Starting iSCSI initiator service: iscsid.
Setting up iSCSI targets:
iscsiadm: No records found!

At this point, we have all we need to start setting up some connections.