• No built in graphing (seriously, Dell - WTF?), but you can do it from the CLI - see here.
  
• Can't resize or change the I/O profile of a virtual disk once it's setup. This is a real pain, so make sure you set things up correctly the first time! You can however change the RAID level of a disk group once it's been created.
  
• You need a Windows or RHEL box to run the administration GUI on - I'm sure you can probably hack a way to get the CLI running under Debian, but I haven't tried. You're probably straight out of luck if you want to run it on anything else like Solaris.
• Can't mix SAS and SATA in the same enclosure. The controllers do support SATA as well as SAS, although SATA drives don't show up as options in the Dell pricing configuration thingy. Our account manager advised us that although technically you can mix SAS and SATA in the same enclosure, they'd experienced a higher than average number of disk failures in that configuration, due to the vibration patterns created by disks spinning at different rates (15K SAS and 7.2K SATA). If you need to mix the two types, your only real option is to attach a MD1000 array to the back (you can add up to two of these) and have each chassis filled with just one type of drive.
  

The hardware failover works nicely - the array is active/passive for each virtual disk, as both controllers are typically active, each handling separate virtual disks for load-balancing purposes. When a controller fails, the remaining "good" controller takes over the virtual disks or disk groups from the failed controller. Failback is pretty transparent - the GUI guides you through the steps, but I found that simply inserting a replacement HD/Controller/etc. just did the job automagically.

Multipath support under RHEL/CentOS with multipath-tools (dm-multipath) works fine with some tweaking - it uses the RDAC modules which lead to some oddness on CentOS 5.3. What tends to happen is that the first time device mapper picks up the paths, RDAC doesn't get a chance to initialise things properly (scsi_dh_rdac module isn't loaded) so you end up with all sorts of SCSI errors showing up in your logs. After flushing your paths (multipath -F) and restarting multipathd, things are OK. This is apparently fixed in RHEL 5.4, so should make it's way out to CentOS from there. I'm unsure what the status is on other distros, though.

It also works great with Citrix Xenserver, although you have to use MPP-RDAC instead of DM-Multipath due to performance issues with the latter.

My multipath.conf contains the following :

devices {
        device {
                vendor "DELL"
                product "MD3000i"
                product_blacklist "Universal Xport"
                path_grouping_policy group_by_prio
                getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
                path_checker rdac
                prio_callout "/sbin/mpath_prio_rdac /dev/%n"
                hardware_handler "1 rdac"
                failback immediate
        }
}

360026b90002ab6f40000056a4aa9e87b dm-12 DELL,MD3000i
[size=409G][features=0][hwhandler=1 rdac][rw]
_ round-robin 0 [prio=200][active]
 _ 21:0:0:1  sdi 8:128 [active][ready]
 _ 22:0:0:1  sdj 8:144 [active][ready]
_ round-robin 0 [prio=0][enabled]
 _ 20:0:0:1  sdg 8:96  [active][ghost]
 _ 23:0:0:1  sdh 8:112 [active][ghost] 

Update: It looks like the admin tool and SMcli are just shell script wrappers that run Java apps. I tried a quick'n'dirty hack of installing everything under RHEL, tarring up /opt/dell and /var/opt/SM and then transferring them over to a Debian Lenny host. All I had to change was the #!/bin/sh to #!/bin/bash at the top of the SMcli and SMclient wrappers, and they seem to work. I haven't put them through any serious testing though...

I was talking a few days ago, and the subject of password security came up. Now, we all know that we're supposed to pick a secure password, use at least 8 characters and never to pick a word from the dictionary. But then I was asked how long it would take to brute-force a password using a dictionary attack, and I had to admit I had no idea. I knew it would only be a matter of minutes, but wanted to give it a try.

So, For anyone who is interested, I knocked up a quick BASH script to compare a MD5 hashed password against the contents of /usr/share/dict/words, which on a Red Hat 5.3 system contains 479,623 words. The script is as follows :

#!/bin/bash
TARGET_HASH=$1
while read WORD; do
        WORD_HASH=$(echo $WORD | md5sum | awk '{print $1}')
        if [ "$WORD_HASH" == "$TARGET_HASH" ]; then
                echo "Found match!"
                echo "Password is : $WORD"
                exit
        fi
done < /usr/share/dict/words

Now, this was just a quick hack to satisfy my curiosity, and only something I threw together after a few seconds. Of particular relevance is the fact that it's a shell script, and uses a lot of forking to generate the MD5 hashes of the dictionary. If I wrote it in C, I'm sure it would be faster by an order of magnitude.

But anyway, on to the test - I created a MD5 phrase for it to crack, and timed it :

# time ./crack.sh 3a783fb2aa3a2318499f0a60d7ef6078
Found match!
Password is : hedgehog

real    8m43.432s
user    1m48.410s
sys     8m27.030s

Not bad - just under 9 minutes. Obviously, that'd take longer if I used a word starting with "x" or "z"!  I then realised it would be a lot faster if I generated a "compiled" version of the dictionary file with the MD5 hashes preprepared :

while read WORD; do echo "$WORD:$(echo $WORD | md5sum | awk '{print $1}')"; done < /usr/share/dict/words  > md5.txt

Obviously, I could then generate compiled dictionary files for each hashing algorithm I wanted to crack (assuming that they are non-Salted algorithms).  This took around 30 minutes, but now I don't have to generate the hashes again, all I need to do is check against the second column of the file for a match. It is also irrelevant whether the word lies near the start or end of the file, it now takes about the same time to find a match :

# time grep ac23b37db0039dda62896bb21f312755 md5.txt | cut -d':' -f1
aardvark

real    0m0.019s
user    0m0.008s
sys     0m0.011s

# time grep 981fe627ab4906b677ce9d3e6eff499f md5.txt | cut -d':' -f1
zoology

real    0m0.019s
user    0m0.006s
sys     0m0.014s

So there you have it. It was an interesting way to spend a few minutes, and I now have an answer whenever someone asks "how long would it take to crack a password based on a dictionary word": Assuming you have the compiled hash files, around 0.019 seconds.

Introduction

I've been using and evaluating Citrix XenServer now for a while, and felt I should really post a review. I haven't seen much detailed coverage of this product at the level I'm interested in, so what follows is my take on it from a Unix Sysadmin's perspective. There won't be any funky screenshots or graphics; instead, I tried to cover the sort of things I wanted to know about when I was looking at it as a candidate for our virtualization solution at work.  

After all, implementing a new hypervisor is a big step, and a decision that you'll likely be stuck with for a long time. If there's anything else you'd like to know, just post in the comments section and I'll do my best to answer.

As some background: I've been using the open source Xen hypervisor as a virtualization platform, alongside VMware for Windows hosts for a good few years now at work. Part of the reason for picking Xen was that it was the standard on the systems I inherited, and also it was free and well-supported on most Linux distributions at the time. To date, I have been using CentOS as a Dom0 - as it's a free "clone" of Red Hat Enterprise Linux, it follows the same support schedules (up to 2014 for RHEL/CentOS 5.x) and is supported by pretty much every hardware vendor out there. It also has the libvirt tools built into it, as well as up to date packages for storage infrastructure such as DRBD and open-iscsi. It's well supported, and even though it is a conservative "stable" distro, point releases occur regularly with back-ported drivers and user-land updates.

With some work, you can roll your own management tools and scripts, and end up with a very flexible solution. However, it lacks some management ease of use, particularly for other systems administrators who may not be totally comfortable in a Linux environment. We also wanted to standardise on one virtualization platform if possible, and this all coincided nicely with a planned upgrade/migration off the VMware stack.

XenServer therefore presents a very attractive proposition: A well known, widely tested and supported open source hypervisor, with a superior management stack. The basic product is free, although support and enterprise features are available for a price. The prices for the advanced features are very reasonable, all the more so when you compare against VMware's offerings. Also consider that the free product allows you to connect to a wide range of networked storage systems and includes live migration, something that the free ESXi doesn't offer.

All of what follows covers the freely downloadable XenServer 5.6; Both Dell and HP offer embedded versions for some of their servers, however running and managing these systems should be near enough identical apart from the installation steps.

Update : Just after writing this, the beta of "FP1" (an update to XenServer 5.6) was announced. Full details of what will be in this update are here in the release notes. It looks like there will be plenty of significant improvements across all areas (including MPP RDAC, scheduled backups, supported jumbo frames, on-line coalescing of snapshot space and various other things of particular interest to me). Bear in mind when reading this review, that many of the little issues I have with XenServer may well be resolved in the upcoming version, and other areas may be totally overhauled. As soon as the final version is released I'll post a full update...

Update 2 : FP1 is indeed a big improvement. I've been using it in production now for a few months and should have an update soon, covering the new features such as the distributed switch, self-service portal etc.

Click the "Continue reading" link for the full review.

We have recently started using Citrix Xenserver in production at work (fantastic product, see my review for more information) and needed a simple backup solution. Our VMs run from an iSCSI SAN and are backed up daily through various methods - e.g. Bacula for the Unix/Linux systems. However, we wanted the ability to quickly roll back to a previous VM snapshot, and get up and running quickly if our SAN failed for whatever reason. Our solution was to create a large shared NFS storage repository, and periodically snapshot VMs and copy the templates over to this SR. Doing this means that if the SAN fails, we can create a new VM quickly from this NFS store (using the Xenserver's local disks, or even the NFS SR itself as storage). Once up and running, we can bring VMs back up to date by restoring the latest backup to them.

In order to automate this, I wrote a quick script which I thought may prove useful to someone else, so decided to post it here : snapback.sh.

Update: This script is now being hosted at GitHub. This means you can check out the latest version from there, by doing :

git clone https://github.com/markround/XenServer-snapshot-backup.git

or accessing the raw script file at https://github.com/markround/XenServer-snapshot-backup/raw/master/snapback.sh

It is very simple, and although it may serve well as your only backup solution, it's really intended as an image-level compliment to your primary file-system based backup system such as Bacula, Amanda, Netbackup etc. It also has not had much testing, and I fully appreciate the scripting is pretty rudimentary and could do with some optimisation - there's no error checking, for instance. I kept it pretty verbose on purpose though, so you can get a good idea of exactly what it's doing at each step; it may be better to think of this as a template you can base your own scripts off!

Overview

The script creates a snapshot of a running VM on a configurable schedule, and then creates a template from this snapshot. It will copy all these backup templates over to a configurable storage repository, and then clean up any old backups according to a specified retention policy. These backups are full backups, so if you have a 10GB VM and keep 7 previous copies you will need a total of 80GB disk space on your backup VM. Non-running VMs, and those not configured (as detailed below) will be skipped.

Important: See http://support.citrix.com/article/CTX123400. After backing up each VM, you will end up with a new VDI, so you may need to manually coalesce your VDIs again to reclaim disk space.

Installation and usage

2 1 * * * root /usr/local/bin/snapback.sh > /var/log/snapback.log 2>&1

This will also record a log of it's actions to /var/log/snapback.log. You now need to edit the script and change the DEST_SR variable to the UUID of your backup storage repository. You can find this value by clicking on the SR in Xencenter; the UUID will be displayed as a value like "2c01dc26-f525-70d6-dedf-00baaec76645".

Lastly, you need to configure your backup and retention policy for your VMs. In Xencenter, right click your VM, and select "Properties". Click on "Custom Fields", and then "Edit Custom Fields". You should add two text fields :

• backup : Can be one of "daily", "weekly", or "monthly". If it is set to weekly, it will by default run on a Sunday, and if it set to monthly, it will run on the first Sunday of the month. This day can be changed at the top of the script - see the WEEKLY_ON and MONTHLY_ON variables.
• retain : How many previous backups (in addition to the currently running backup) to keep. So, setting this to a value of "2" would mean that after a backup has run, you would end up with 3 backups in total.

Adding a custom field

The script will look for these fields when it is run, and will skip any VM that doesn't have them set. You can also see them in the Xencenter summary and properties for the VM :

VM summary showing the custom fields

You can now either run the script manually, or wait until the cron job kicks off. It will produce a detailed log to the console (or log file if run through cron), and when it's finished, you'll see your template backup VMs listed in Xencenter, similar to this :

Backups listed in Xencenter

I've been looking for ages for a tool to parse the output from "iostat" on Linux, and graph it in Cacti. I found a few scripts and templates that did some of what I was looking for (disk I/O etc.), but nothing that gave me the full set of statistics such as queue length, utilisation, service time etc. I finally got round to writing my own set of templates and a data gathering script to provide this information, and it seems to work very well. So that others can benefit, I've posted the package archive and a brief description over on the Cacti forums (click Continue Reading for a download link to an updated version - the one on the Cacti forums has a bug so that it won't work with all versions of sysstat). Below are a couple of sample graphs to give you an idea of what it can do - there's also a few more samples posted in the Cacti forums thread :

Displaying iostat's kB/s data source.

Timing information from iostat (average wait and service time)

Installation is a simple matter of creating a cron job to gather iostat data, extending your snmpd.conf to call the included iostat.pl script, and then importing the templates. Full instructions are included in the README within the archive (click the Continue Reading link to see them), but if you have any comments, suggestions or problems please let me know!

While I've been preparing an update to the 2.2.6 Blastwave packages of nessus, Teneable just released their new 3.0 package - offering a whole host of enhancements including a very funky looking RSS feed for plugin updating, and major performance improvements to name just two. Except this time, I'm not doing my usual w00t-dance, and I won't be packaging it, or even running it, for that matter.

The reason being that Tenable chose to make this version closed source. Now, that's all well and good and they're obviously well within their rights to do so. But as with so many closed source products (Zend, I'm looking at you), it's released for Linux/x86 first (although FreeBSD packages are also available), and everything else takes a back seat until some unspecified time in the future. It it is this ramification of the license change that I find most infuriating. It wouldn't perhaps be so bad if Tenable could guarantee that all platforms would have binaries available for them - but this means they're leaving a large section of their userbase out in the cold. And woe betide you if you're running anything they consider really obscure or not worth supporting. Even something like Solaris/x86 is frequently ignored, and I can't begin to imagine what people running something like NetBSD on Alpha must have to contend with...

With the open source model (take MySQL as an example), you can get the source code, and can be pretty sure that you can build it on pretty much any platform you want. MySQL runs on most platforms - from Unix to Windows, OpenVMS to Linux/S390. If it doesn't run on your chosen platform, or the developers don't have access to the relevant development environment, you can hack it yourself and contribute patches back to the community. Once the source is closed, that option is gone forever. You're then totally dependant on the developer to continue supporting your platform. You also, by extension, have to hope they never go out of business, especially if their product incorporates some sort of time-locked licensing! If they wake up one morning and decide that it's no longer economically viable to continue building their product for your platform, you're screwed.

Never mind that you may have built your entire infrastructure around a certain technology, and it's not economically viable for you to jump ship to whatever the flavour of the month is; if you want to continue running closed source product X, you have to dance to the beat of the developers' drum. It's for precisely this reason that I was so glad to see Sun open up Solaris (SPARC has been an open architecture for a long while now, so that's never been an issue). Yeah, the community Sun has built up around it is fantastic, as is the ability to get a sneak preview of all the latest features and browse the code yourself. But it now means that whatever happens to Sun (although I seriously doubt they're going anywhere anytime soon), our investment is secure. So, I'm sorry that Tenable felt they had no other option than to close the source of Nessus - but I for one look forward to the continued development of the forked GPL version. As soon as there is code released, there will be Blastwave packages...