Archives
Categories
Quicksearch
Syndicate This Blog
Xenserver snapshot and template based backup script
In order to automate this, I wrote a quick script which I thought may prove useful to someone else, so decided to post it here : snapback.sh.
It is very simple, and although it may serve well as your only backup solution, it's really intended as an image-level compliment to your primary file-system based backup system such as Bacula, Amanda, Netbackup etc. It also has not had much testing, and I fully appreciate the scripting is pretty rudimentary and could do with some optimisation - there's no error checking, for instance. I kept it pretty verbose on purpose though, so you can get a good idea of exactly what it's doing at each step; it may be better to think of this as a template you can base your own scripts off!
Overview
The script creates a snapshot of a running VM on a configurable schedule, and then creates a template from this snapshot. It will copy all these backup templates over to a configurable storage repository, and then clean up any old backups according to a specified retention policy. These backups are full backups, so if you have a 10GB VM and keep 7 previous copies you will need a total of 80GB disk space on your backup VM. Non-running VMs, and those not configured (as detailed below) will be skipped.Installation and usage
First, copy the script to your Xenserver pool master, and make it executable. A good location for this is /usr/local/bin/snapback.sh.Next, create a cron entry for the script - to make it run daily just after 1AM, you'd create /etc/cron.d/backup with the following contents :
2 1 * * * root /usr/local/bin/snapback.sh > /var/log/snapback.log 2>&1
This will also record a log of it's actions to /var/log/snapback.log. You now need to edit the script and change the DEST_SR variable to the UUID of your backup storage repository. You can find this value by clicking on the SR in Xencenter; the UUID will be displayed as a value like "2c01dc26-f525-70d6-dedf-00baaec76645".
Lastly, you need to configure your backup and retention policy for your VMs. In Xencenter, right click your VM, and select "Properties". Click on "Custom Fields", and then "Edit Custom Fields". You should add two text fields :
- backup : Can be one of "daily", "weekly", or "monthly". If it is set to weekly, it will be default run on a Sunday, and if it set to monthly, it will run on the first Sunday of the month. This day can be changed at the top of the script - see the WEEKLY_ON and MONTHLY_ON variables.
- retain : How many previous backups (in addition to the currently running backup) to keep. So, setting this to a value of "2" would mean that after a backup has run, you would end up with 3 backups in total.
The script will look for these fields when it is run, and will skip any VM that doesn't have them set. You can also see them in the Xencenter summary and properties for the VM :
You can now either run the script manually, or wait until the cron job kicks off. It will produce a detailed log to the console (or log file if run through cron), and when it's finished, you'll see your template backup VMs listed in Xencenter, similar to this :
If you find that this clutters up the Xencenter view a little, you can always hide them (View->Server View->Custom Templates). To restore a VM from a backup, just right click, and choose "New template from backup". Anyway, I hope this helps someone else!
The setting sun
Well, that's that, then. Solaris as we knew it is pretty much dead. I've suspected for a while now that Oracle's intentions regarding Solaris were not what the community, or us "old-school" Solaris sysadmins wanted or had hoped for.
In the last few months, Oracle have completely alienated and scared off the community around OpenSolaris, killed any lines of communication by clamping down on employee blogs and have ignored open letters from highly influential and important community members begging for any kind of information. They’ve forbidden Sun/Oracle employees from heading up the Solaris user groups and booted the meetings out of their buildings; turned Solaris 10 into a 90-day trial, and pushed back the 2010.x release of OpenSolaris with no word as to it’s planned release date, or even if it is being continued as a product. And now, in a final act of desperation, the OGB has essentially threatened to "shoot itself in the head".
Leaving the iPhone for the Palm Pre
While I've been a long-time Apple user and fan of their hardware, I have to say that I've become a little disillusioned of late. Mostly, this has been down to the iPhone: While it's undeniably been a game-changer and raised everyone's expectations of what a mobile phone could do, there's just been an uncomfortable, claustrophobic feel around the whole thing. From the start, there were ridiculous NDAs preventing developers from even discussing programming the device; an insane culture of overbearing secrecy and control; not to mention Apple pulling applications from the app store or rejecting them seemingly on a whim, censoring content and being generally hacker-unfriendly.The final straw however was the news that they are suing HTC over patent violation. Software patents are a terrible thing for this industry, and it appears that instead of competing on their own merits, Apple has chosen to resort to being a patent troll - unsurprising perhaps, given their past behaviour.
All this neatly coincided with my contract with O2 coming up for renewal, so I could get a new phone. I'd done some research and had narrowed down my iPhone alternatives to just 2 options - Palm Pre, or some Android-powered device. Android would have required me to leave O2, so that really just left the Palm Pre (and Windows Mobile or a Blackberry were right out). All the reviews I'd read were pretty good for the Pre but I wanted to try WebOS out first. Fortunately, this was easy as the Palm SDK includes an emulator. This got me hooked - not only is the emulator based on VirtualBox (which is massively cool), but all the tools such as "WebOS Doctor" are available for Windows, MacOS AND Linux. I even installed it with no problems on the latest Ubuntu Lucid Alpha.
Playing with WebOS blew me away. I won't go into too much detail, as all the review sites have covered the WebOS experience in far greater depth. I should point out though that since most of the reviews were written, there have been numerous software updates which have brought a great deal of new features and improvements - the last 1.4 update for instance added video recording support! I will however simply say this : It actually makes the iPhone feel clunky in comparison. The multitasking card system combined with the finger gestures is amazingly fluid and intuitive. Little touches throughout such as the notification system, "synergy" contacts system and email handling put this streets ahead of the iPhone.
I was also pleased to see that there is a thriving "homebrew" community producing some awesome applications, games and patches to modify the behaviour of the Pre. Case in point - by default, the calendar application doesn't open in month view by default, nor does it show all-day appointments in that view. A few quick patches later, and I had customised it to my liking. These patches cover a huge range of tweaks, even including an on-screen keyboard.
The open Linux-based nature of this phone makes it a real hackers playground, yet despite all that geekiness it actually works brilliantly as a phone and communications device. I have all my contacts, Facebook friends, home IMAP mail, work Exchange mail (which works with EAS and remote wipe) and Google calendar syncing flawlessly. As an experiment, after the initial (short) learning curve, I swapped my SIM back out and into my old iPhone. I lasted a few hours before I'd gone back to the Pre in sheer frustration.
Not everything is quite so rosy, though. I have to say that the hardware build quality is a bit of a let-down. While it feels amazing in your hand (there's something about the form factor, dimensions and curves that make it feel comfortable) it does feel a bit.... flimsy. Replacing the back with the awesome Touchstone kit helps a great deal, as does adding a case and screen protector.
There's also the continued doom-and-gloom from the Tech pundits casting a cloud over Palm's future; but with news of a full App Market coming at the end of this month to UK users; an updated Facebook application; the PDK and Flash coming; and even QT being ported I'd say things are looking up.
Bye bye, Apple...
Centreon review
It was always an interesting question to ask, because it gave you an insight into the kind of sysadmin tasks someone had been doing before, and it also served as a nice, relaxed "ice breaker" type question. For my money, aside from some tools like rsync and screen which I couldn't live without, a decent monitoring package would have to be top of my priorities. There are a bunch out there: some of them free; some of them commercial, but the one that would make it on to my USB stick would have to be Nagios.
It's open source, extremely well documented and widely implemented, and there are a ton of useful add-ons and plugins available for it. The only draw backs I can find with it are it's ugly web interface, the complexity involved in setting up a new system for monitoring, and the disjoint between availability and performance monitoring. If you have money to throw at a problem, then software like Uptime or Hyperic neatly deal with all of these issues, but they can be quite pricey if you have a large number of systems to manage and a tight budget.
So, you can imagine my excitement when I first discovered Centreon. It's essentially a monitoring platform that uses Nagios at it's core. You could think of it as a fancy frontend to "stock" Nagios, but it's so much more than that: besides the attractive interface, it also bridges the gap between availability and performance monitoring, and makes Nagios administration a snap. Due to the reliance on Nagios though, I'd go so far as to say that before you experiment with Centreon, you really should have set up "stock" Nagios, and be familiar with the plugin architecture, NRPE and how alerts / escalations are managed. Ideally, you should have a stock Nagios installation you can use to duplicate on Centreon/Nagios.
Continue reading "Centreon review"
Cacti iostat scripts now support FreeBSD
Thanks to the awesome work of Boogie Shafer, there is now a FreeBSD port of my iostat scripts and templates for Cacti. I have included the modified tarball that was sent to me, this is inside the archive as "cacti-iostat-1.x-boogie_freebsd_linux_changes.tar.gz".
FreeBSD users should unpack this archive and follow the instructions inside. I have not had time to go through and merge these changes into one unified distribution yet, but as people were asking for the FreeBSD port, here it is! The next release of these scripts should see the FreeBSD scripts and templates etc. merged in, much the same as the Solaris modifications by Marwan Shaher and Eric Schoeller.
Follow the link to the original post to find the download link.
Dell MD3000i
We've got it configured with dual controllers, 8x300Gb and 7x146GB 15k SAS drives. Throughput is around GigE wire speed - 110MB/s for both reads and writes. I'm also seeing a respectable IOPS figure depending on workloads: During an iozone run, I could see it sustaining around 1.5k IOPS.
True, the management features fall a little short when compared to the usual Sun and HP storage kit I'm used to, but it does the job. My main gripes are :
- No built in graphing (seriously, Dell - WTF?), but you can do it from the CLI - see here.
- Can't resize or change the I/O profile of a virtual disk once
it's setup. This is a real pain, so make sure you set things up correctly
the first time! You can however change the RAID level of a disk group
once it's been created.
- You need a Windows or RHEL box to run the administration GUI on - I'm sure you can probably hack a way to get the CLI running under Debian, but I haven't tried. You're probably straight out of luck if you want to run it on anything else like Solaris.
- Can't mix SAS and SATA in the same enclosure. The controllers
do support SATA as well as SAS, although SATA drives don't show up as
options in the Dell pricing configuration thingy. Our account manager
advised us that although technically you can mix SAS and SATA in the
same enclosure, they'd experienced a higher than average number of disk
failures in that configuration, due to the vibration patterns created
by disks spinning at different rates (15K SAS and 7.2K SATA). If you
need to mix the two types, your only real option is to attach a MD1000
array to the back (you can add up to two of these) and have each
chassis filled with just one type of drive.
Multipath support under RHEL/CentOS with multipath-tools (dm-multipath) works fine with some tweaking - it uses the RDAC modules which lead to some oddness on CentOS 5.3. What tends to happen is that the first time device mapper picks up the paths, RDAC doesn't get a chance to initialise things properly (scsi_dh_rdac module isn't loaded) so you end up with all sorts of SCSI errors showing up in your logs. After flushing your paths (multipath -F) and restarting multipathd, things are OK. This is apparently fixed in RHEL 5.4, so should make it's way out to CentOS from there. I'm unsure what the status is on other distros, though.
My multipath.conf contains the following :
devices {
device {
vendor "DELL"
product "MD3000i"
product_blacklist "Universal Xport"
path_grouping_policy group_by_prio
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
path_checker rdac
prio_callout "/sbin/mpath_prio_rdac /dev/%n"
hardware_handler "1 rdac"
failback immediate
}
}
360026b90002ab6f40000056a4aa9e87b dm-12 DELL,MD3000i [size=409G][features=0][hwhandler=1 rdac][rw] _ round-robin 0 [prio=200][active] _ 21:0:0:1 sdi 8:128 [active][ready] _ 22:0:0:1 sdj 8:144 [active][ready] _ round-robin 0 [prio=0][enabled] _ 20:0:0:1 sdg 8:96 [active][ghost] _ 23:0:0:1 sdh 8:112 [active][ghost]
Update: It looks like the admin tool and SMcli are just shell script wrappers that run Java apps. I tried a quick'n'dirty hack of installing everything under RHEL, tarring up /opt/dell and /var/opt/SM and then transferring them over to a Debian Lenny host. All I had to change was the #!/bin/sh to #!/bin/bash at the top of the SMcli and SMclient wrappers, and they seem to work. I haven't put them through any serious testing though...
Building a redundant iSCSI and NFS cluster with Debian - Part 5
Part 1 - Overview, network layout and DRBD installation
Part 2 - DRBD and LVM
Part 3 - Heartbeat and automated failover
Part 4 - iSCSI and IP failover
Part 5 - Multipathing and client configuration
Part 6 - Anything left over!
In this part of the series, we'll configure an iSCSI client ("initiator"), connect it to the storage servers and set up multipathing. Note : Since Debian Lenny has been released since this series of articles started, that's the version we'll use for the client.
If you refer back to part one to refresh your memory of the network layout, you can see that the storage client ("badger" in that diagram) should have 3 network interfaces :
- eth0 : 172.16.7.x for the management interface, this is what you'll use to SSH into it.
And two storage interfaces. As the storage servers ("targets") are using 192.168.x.1 and 2, I've given this client the following addresses :
- eth1: 192.168.1.10
- eth2: 192.168.2.10
Starting at .10 on each range keeps things clear - I've found it can help to have a policy of servers being in a range of, say, 1 to 10, and clients being above this. Before we continue, make sure that these interfaces are configured, and you can ping the storage server over both interfaces, e.g. try pinging 192.168.1.1 and 192.168.2.1.
Assuming the underlying networking is configured and working, the first thing we need to do is install open-iscsi (which is the "initiator" - the iSCSI client). This is done by a simple :
# aptitude install open-iscsi
You should see the package get installed, and the service started :
Setting up open-iscsi (2.0.870~rc3-0.4) ... Starting iSCSI initiator service: iscsid. Setting up iSCSI targets: iscsiadm: No records found!
At this point, we have all we need to start setting up some connections.
Continue reading "Building a redundant iSCSI and NFS cluster with Debian - Part 5"
Updated Cacti iostat package now supports Solaris
Just a quick update to my Cacti iostat monitoring scripts and templates - thanks to the work of Marwan Shaher and Eric Schoeller, the package now supports Solaris! The updated package is available here : cacti-iostat-1.4.tar.gz.
I have also updated the original blog post with the new package.
Cracking dictionary passwords
I was talking with my wife a few days ago, and the subject of password security came up. Now, we all know that we're supposed to pick a secure password, use at least 8 characters and never to pick a word from the dictionary. But then she asked how long it would take to brute-force a password using a dictionary attack, and I had to admit I had no idea. I knew it would only be a matter of minutes, but wanted to give it a try.
So, For anyone who is interested, I knocked up a quick BASH script to compare a MD5 hashed password against the contents of /usr/share/dict/words, which on a Red Hat 5.3 system contains 479,623 words. The script is as follows :
#!/bin/bash
TARGET_HASH=$1
while read WORD; do
WORD_HASH=$(echo $WORD | md5sum | awk '{print $1}')
if [ "$WORD_HASH" == "$TARGET_HASH" ]; then
echo "Found match!"
echo "Password is : $WORD"
exit
fi
done < /usr/share/dict/words
Now, this was just a quick hack to satisfy my curiosity, and only something I threw together after a few seconds. Of particular relevance is the fact that it's a shell script, and uses a lot of forking to generate the MD5 hashes of the dictionary. If I wrote it in C, I'm sure it would be faster by an order of magnitude.
But anyway, on to the test - I created a MD5 phrase for it to crack, and timed it :
# time ./crack.sh 3a783fb2aa3a2318499f0a60d7ef6078
Found match!
Password is : hedgehog
real 8m43.432s
user 1m48.410s
sys 8m27.030s
Not bad - just under 9 minutes. Obviously, that'd take longer if I used a word starting with "x" or "z"! I then realised it would be a lot faster if I generated a "compiled" version of the dictionary file with the MD5 hashes preprepared :
while read WORD; do echo "$WORD:$(echo $WORD | md5sum | awk '{print $1}')"; done < /usr/share/dict/words > md5.txt
Obviously, I could then generate compiled dictionary files for each hashing algorithm I wanted to crack (assuming that they are non-Salted algorithms). This took around 30 minutes, but now I don't have to generate the hashes again, all I need to do is check against the second column of the file for a match. It is also irrelevant whether the word lies near the start or end of the file, it now takes about the same time to find a match :
# time grep ac23b37db0039dda62896bb21f312755 md5.txt | cut -d':' -f1
aardvark
real 0m0.019s
user 0m0.008s
sys 0m0.011s
# time grep 981fe627ab4906b677ce9d3e6eff499f md5.txt | cut -d':' -f1
zoology
real 0m0.019s
user 0m0.006s
sys 0m0.014s
So there you have it. It was an interesting way to spend a few minutes, and I now have an answer whenever someone asks "how long would it take to crack a password based on a dictionary word": Assuming you have the compiled hash files, around 0.019 seconds.
OpenVPN on Windows XP and Vista
Just a quick post this time, as I thought this may help others in the same situation I found myself in recently. At work, we've been using OpenVPN which works a treat with Unix clients; Windows clients (Vista in particular) were more problematic, though.
None of our regular users have admin privileges (for obvious reasons), but this caused problems with the routing setup: users could use the GUI tool, but could not create the necessary routes required to direct traffic over the VPN. We experimented for a while with setting up persistent routes, but this didn't work for multiple users. I'd read all kinds of posts about running the executables as an Administrator, disabling Vista UAC, registry tweaks and other voodoo - either they didn't work, or they were unacceptable in our environment.
I then hit upon a simple workaround that also seems to work on Windows XP: Just add the user to the "Network Configuration Operators" group:
Administrative Tools -> Computer Management -> Local Users and Groups -> Groups -> Network Configuration Operators
Now, everything works right out of the box on Vista SP1 with the 2.1RC builds of OpenVPN (OpenVPN 2.1_rc15 was the version we tested). You have to install this as an Administrator, and you do have to be happy with giving your VPN users slightly elevated privileges - but at least it stops way short of having to give them administrator rights.
For reference, here's the client config file as well :
client
script-security 3 system
dev tun
proto udp
remote <openvpn server address> 1194
nobind
persist-key
persist-tun
ca ca.crt
cert <user.name>.crt
key <user.name>.key
cipher BF-CBC
comp-lzo
verb 3
mute 20
route-method exe
route-delay 2
Building a redundant iSCSI and NFS cluster with Debian - Part 4
This is part 4 of a series on building a redundant iSCSI and NFS SAN with Debian.
Part 1 - Overview, network layout and DRBD installation
Part 2 - DRBD and LVM
Part 3 - Heartbeat and automated failover
Part 4 - iSCSI and IP failover
Part 5 - Multipathing and client configuration
Part 6 - Anything left over!
In this part, we'll configure Heartbeat to manage IP address failover on the two storage interfaces. We'll also install and configure an iSCSI target to provide block-level storage to clients.
Continue reading "Building a redundant iSCSI and NFS cluster with Debian - Part 4"
Building a redundant iSCSI and NFS cluster with Debian - Part 3
This is part 3 of a series on building a redundant iSCSI and NFS SAN with Debian.
Part 1 - Overview, network layout and DRBD installation
Part 2 - DRBD and LVM
Part 3 - Heartbeat and automated failover
Part 4 - iSCSI and IP failover
Part 5 - Multipathing and client configuration
Part 6 - Anything left over!
Introduction
In the last two guides, we set up a DRBD resource and LVM volume group which we could manually migrate between the two cluster nodes. In this guide, we'll set up the Heartbeat cluster software to handle automatic migration of services between the two nodes in our cluster ("failover").
The version of Heartbeat included in Debian Etch is 1.x. It is a very simple system, and is limited to two node clusters, making it ideal for something simple such as failover for services between two nodes. The current 2.x branch is a lot more complicated, and has a new XML configuration format, although it can still be used with the original 1.x format files. Although it adds many useful features, it's overkill for our needs at the moment - plus, sticking to 1.x avoids the need to install software not included in the current stable distribution.
Linux, Solaris and FreeBSD iostat monitoring with Cacti
I've been looking for ages for a tool to parse the output from "iostat" on Linux, and graph it in Cacti. I found a few scripts and templates that did some of what I was looking for (disk I/O etc.), but nothing that gave me the full set of statistics such as queue length, utilisation, service time etc. I finally got round to writing my own set of templates and a data gathering script to provide this information, and it seems to work very well. So that others can benefit, I've posted the package archive and a brief description over on the Cacti forums (click Continue Reading for a download link to an updated version - the one on the Cacti forums has a bug so that it won't work with all versions of sysstat). Below are a couple of sample graphs to give you an idea of what it can do - there's also a few more samples posted in the Cacti forums thread :
Installation is a simple matter of creating a cron job to gather iostat data, extending your snmpd.conf to call the included iostat.pl script, and then importing the templates. Full instructions are included in the README within the archive (click the Continue Reading link to see them), but if you have any comments, suggestions or problems please let me know!
Continue reading "Linux, Solaris and FreeBSD iostat monitoring with Cacti"
Blastwave is dead
Blastwave is a registered trademark of Blastwave.org Inc. in the
United States and Canada. All assets of Blastwave.org Inc. are frozen
until further notice. All Solaris(tm) related open source software
work and services are cancelled. All websites, documents and binary
software packages that bear the mark Blastwave or Blastwave(tm) are no
longer available until further notice.
At the same time, mailing lists, shell logins and other services seem to have been shutdown and/or removed from DNS. None of this came with any warning or notification to the maintainers, and I still don't know what's going on. I can't access any of the build servers, so it's fairly safe to assume that my build scripts, packages, documentation, and everything else I've been working on for the Solaris community over the last 5 years is gone also. As if that wasn't enough, there are also reports that someone has been attempting to sabotage various mirror sites. I don't know how to take that - but frankly, right now, I don't care. I'm out. I've had it with the political fighting and drama. Many maintainers had already left following the last spat - I simply don't have the will to get involved in it any more, the damage has already been done. If anyone is still using my Blastwave packages (PostgreSQL, Nessus, PHP4, and some others) I recommend you switch to something else, like Sun's own CoolStack or OpenSolaris.
There's plenty more I could say, but at this point I think it's perhaps better to simply leave it. It's a sad day for me: seeing years of work towards something that I believed in, and helped a great many people, all go to ruin. It's even sadder for the Solaris community as a whole; this was a true grass-roots organisation - made up from like-minded Solaris users, admins, programmers and fans - who gave up countless hours of their own time to help others. I think the least we deserve is an explanation, but somehow I don't think one at this stage would make any difference anyway.
Update : People have been mailing me to say the main page is back up - true, but it's a case of "the lights are on, but no one's home". Check the thread in comp.unix.solaris.
Even ignoring the OpenSolaris project, It's not at all rosy in "real" Solaris land, either. Requests for information and clarification are going unanswered, and I know of several managers who have had hardware quotes and support tickets ignored - there's a near total blackout of information from Oracle. People are fleeing Solaris in droves, and migrating to anything they can: Linux, FreeBSD (Dtrace and ZFS), AIX - hell, even HP-UX looks like a safer bet at the moment. And I never thought I'd find myself saying that!
It certainly appears that Oracle are doing a superb job of killing Solaris. But why would they do this, having paid all that money for Sun and announcing that they will increase spending on Solaris development ?
Well, this post on Slashdot (allegedly from a Sun/Oracle employee) confirms my suspicions as to why they may be doing this. Oracle just really doesn't care about Solaris as a general purpose data centre OS any more. There's just no money in it, and although I personally find it tragic it does make sense. It's probably also why they're killing all their OEM deals - why help a competitor sell hardware, when all you'll see from it is a possible support contract for the OS ? Oracle's overall aim is to have Solaris relegated to the role of running as the bottom layer in an Oracle "database machine", Java appserver bundle or inside a "Fishworks" storage appliance.
It excels at these tasks, and it would obviously fit into Oracle's stated goal of being a one stop shop, where if you want to run Oracle, they'll sell you the bundle - hardware, storage, OS and software. If they no longer want Solaris to be a dominant general purpose OS, then their approach makes sense. They don't need a "community" around the product, they don't need open source developers porting applications to it, and they certainly don't need the overhead of running and managing a community portal any more. Unless you are running (and paying for) Oracle applications on Solaris, you're probably more of an annoyance to them at the moment and I get the very strong idea that they'd rather you just quietly went elsewhere.
I just wish that if this was their plan, they'd make some sort of statement about it; rather than ignoring the Solaris community in the hopes that they'll eventually get frustrated and leave without Oracle having to spell it out for them. I think the way they are going about it reprehensible and it's a tragic end for such a historic and innovative OS. Sadly though, Larry is all about the bottom line and the old, altruistic Sun approach just wasn't bringing in the big bucks. As the Slashdot poster said : "Profit is king here. Anything else is overhead, and overhead eats into Larry's yacht fund."
Edit: Now it's official : http://www.theregister.co.uk/2010/08/13/opensolaris_is_dead/