Mark's blog

It's been a while now since I last updated this blog with any decent material (The Poo Truck notwithstanding, as honestly, that's a classic) so I thought I'd dust off some of my notes on building a redundant iSCSI and NFS SAN using Debian Etch.

The following post takes the form of a "HOWTO" guide - I'll include all the relevant commands, configuration files and output produced so you can follow along. This is the first part of the series; I'll post the different sections in phases, each covering a different part of the setup. The plan is to cover all this in 5 (possibly 6) separate posts, with the following content :

Part 1 - Overview, network layout and DRBD installation
Part 2 - DRBD and LVM
Part 3 - Heartbeat and automated failover
Part 4 - iSCSI and IP failover
Part 5 - Multipathing and client configuration
Part 6 - Anything left over!

So, this being part one, I'll start with a quick overview of what I'm trying to achieve here :

Cluster overview

The cluster will consist of 2 storage servers, providing iSCSI and NFS services to a number of clients, over floating IP addresses and from a replicated pool of storage. This storage will be used for file sharing (NFS), and block devices (iSCSI) - although you could add any kind of service on top of the cluster; an obvious option would be to provide SMB (Microsoft Windows file services), although I won't explore that particular avenue.

This will be replicated with DRBD, and managed using LVM2. I'll also be using multipathing to the storage, so that a component (NIC, switch, cable etc.) can fail in one channel but the storage will still be accessible. Failover and cluster management will be provided by the Linux-HA project.

The distribution I'm using is Debian Etch (4.0), although most of the configuration files and commands used will work on any distro, although file locations and the package management commands will obviously differ.

Network layout

The two storage nodes (which I'll call "otter" and "weasel") will have the following 4 network interfaces configured :

• eth0: 172.16.1.x -Management interface (the address we SSH into to manage the system)
• eth1: 10.0.0.x - This is for data replication and heartbeat between the two nodes, and will be via a cross-over cable connected directly between the two servers
• eth2 and eth3: 192.168.1.x - This is the storage network, clients will connect to this for their storage.

And the client (which I'll call "badger") will have the following 3 network interfaces configured :

• eth0 : 172.16.1.x - Management / public interface
• eth1 and eth2 : 192.168.x.1 - Storage network (where we access the iSCSI and NFS storage). These will use 192.168.1.1 and 192.168.2.1, both with a netmask of 255.255.255.0 to ensure that requests go to the correct interface when using multipathing (more on that later).

In a real-world scenario, these would be on physically different NICs, and would also be on separate switches - particularly the multipathed storage interfaces. Utilising the different private ranges makes it easier to see at a glance what is going on, and makes trouble-shooting a lot easier. It's also obviously a good idea to separate your storage network from the rest of your regular network traffic.

Of course, there is nothing stopping you from utilising virtual NICs and having each address on eth0:1, eth0:2 and so on. Obviously, GigE or higher would be required in a production network, but there's nothing stopping you from using 100Mb in a test/development environment. Just don't expect stellar performance!

There will also be a null-modem cable connected between the two serial ports on each storage node. This is to supplement the network heartbeat, and will help avoid the problem of "split-brain" that can occur in clusters. If there was a problem with the heartbeat network - the switch failing, for instance - both nodes would then see the other as failed, and try to assume the master role. Having a secondary heartbeat connection between the nodes will help avoid this problem - particularly as it is a "straight-through" connection, and does not rely on any intermediate devices such as a network switch.

At this point, a diagram might be in order - you'll have to excuse my "Dia" skills, which are somewhat lacking!

This diagram shows all the important connections between the various hosts, so hopefully this will make things a little clearer. You can see that with this architecture, we could loose one storage server or one switch, and we'd still have a valid path to the storage from our client.