If you’ve ever had to migrate a service to a new machine, you’ve probably
rsync to be a godsend. It’s ability to pre-sync most data while the
service is still running, then perform the much quicker “sync the new
changes” action after the service has been taken down is fantastic.
For a long time, I’ve wanted a similar tool for block devices. I’ve managed
ridiculous numbers of VMs in my time, almost all stored in LVM logical
volumes, and migrating them between machines is a downtime hassle. You need
to shutdown the VM, do a massive
dd | netcat, and then bring the machine
back up. For a large disk, even over a fast local network, this can be
quite an extended period of downtime.
The naive implementation of a tool that was capable of doing a block-device
rsync would be to checksum the contents of the device, possibly in blocks,
and transfer only the blocks that have changed. Unfortunately, as network
speeds approach disk I/O speeds, this becomes a pointless operation.
Scanning 200GB of data and checksumming it still takes a fair amount of time
– in fact, it’s often nearly as quick to just send all the data as it is to
checksum it and then send the differences.1
No, a different approach is needed for block devices. We need something that keeps track of the blocks on disk that have changed since our initial sync, so that we can just transfer those changed blocks.
As it turns out, keeping track of changed blocks is exactly what LVM snapshots do. They actually keep a copy of what was in the blocks before it changed, but we’re not interested in that so much. No, what we want is the list of changed blocks, which is stored in a hash table on disk.
All that was missing was a tool that read this hash table to get the list of blocks that had changed, then sent them over a network to another program that was listening for the changes and could write them into the right places on the destination.
That tool now exists, and is called
lvmsync. It is a slightly crufty
chunk of ruby that, when given a local LV and a remote machine and block
device, reads the snapshot metadata and transfers the changed blocks over an
SSH connection it sets up.
Be warned: at present, it’s a pretty raw piece of code. It does nothing but the “send updated blocks over the network”, so you have to deal with the snapshot creation, initial sync, and so on. As time goes on, I’m hoping to polish it and turn it into something Very Awesome. “Patches Accepted”, as the saying goes.
rsyncavoids a full-disk checksum because it cheats and uses file metadata (the last-modified time, or
mtimeof a file) to choose which files can be ignored. No such metadata is available for block devices (in the general case). ↩