Jethro Carr

LVM is a fantastic technology that allows block devices on Linux to be sliced into volumes(partitions) and allows for easy resizing and changing of the volumes whilst running, in a much easier fashion than traditional MSDOS or GPT partition tables. LVM is commonly used as the default at install of most popular Linux distributions due to the ease of management.

I use LVM for providing logical volumes for each virtual machine on my KVM server. This makes it easy to add, delete, resize and snapshot my virtual machines and also reduces I/O overhead compared to having virtual machines inside files ontop of a file system.

I recently had the misfortune of needing to rebuild this array after an underlying storage failure in my RAID array. Doing so manually would be time consuming and error prone, I would have to try and determine what the sizes of my different VMs used to be before the failure and then issue lvcreate commands for them all to restore the configuration.

LVM Configuration Backup & Restore

Thankfully I don’t need to restore configuration manually – LVM keeps a backup of all configured volumes/groups, which can be used to restore the original settings of the LVM volume group, including the physical volume settings, the volume group and all the logical volumes inside it.

There is a copy of the latest/current configuration in /etc/lvm/backup/, but also archived versions in /etc/lvm/archive/, which is useful if you’ve broken something badly and actually want an older version of the configuration.

Restoring is quite simple, assuming that firstly your server still has /etc/lvm/ in a good state, or you’ve backed it up to external media and secondly that the physical volume(s) that you are restoring to are at least as large as they were at the time of the configuration backup.

Firstly, you need to determine which configuration you want to restore. In my case, I want to restore my “vg_storage” volume group.

# ls -l /etc/lvm/archive/vg_storage_00*
-rw-------. 1 root root 13722 Oct 28 23:45 /etc/lvm/archive/vg_storage_00419-1760023262.vg
-rw-------. 1 root root 14571 Oct 28 23:52 /etc/lvm/archive/vg_storage_00420-94024216.vg
...
-rw-------. 1 root root 14749 Nov 23 15:11 /etc/lvm/archive/vg_storage_00676-394223172.vg
-rw-------. 1 root root 14733 Nov 23 15:29 /etc/lvm/archive/vg_storage_00677-187019982.vg
#

Once you’ve chosen which configuration to restore, you need to fetch the UUID of the old physical volume(s) from the LVM backup file:

# less /etc/lvm/archive/vg_storage_00677-187019982.vg
...
   physical_volumes {
      pv0 {
         id = "BgR0KJ-JClh-T2gS-k6yK-9RGn-B8Ls-LYPQP0"
...

Now we need to re-create the physical volume with pvcreate. By specifying both the UUID and the configuration file, pvcreate rebuilds the physical volume with all the original values, including extent size and other details to ensure that the volume group configuration applies correctly onto it.

# pvcreate --uuid "BgR0KJ-JClh-T2gS-k6yK-9RGn-B8Ls-LYPQP0" \
  --restorefile /etc/lvm/archive/vg_storage_00677-187019982.vg

If your volume group is made up of multiple physical volumes, you’ll need to do this for each physical volume – make sure you give the right volumes the right UUIDs if the sizes of the physical volumes differ.

Now that the physical volume(s) are created, we can restore the volume group with vgcfgrestore. Because we’ve created the physical volume(s) with the same UUIDs, it will correctly rebuild using the right volumes.

# vgcfgrestore -f /etc/lvm/archive/vg_storage_00677-187019982.vg vg_storage

At this time, you should be able to display the newly restored volume group:

# vgdisplay
  --- Volume group ---
  VG Name               vg_storage
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1044
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                30
  Open LV               11
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               4.55 TiB
  PE Size               4.00 MiB
  Total PE              1192176
  Alloc PE / Size       828160 / 3.16 TiB
  Free  PE / Size       364016 / 1.39 TiB
  VG UUID               OIs8bs-nIE1-bMTj-vSt2-ue80-jf1X-oZCYiO
#

Finally make sure you activate all logical volumes – sometimes LVM restores with volumes set to inactive so they can’t be modified/used – use vgchange -a y to activate all volumes for the volume group:

# vgchange -a y vg_storage

The array is now restored and ready to use. Run lvdisplay and you’ll see all the restored logical volumes. Of course, this is only the configuration – there’s no data in those volumes, so you still have to restore the volumes from backups (you DO have backups right?).

Data Backup & Restore

Once you have your volume configuration restored, you need restore your data. The way you do this is really going to depend on the nature of your data and the volume of it, however I use a combination of two of the most common approaches to backups:

1. A regular data-only level backup to protect all the data I care about. However in the event of a storage failure, I’m left having to rebuild all my virtual machines OS images, before being able to load in the data-only backups into the created VMs.
2. An infrequent block-level backup of all the LVM volumes, where images of each volume are dumped onto external media, which can be loaded back in once the LVM configuration has been restored.

I have a small Perl script I’ve written which does this snapshot & backup process for me. It makes use of LVM snapshots to ensure data consistency during the backup process and compresses with gzip to reduce the on disk backup size without too much CPU overhead.

#!/usr/bin/perl -w
#
# Run through a particular LVM volume group and perform a snapshot
# and compressed file backup of each volume.
#
# This script is intended for use with backing up complete system
# images of VMs, in addition to data level backups.
#

my $source_lvm_volgroup = 'vg_storage';
my $source_lvm_snapsize = '5G';
my @source_lvm_excludes = ('lv_unwanted', 'lv_tmpfiles');

my $dest_dir='/mnt/backup/snapshots';

foreach $volume (glob("/dev/$source_lvm_volgroup/*"))
{
    $volume            =~ /\/dev\/$source_lvm_volgroup\/(\S*)$/;
    my $volume_short    = $1;

    if ("$volume_short" ~~ @source_lvm_excludes)
    {
        # Excluded volume, we skip it
        print "[info] Skipping excluded volume $volume_short ($volume)\n";
        next;
    }

    print "[info] Processing volume $volume_short ($volume)\n";

    # Snapshot volume
    print "[info] Creating snapshot...\n";
    system("lvcreate -n ${volume_short}_snapshot --snapshot $volume -L $source_lvm_snapsize");

    # Write compressed backup file from snapshot, but only replace existing one once finished
    print "[info] Creating compressed snapshot file...\n";
    system("dd if=${volume}_snapshot | gzip --fast > $dest_dir/$volume_short.temp.gz");
    system("mv $dest_dir/$volume_short.temp.gz $dest_dir/$volume_short.gz");

    # Delete snapshot
    print "[info] Removing snapshot...\n";
    system("lvremove --force ${volume}_snapshot");

    print "[info] Volume $volume_short backup completed.\n";
}

print "[info] FINISHED! VolumeGroup backup completed\n";

exit 0;

Since I was doing these backups before my storage failure, by restoring the LVM volume group configuration, it was an easy process to restore all my LVM volume data with a simple matching restore script:

#!/usr/bin/perl -w
#
# Runs through LVM snapshots taken by backup_snapshot_lvm.pl and
# restores them to the LVM volume in question.
#

my $dest_lvm_volgroup = 'vg_storage';
my $source_dir = '/mnt/backup/snapshots';

print "[WARNING] Beginning restore process in 5 SECONDS!!\n";
sleep(5);

foreach $volume (glob("$source_dir/*"))
{
    $volume            =~ /$source_dir\/(\S*).gz$/;
    my $volume_short    = $1;

    print "[info] Processing volume $volume_short ($volume)\n";

    # Just need to decompress & write into LVM volume
    system("zcat $source_dir/$volume_short.gz > /dev/$dest_lvm_volgroup/$volume_short");

    print "[info] Volume $volume_short restore completed.\n";
}

print "[info] FINISHED! VolumeGroup restore completed\n";

exit 0;

Because the block-level backups are less frequent than the data ones, the way I do a restore is to restore these block level backups and then apply the latest data backup over the top of the VMs once they’re booted up again.

Using both these techniques, it can be very fast and hassles free to rebuild a large LVM volume group after a serious storage failure – I’ve used these exact scripts a few times and can restore a several TB volume with 30+ VMs in a matter of hours simply by kicking off a few commands and going out for a beer to wait for the data to transfer from the backup device.