Category: Training

ZFS Training Exercises

In 2015 I ran a training session on BTRFS and ZFS. Here are the notes for ZFS updated for the latest versions of everything when running on KVM. You can do all the same exercises on a non virtual machine, but it’s safest not to do this on a machine with important data. For the KVM setup I use /dev/vdc, /dev/vdd, and /dev/vde are the devices for test data.

For running ZFS on Linux in serious use I recommend having more than 4G of RAM in the system or VM that’s doing it to prevent kernel OOM issues. You should be able to complete this training exercise on a 64bit VM with 768M of RAM.

  1. Installing zfsonlinux on Debian/Buster
    1. Note that /etc/apt/sources.list has the following line (the “contrib” is very important):
      deb http://mirror.internode.on.net/pub/debian/ buster-updates contrib main
    2. Run the following commands:
      apt install linux-headers-amd64
      apt install -t buster-backports dkms spl-dkms
      apt install -t buster-backports zfs-dkms zfsutils-linux
      The last 2 commands take ages, the last one is good for a coffee break.
  2. Making the filesystem
    1. Create the pool:
      zpool create -o ashift=12 nyancat raidz /dev/vdc /dev/vdd /dev/vde
      NB You need to add a “-f” to overwrite the BTRFS filesystem
    2. Make some default filesystem settings:
      zfs set devices=off nyancat
      zfs set atime=off nyancat
      zfs set setuid=off nyancat
      zfs set compression=on nyancat
    3. Get information on the pool:
      zfs get all nyancat|less
      zpool list
      zpool status
    4. Make a filesystem:
      zfs create nyancat/friday
      df -h
    5. See if there are any errors, shouldn’t be any (yet):
      zpool scrub nyancat
      zpool status
    6. Copy some files to the filesystem:
      cp -r /usr /nyancat/friday
    7. Scrub the filesystem:
      zpool scrub nyancat
      zpool status
    3. Online corruption

    1. Corrupt the filesystem:
      dd if=/dev/zero of=/dev/vdd bs=1024k count=2000 seek=50
    2. Scrub again, should give a warning about errors:
      zpool scrub nyancat
      zpool status
    3. Verify that the data is intact:
      diff -ru /usr /nyancat/friday/usr
    4. Corrupt it again:
      dd if=/dev/zero of=/dev/vdd bs=1024k count=2000 seek=50
    5. Unmount the ZFS filesystems and export (stop using) the zpool:
      zfs umount -a
      zpool export -a
      zpool status
    6. “Import” the pool and mount the filesystems:
      zpool import -a
      zfs mount -a
    7. Check that the data is still intact and look for increases in the error count:
      zpool status
      diff -ru /usr /nyancat/friday/usr
      zpool status
    4. Offline corruption

    1. Umount the filesystem and corrupt the start of one device:
      zfs umount -a
      zpool export -a
      dd if=/dev/zero of=/dev/vdd bs=1024k count=200
    2. Try mounting it again which will work even though one block device is unavailable:
      zpool import -a
      zpool status
    3. Replace the missing device, look for the device name in the “zpool status” output to replace “123456789” in the following sample command:
      zpool replace -f nyancat 123456789 /dev/vdd
    4. Check that everything is ok:
      zpool status
      diff -ru /usr /nyancat/friday/usr

BTRFS Training Exercises

In 2015 I ran a training session on BTRFS and ZFS. Here are the notes for BTRFS updated for the latest versions of everything when running on KVM. You can do all the same exercises on a non virtual machine, but it’s safest not to do this on a machine with important data. For the KVM setup I use /dev/vdc and /dev/vdd are the devices for test data.

On a Debian/Buster system all that you need to do to get ready for this is to install the btrfs-progs package.

  1. Making the filesystem
    1. Make the filesystem, this makes a filesystem that spans 2 devices (note you must use the-f option if there was already a filesystem on those devices):
      mkfs.btrfs /dev/vdc /dev/vdd
    2. Use file(1) to see basic data from the superblocks:
      file -s /dev/vdc /dev/vdd
    3. Mount the filesystem (can mount either block device, the kernel knows they belong together):
      mount /dev/vdd /mnt/tmp
    4. See a BTRFS df of the filesystem, shows what type of RAID is used (note the use of RAID-0 for data):
      btrfs filesystem df /mnt/tmp
    5. See more information about FS device use:
      btrfs filesystem show /mnt/tmp
    6. Balance the filesystem to change it to RAID-1 and verify the change, note that some parts of the filesystem were single and RAID-0 before this change):
      btrfs balance start -dconvert=raid1 -mconvert=raid1 -sconvert=raid1 --force /mnt/tmp
      btrfs filesystem df /mnt/tmp
    7. See if there are any errors, shouldn’t be any (yet):
      btrfs device stats /mnt/tmp
    8. Copy some files to the filesystem:
      cp -r /usr /mnt/tmp
    9. Check the filesystem for basic consistency (only checks checksums):
      btrfs scrub start -B -d /mnt/tmp
  2. Online corruption
    1. Corrupt the filesystem:
      dd if=/dev/zero of=/dev/vdd bs=1024k count=2000 seek=50
    2. Scrub again, should give a warning about errors:
      btrfs scrub start -B /mnt/tmp
    3. Check error count:
      btrfs device stats /mnt/tmp
      dmesg
    4. Corrupt it again:
      dd if=/dev/zero of=/dev/vdd bs=1024k count=2000 seek=50
    5. Unmount it:
      umount /mnt/tmp
    6. In another terminal follow the kernel log:
      tail -f /var/log/kern.log
    7. Mount it again and observe it correcting errors on mount:
      mount /dev/vdd /mnt/tmp
    8. Run a diff, observe kernel error messages and observe that diff reports no file differences:
      diff -ru /usr /mnt/tmp/usr/
    9. Run another scrub, this will probably correct some errors which weren’t discovered by diff:
      btrfs scrub start -B -d /mnt/tmp
  3. Offline corruption
    1. Umount the filesystem, corrupt the start, then try mounting it again which will fail because the superblocks were wiped:
      umount /mnt/tmp
      dd if=/dev/zero of=/dev/vdd bs=1024k count=200
      mount /dev/vdd /mnt/tmp
      mount /dev/vdc /mnt/tmp
    2. Note that the filesystem was not mountable due to a lack of a superblock. It might be possible to recover from this but that’s more advanced so we will restore the RAID.
      Mount the filesystem in a degraded RAID mode, this allows full operation.
      mount /dev/vdc /mnt/tmp -o degraded
    3. Add /dev/vdd back to the RAID:
      btrfs device add /dev/vdd /mnt/tmp
    4. Show the filesystem devices, observe that vdd is listed twice, the missing device and the one that was just added:
      btrfs filesystem show /mnt/tmp
    5. Remove the missing device and observe the change:
      btrfs device delete missing /mnt/tmp
      btrfs filesystem show /mnt/tmp
    6. Balance the filesystem, not sure this is necessary but it’s good practice to do it when in doubt:
      btrfs balance start /mnt/tmp
    7. Umount and mount it, note that the degraded option is not needed:
      umount /mnt/tmp
      mount /dev/vdc /mnt/tmp
  4. Experiment
    1. Experiment with the “btrfs subvolume create” and “btrfs subvolume delete” commands (which act like mkdir and rmdir).
    2. Experiment with “btrfs subvolume snapshot SOURCE DEST” and “btrfs subvolume snapshot -r SOURCE DEST” for creating regular and read-only snapshots of other subvolumes (including the root).