ZFS Training Exercises

In 2015 I first ran a training session on BTRFS and ZFS. Here are the notes for ZFS updated for the latest versions of everything when running on KVM. You can do all the same exercises on a non virtual machine, but it’s safest not to do this on a machine with important data. For the KVM setup I use /dev/vdb, /dev/vdc, /dev/vdd, and /dev/vde are the devices for test data.

For running ZFS on Linux in serious use I recommend having more than 4G of RAM in the system or VM that’s doing it to prevent kernel OOM issues. In 2015 when I first ran this training it was possible to complete it on a 64bit VM with 2G of RAM, but in recent times I have used 6G VMs and now use 8G, having more RAM is always good if you can do it.

Installing zfsonlinux on Debian/Trixie

1. Note that /etc/apt/sources.list has the following lines (the “contrib” is very important):

   deb http://ftp.au.debian.org/debian/ trixie main non-free non-free-firmware contrib
   deb http://deb.debian.org/debian trixie-backports contrib main

2. Run the following commands (zfs-dkms takes a long time):

   apt update
   apt install linux-headers-amd64
   apt install -t stable-backports zfsutils-linux zfs-dkms

   The apt command installing zfs-dkms takes a while, 4 minutes real time and 9 minutes CPU time on a 4 core VM on a Dell R710 and 2 minutes real time and 9 minutes CPU time on an 8 core VM on a HP Z840. It will take much longer if several people are doing it at the same time.

Making the filesystem

1. Create the pool (data set to contain multiple filesystems):

   zpool create -o ashift=12 nyancat raidz /dev/vdb /dev/vdc /dev/vdd /dev/vde

   NB You need to add a “-f” if you need to overwrite a previous filesystem

2. Make some default filesystem settings:

   zfs set devices=off nyancat
   zfs set atime=off nyancat
   zfs set setuid=off nyancat
   zfs set compression=on nyancat

3. Get information on the pool:

   zfs get all nyancat|less
   zpool list
   zpool status

4. Make a filesystem (which will be auto mounted on /nyancat/friday):

   zfs create nyancat/friday
   df -h

5. See if there are any errors, shouldn’t be any (yet):

   zpool scrub nyancat
   zpool status

6. Copy some files to the filesystem:

   cp -r /usr /nyancat/friday

7. Scrub the filesystem, normally scrubbing once a month is plenty but more often is good when learning:

   zpool scrub nyancat
   zpool status

Online corruption

1. Corrupt the filesystem:

   dd if=/dev/zero of=/dev/vdd bs=1024k count=2000 seek=50

2. Scrub again, should give a warning about errors:

   zpool scrub nyancat
   zpool status

3. Verify that the data is intact:

   diff -ru /usr /nyancat/friday/usr

4. Corrupt it again:

   dd if=/dev/zero of=/dev/vdd bs=1024k count=2000 seek=50

5. Unmount the ZFS filesystems and export (stop using) the zpool:

   zfs umount -a
   zpool export -a
   zpool status

6. “Import” the pool, mount the filesystems, and note that the checksum error count is back to zero (it counts since the last mount not for all time – unlike BTRFS):

   zpool import -a
   zfs mount -a
   zpool status

7. Check that the data is still intact and look for increases in the error count:

   zpool status
   diff -ru /usr /nyancat/friday/usr
   zpool status

Offline corruption

1. Umount the filesystem and corrupt the start of one device:

   zfs umount -a
   zpool export -a
   dd if=/dev/zero of=/dev/vdd bs=1024k count=200

2. Try mounting it again which will work even though one block device is unavailable:

   zpool import -a
   zpool status

3. Replace the missing device, look for the device name in the “zpool status” output to replace “123456789” in the following sample command:

   zpool replace -f nyancat 123456789 /dev/vdd

4. Check that everything is ok:

   zpool status
   diff -ru /usr /nyancat/friday/usr