Blog: Full Disk Encryption (with ZFS root) for FreeBSD 9.x

WARNING: This post has been marked as obsolete and may be incorrect. It is kept for archival purposes only. To follow on from my post about full disk encryption (well almost), this is how to do the same but with a ZFS filesystem.  Like the other post, your /boot folder (which contains your kernel and modules) will not be encrypted, but the rest of your filesystem will be.

One disadvantage of this method is that you have to enter a passphrase for EACH disk in your ZFS system each boot.  Encryption inside ZFS will appear at some point – but until then this will suffice !

Boot from any FreeBSD 9 install medium (except bootonly), and choose Live CD at the install menu.

For the purposes of this article, I will assume that you're using 4 disks (da0, da1, da2, da3), a 10GB /boot (this will be mirrored on each of the 4 disks), and the remaining space as a raidz1 (roughly similar to RAID5) ZFS filesystem.  The contents will be encrypted with 256-bit AES-XTS encryption with a 4 kb random data partial key and a secondary passphrase (required to type on each boot).

If your CPU supports the AESNI flag, the crypto(4) framework will utilise this too.

First we need to remove any existing GPT or MBR partition tables on each of the disks (ignore any 'invalid argument' messages):

gpart destroy -F da0
gpart destroy -F da1
gpart destroy -F da2
gpart destroy -F da3 Copy

Now we need to create a new GPT partition table on each disk:

gpart create -s gpt da0
gpart create -s gpt da1
gpart create -s gpt da2
gpart create -s gpt da3 Copy

We will now create a 64kb boot partition (this contains the boot loader only, so is safe and required to be unencrypted):

gpart add -s 128 -t freebsd-boot da0
gpart add -s 128 -t freebsd-boot da1
gpart add -s 128 -t freebsd-boot da2
gpart add -s 128 -t freebsd-boot da3 Copy

Next, we will create the /boot partition – you can adjust the sizes here if you need, but i'd suggest not shrinking it too much or you'll get into problems when doing OS upgrades later... Note: this is mirrored not striped across the disks for maximum resilience – so will use 10GB on each disk for 10GB total usable space.

gpart add -s 10G -t freebsd-zfs da0
gpart add -s 10G -t freebsd-zfs da1
gpart add -s 10G -t freebsd-zfs da2
gpart add -s 10G -t freebsd-zfs da3 Copy

Finally, we will assign the remaining space on each disk to the root ZFS partition.  This will be encrypted before we build ZFS on top of it.

gpart add -t freebsd-zfs da0
gpart add -t freebsd-zfs da1
gpart add -t freebsd-zfs da2
gpart add -t freebsd-zfs da3 Copy

Now that we've created daXp1 (bootloader), daXp2 (/boot partition), daXp3 (root partition) – we need to write the boot loader code to each disk:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da2
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da3 Copy

Ok, next we will build a ramdisk to mount on /boot/zfs – this helps us mount things temporarily...

mdconfig -a -t malloc -s 128m -u 2
newfs -O2 /dev/md2
mount /dev/md2 /boot/zfs Copy

Now we will load the modules required for ZFS and encryption:

kldload opensolaris
kldload zfs
kldload geom_eli Copy

Next, we build a ZFS mirror for the /boot partition and mount it temporarily (to house the encryption key) – ignore any mention of unable to mount:

zpool create bootdir mirror /dev/da0p2 /dev/da1p2 /dev/da2p2 /dev/da3p2
zpool set bootfs=bootdir bootdir
mkdir /boot/zfs/bootdir
zfs set mountpoint=/boot/zfs/bootdir bootdir
zfs mount bootdir Copy

Now we generate a random 4kb encryption key that will form (along with passphrase) the encryption key for the disk:

dd if=/dev/random of=/boot/zfs/bootdir/encryption.key bs=4096 count=1 Copy

We have everything we need to start encrypting the disks now... Enter your passphrase twice for each init phase and once again for each attach phase below:

geli init -b -B /boot/zfs/bootdir/da0p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da0p3
geli init -b -B /boot/zfs/bootdir/da1p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da1p3
geli init -b -B /boot/zfs/bootdir/da2p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da2p3
geli init -b -B /boot/zfs/bootdir/da3p3.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da3p3
geli attach -k /boot/zfs/bootdir/encryption.key /dev/da0p3
geli attach -k /boot/zfs/bootdir/encryption.key /dev/da1p3
geli attach -k /boot/zfs/bootdir/encryption.key /dev/da2p3
geli attach -k /boot/zfs/bootdir/encryption.key /dev/da3p3 Copy

Now that we have encrypted and mounted the partitions, we can build a ZFS root filesystem on top of it like so:

zpool create zroot raidz1 /dev/da0p3.eli /dev/da1p3.eli /dev/da2p3.eli /dev/da3p3.eli
zfs set mountpoint=/boot/zfs/zroot zroot
zfs mount zroot
zfs unmount bootdir
mkdir /boot/zfs/zroot/bootdir
zfs set mountpoint=/boot/zfs/zroot/bootdir bootdir
zfs mount bootdir Copy

Note we unmounted the old boot mirror and re-mounted it within the root filesystem.  This will be used later to copy the kernel and modules into.

Ok, now we create all our ZFS mounts with various options as follows:

zfs set checksum=fletcher4 zroot
zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp
chmod 1777 /boot/zfs/zroot/tmp
zfs create zroot/usr
zfs create zroot/usr/home
cd /boot/zfs/zroot; ln -s /usr/home home
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages
zfs create zroot/var
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
chmod 1777 /boot/zfs/zroot/var/tmp Copy

Now we're ready to install FreeBSD onto the new ZFS partitions.  We're going to install the base, all sources and a generic kernel – this takes some time so please be patient...

cd /boot/zfs/zroot
unxz -c /usr/freebsd-dist/base.txz | tar xpf –
unxz -c /usr/freebsd-dist/kernel.txz | tar xpf –
unxz -c /usr/freebsd-dist/src.txz | tar xpf – Copy

Now we can set /var/empty to readonly:

zfs set readonly=on zroot/var/empty Copy

And now we're ready to chroot into the installed system to setup the configuration:

chroot /boot/zfs/zroot Copy

Now that the base system and kernel are installed, we can move our /boot folder to it's final place on the ZFS unencrypted mirror and do a little housekeeping:

cd /
mv boot bootdir/
ln -fs bootdir/boot
mv bootdir/encryption.key bootdir/boot/
mv bootdir/*.eli bootdir/boot/ Copy

We need to setup an initial /etc/rc.conf which will mount all ZFS filesystems on boot:

echo 'zfs_enable="YES"' > /etc/rc.conf
touch /etc/fstab Copy

And an initial /boot/loader.conf that will load ZFS, encryption and settings for encrypted disks on boot:

echo 'vfs.zfs.prefetch_disable="1"' > /boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /boot/loader.conf
echo 'zfs_load="YES"' >> /boot/loader.conf
echo 'aesni_load="YES"' >> /boot/loader.conf
echo 'geom_eli_load="YES"' >> /boot/loader.conf
echo 'geli_da0p3_keyfile0_load="YES"' >> /boot/loader.conf
echo 'geli_da0p3_keyfile0_type="da0p3:geli_keyfile0"' >> /boot/loader.conf
echo 'geli_da0p3_keyfile0_name="/boot/encryption.key"' >> /boot/loader.conf
echo 'geli_da1p3_keyfile0_load="YES"' >> /boot/loader.conf
echo 'geli_da1p3_keyfile0_type="da1p3:geli_keyfile0"' >> /boot/loader.conf
echo 'geli_da1p3_keyfile0_name="/boot/encryption.key"' >> /boot/loader.conf
echo 'geli_da2p3_keyfile0_load="YES"' >> /boot/loader.conf
echo 'geli_da2p3_keyfile0_type="da2p3:geli_keyfile0"' >> /boot/loader.conf
echo 'geli_da2p3_keyfile0_name="/boot/encryption.key"' >> /boot/loader.conf
echo 'geli_da3p3_keyfile0_load="YES"' >> /boot/loader.conf
echo 'geli_da3p3_keyfile0_type="da3p3:geli_keyfile0"' >> /boot/loader.conf
echo 'geli_da3p3_keyfile0_name="/boot/encryption.key"' >> /boot/loader.conf Copy

The above settings tell the OS which encryption keyfile to use for each disk partition.

Now you can set your root password:

passwd root Copy

And configure your timezone:

tzsetup Copy

And setup a dummy /etc/mail/aliases file to prevent sendmail warnings:

cd /etc/mail
make aliases Copy

Now you can configure any additional settings you require (such as adding new users, configuring networking or setting sshd to run on boot) – when you're done, we need to exit the chroot:

exit Copy

Now, we need to make sure the bootloader can read our ZFS pool cache (or it wont mount our ZFS disks on boot):

cd /boot/zfs
cp /boot/zfs/zpool.cache /boot/zfs/zroot/boot/zfs/zpool.cache Copy

Finally, we need to unmount all the ZFS filesystems and configure their final mountpoints...

zfs unmount -a
zfs set mountpoint=legacy zroot
zfs set mountpoint=/tmp zroot/tmp
zfs set mountpoint=/usr zroot/usr
zfs set mountpoint=/var zroot/var
zfs set mountpoint=/bootdir bootdir Copy

Now we can 'reboot' and remove the media while the computer reboots.  Do this as soon as you can.

The computer should reboot into a ZFS-based filesystem, booted from a software RAID array on fully protected disks with all but /boot partition encrypted.  Note: it will ask you to enter a passphrase for each disk parition used above (4 times) – you should take care to enter the correct passwords as it will treat any passwords missed as a failed disk (you get 3 attempts at each password)

Once it's booted, you can login and run sysinstall to configure other options like networking and startup programs (like SSH!)

The only point to note is that when you do an OS upgrade, during the "mergemaster" stage, it will complain that /boot is a symlink not a directory.  Simply tell it to ignore/do nothing and it will install the files as normal.

Enjoy!

Text Console resolution and geometry settings in FreeBSD 9.x Reducing the effects of a DNS Amplification attack on authorative DNS server with FreeBSD and ipfw