Blog: Using FreeBSD as a Hypervisor (using bhyve and vm-bhyve to manage them)

To do this, you must be running at least FreeBSD 10.3-RELEASE – if you're running an older version of FreeBSD, you should update!  Also, ensure that you have a CPU that supports hardware virtualisation.  Most new server CPUs seem to work just fine.

First, we need to install some packages to make things much easier:

pkg install vm-bhyve grub2-bhyve Copy

Now we need to load some kernel modules on boot to allow us to virtualise and bridge networking... add the following to /boot/loader.conf:

if_bridge_load="YES"
if_tap_load="YES"
nmdm_load="YES"
vmm_load="YES" Copy

You need to reboot to activate the modules, or you can load them now:

kldload if_bridge if_tap nmdm vmm Copy

And we need to add some configuration to /etc/rc.conf:

vm_enable="YES"
vm_dir="zfs:zroot/vms"
vm_list=""
vm_delay="5" Copy

Now we will setup some storage for the hypervisor – For this, we'll assume 'zroot' zfs pool has enough capacity:

zfs create -o mountpoint=/vms zroot/vms
vm init
cp /usr/local/share/examples/vm-bhyve/* /vms/.templates/ Copy

Now we can create a virtual switch (bridge) and bridge it to our network interface.  Change igb0 to your network interface below:

vm switch create public
vm switch add public igb0 Copy

Now, fetch an installation ISO for a FreeBSD guest:

vm iso ftp://ftp.uk.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/10.3/FreeBSD-10.3-RELEASE-amd64-disc1.iso Copy

Now edit the default template /vms/.templates/default.conf (actually every .conf file in the folder) and change the disk config:

disk0_name="disk0"
disk0_dev="sparse-zvol" Copy

This will make vms use zvols instead of disk image files.  Now we're all prepared.

To create a test 16GB FreeBSD vm:

vm create -s 16G testvm
vm -f install testvm FreeBSD-10.3-RELEASE-amd64-disc1.iso Copy

Select 'vt100' when asked for console type, and install as normal.  Once finished, login to the new vm and power it off (use 'poweroff') – control will return to the host.

Here's some useful commands to manage VMs...

To list all your VMs:

vm list Copy

To start a VM called testvm:

vm start testvm Copy

To stop a VM (gracefully via ACPI) called testvm:

vm stop testvm Copy

To forcibly power off a VM called testvm:

vm poweroff testvm Copy

To press reset on a VM called testvm:

vm reset testvm Copy

To connect to the serial console on a VM called testvm:

vm console testvm Copy

(to disconnect, send ~. to exit the console session)

To permanently destroy a VM called testvm (must be turned off first):

vm destroy testvm Copy

To edit the configuration of a VM called testvm (e.g. to change the MAC address):

vm configure testvm Copy

To increase the size of the disk on a VM called testvm to 32GB:

zfs set volsize=32G zroot/vms/testvm/disk0 Copy

To create an Ubuntu guest using a previously downloaded ISO called ubuntu.iso:

vm create -t ubuntu -s 16G testubuntu
vm install testubuntu ubuntu.iso Copy

To make Ubuntu boot unaided each time, edit the configuration file for the VM just created (in /vms/testubuntu/testubuntu.conf) and add:

grub_run_dir="/grub"
grub_run_file="grub.cfg" Copy

You can also add the above to the ubuntu template file for it to be automatically added to ubuntu VMs.

Windows guests take a lot more effort as there is no GUI console.  This also means debugging a broken windows install is almost impossible so I would discourage its use for production.  However, this is how to prepare to install Windows VMs:

fetch -o /vms/.config/BHYVE_UEFI.fd http://people.freebsd.org/%7Egrehan/bhyve_uefi/BHYVE_UEFI_20151002.fd
pkg install git p7zip cdrtools-devel
mkdir /vms/work
cd /vms/work
git clone https://github.com/nahanni/bhyve-windows-unattend-xml
fetch https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.96/virtio-win-0.1.96.iso Copy

This fetches a special UEFI loader, a collection of automated unattended Windows installation scripts, and the VirtIO drivers to allow storage to work.

Now copy a Windows installation ISO to /vms/work/win.iso and then:

mkdir tmp
cd tmp
7z x ../win.iso
cp /vms/work/bhyve-windows-unattend-xml/files/win2012r2_AutoUnattend.xml AutoUnattend.xml Copy

This extracts the Windows ISO and copies an automated unattended installation script to it.  There are several AutoUnattend.xml files so choose the one matching the OS you’re installing... we're doing a Windows 2012R2 install here.

You can edit the AutoUnattend.xml file to set a product key if required (not required for recent Windows OS)

Now we copy the VirtIO drivers into the temporary work directory and build a new ISO image:

mkdir virtio
cd virtio
tar xf /vms/work/virtio-win-0.1.96.iso
cd /vms/work/tmp
mkisofs \
-b boot/etfsboot.com -no-emul-boot -c BOOT.CAT \
-iso-level 4 -J -l -D \
-N -joliet-long \
-relaxed-filenames -v \
-V Custom -udf \
-boot-info-table -eltorito-alt-boot -eltorito-platform 0xEF \
-eltorito-boot efi/microsoft/boot/efisys_noprompt.bin \
-no-emul-boot \
-o /vms/.iso/win2k12r2.iso . Copy

You now have a win2k12r2.iso file ready to build a windows VM:

vm create -t windows -s 64G wintest
vm install wintest win2k12r2.iso Copy

This will start the unattended installation of windows.  It can take up to around 25 minutes.  You can monitor its progress by looking at the logfile:

tail -f /vms/wintest/vm-bhyve.log Copy

You need to wait for it to restart twice (so three "starting bhyve" messages) – you can then get the IP from the server:

vm console wintest Copy

and press "i" to view the IP address – this only seems to work on win2k12r2 and above.  You may need to check your DHCP server logs otherwise.

You can now RDP to the VM using Administrator and Test123 as the password.

Again, you should shutdown the VM and start it using the vm start wintest command.

To make VMs automatically start on boot, you can add them to /etc/rc.conf in the vm_list variable, for example to start wintest and testvm:

vm_list="wintest testvm" Copy

Enjoy.

Zen Broadband with IPv6 using FreeBSD ppp router OpenVPN Setup in FreeBSD (with NAT for IPv4 and IPv6)