Setting up a FreeBSD Virtualization Host¶

References¶

https://wiki.freebsd.org/bhyve
https://wiki.freebsd.org/bhyve/UEFI
https://man.freebsd.org/cgi/man.cgi?query=bhyve
https://docs.freebsd.org/en/books/handbook/virtualization/#virtualization-host-bhyve
https://github.com/churchers/vm-bhyve
https://libvirt.org/drvbhyve.html
https://www.freshports.org/devel/libvirt
https://people.freebsd.org/~rodrigc/libvirt-bhyve/libvirt-bhyve.html
http://empt1e.blogspot.com/2014/03/bhyve-in-libvirt.html
https://wiki.freebsd.org/qemu

Minimal Setup Requirements¶

Dedicated networking
Dedicated storage
Dedicated mindset

Install Packages¶

Minimally, we need the Bhyve packages. Optionally, qemu and libvirt related packages.

sudo pkg inst qemu-tools qemu edk2-qemu-x64 aqemu
sudo pkg inst bhyve+ bhyve-firmware bhyve-rc-4 edk2-bhyve grub2-bhyve sysutils/vm-bhyve
sudo pkg inst libvirt libvirt-dbus

Create ZFS Dataset Structure¶

Note: encryption is optional

zfs create -o mountpoint=/srv/virt \
    -o utf8only=on \
    -o sharenfs=off \
    -o reservation=32M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=256K \
    -o dedup=sha256,verify \
    -o compression=zstd-2 \
    -o encryption=aes-256-gcm \
    -o keyformat=raw \
    -o keylocation=file:///root/.zfs/keys/optane-aics.zfs.key \
    optane-aics/virt || exit 1

zfs create -o mountpoint=/srv/virt/vmm \
    -o utf8only=on \
    -o sharenfs=off \
    -o reservation=32M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=256K \
    -o dedup=sha256,verify \
    -o compression=zstd-2 \
    -o encryption=aes-256-gcm \
    -o keyformat=raw \
    -o keylocation=file:///root/.zfs/keys/optane-aics.zfs.key \
    optane-aics/virt/vmm || exit 1

zfs create -o mountpoint=/srv/virt/mach \
    -o utf8only=on \
    -o sharenfs=off \
    -o reservation=32M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=256K \
    -o dedup=sha256,verify \
    -o compression=zstd-2 \
    -o encryption=aes-256-gcm \
    -o keyformat=raw \
    -o keylocation=file:///root/.zfs/keys/optane-aics.zfs.key \
    optane-aics/virt/mach || exit 1

Adding DevFS Rules¶

/etc/devfs.rules

[devfsrules_usb_devices=20]
add path 'ttyU*' mode 0660 user operator group dialer
add path 'cuaU*' mode 0660 user operator group dialer
add path 'ugen*' mode 0660 user operator group operator

Additions to Boot Loader Config¶

Add the following to /boot/loader.conf or /boot/loader.conf.local
This example shows a secondary GPU being set to PCIe Passthrough for dedicated VM usage.
Note: if you’re specifying PCIe Passthrough features then a reboot will be required.

# Virtualization
#-----------------------------------------------------#
# Ref: man vmm(4)
# https://wiki.freebsd.org/bhyve/pci_passthru
#-----------------------------------------------------#
kern.racct.enable=1
hw.vmm.maxcpu="16"
vmm_load="YES"
nmdm_load="YES"
if_bridge_load="YES"
bridgestp_load="YES"
#kqemu_load="YES"

#-----------------------------------------------------#
# VM Passthrough GPU: RTX A4000 @ upper pcie slot
# - slot spec: gen4 x16
# - slot dev: vgapci1@pci0:24:0:0 == 24/0/0
#-----------------------------------------------------#
pptdevs="24/0/0"

Configuring VM-Bhyve¶

RC System Additions¶

vm_enable="YES"
vm_dir="zfs:optane-aics/virt/vmm"

Download FreeBSD VM Images + Linux ISOs¶

cat<< EOF> /tmp/vm-bhyve-img-iso.download.sh
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/amd64/Latest/FreeBSD-14.1-RELEASE-amd64-BASIC-CLOUDINIT-ufs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/amd64/Latest/FreeBSD-14.1-RELEASE-amd64-BASIC-CLOUDINIT-zfs.raw.xz

vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/aarch64/Latest/FreeBSD-14.1-RELEASE-arm64-aarch64-ufs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/aarch64/Latest/FreeBSD-14.1-RELEASE-arm64-aarch64-zfs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/aarch64/Latest/FreeBSD-14.1-RELEASE-arm64-aarch64.raw.xz

vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/amd64/Latest/FreeBSD-14.1-RELEASE-amd64-ufs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/amd64/Latest/FreeBSD-14.1-RELEASE-amd64-zfs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/amd64/Latest/FreeBSD-14.1-RELEASE-amd64.raw.xz

vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/i386/Latest/FreeBSD-14.1-RELEASE-i386-ufs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/i386/Latest/FreeBSD-14.1-RELEASE-i386-zfs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/i386/Latest/FreeBSD-14.1-RELEASE-i386.raw.xz

vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/riscv64/Latest/FreeBSD-14.1-RELEASE-riscv-riscv64-ufs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/riscv64/Latest/FreeBSD-14.1-RELEASE-riscv-riscv64-zfs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/riscv64/Latest/FreeBSD-14.1-RELEASE-riscv-riscv64.raw.xz

vm iso http://mirrors.ocf.berkeley.edu/debian-cd/12.5.0/amd64/iso-dvd/debian-12.5.0-amd64-DVD-1.iso
vm iso http://mirrors.ocf.berkeley.edu/fedora/fedora/linux/releases/40/Server/x86_64/iso/Fedora-Server-dvd-x86_64-40-1.14.iso
vm iso http://mirrors.ocf.berkeley.edu/ubuntu-releases/20.04/ubuntu-20.04.6-live-server-amd64.iso
vm iso http://mirrors.ocf.berkeley.edu/ubuntu-releases/22.04.4/ubuntu-22.04.4-live-server-amd64.iso
EOF

Execute Download Script¶

This process could take a while, depending on the speed of your network ingress.

sudo mkdir /srv/virt/vmm/.scripts
sudo cp /tmp/vm-bhyve-img-iso.download.sh /srv/virt/vmm/.scripts/
sudo /srv/virt/vmm/.scripts/vm-bhyve-img-iso.download.sh

Configuration Sequence Commands¶

sudo cp /usr/local/share/examples/vm-bhyve/* /srv/virt/vmm/.templates/

sudo vm switch create public
sudo vm switch add public lagg2

sudo vm create testnode
sudo vm install -f testnode FreeBSD-14.1-RELEASE-amd64-bootonly.iso

Setting up a GPU Passthrough VM¶

vm datastore add mach zfs:optane-aics/virt/mach
vm create -d mach -t linux-iommu-zvol -s 32G testnode-gpu-iommu

Using LibVirt¶

sysrc libvirtd_enable="YES"

Notes¶

The libvirt port does not come with networking configuration enabled.
The 'default' network definition is available at:
  /usr/local/share/examples/libvirt/networks/default.xml

To enable this network please do the following:
  cp /usr/local/share/examples/libvirt/networks/default.xml /usr/local/etc/libvirt/qemu/networks

To configure this network for autostart, execute the following:
  ln -s ../default.xml /usr/local/etc/libvirt/qemu/networks/autostart/default.xml