Setting up a FreeBSD Virtualization Host

References

Minimal Setup Requirements

Install Packages

Minimally, we need the Bhyve packages. Optionally, qemu and libvirt related packages.

sudo pkg inst qemu-tools qemu edk2-qemu-x64 aqemu
sudo pkg inst bhyve+ bhyve-firmware bhyve-rc-4 edk2-bhyve grub2-bhyve sysutils/vm-bhyve
sudo pkg inst libvirt libvirt-dbus

Create ZFS Dataset Structure

Note: encryption is optional

zfs create -o mountpoint=/srv/virt \
    -o utf8only=on \
    -o sharenfs=off \
    -o reservation=32M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=256K \
    -o dedup=sha256,verify \
    -o compression=zstd-2 \
    -o encryption=aes-256-gcm \
    -o keyformat=raw \
    -o keylocation=file:///root/.zfs/keys/optane-aics.zfs.key \
    optane-aics/virt || exit 1

zfs create -o mountpoint=/srv/virt/vmm \
    -o utf8only=on \
    -o sharenfs=off \
    -o reservation=32M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=256K \
    -o dedup=sha256,verify \
    -o compression=zstd-2 \
    -o encryption=aes-256-gcm \
    -o keyformat=raw \
    -o keylocation=file:///root/.zfs/keys/optane-aics.zfs.key \
    optane-aics/virt/vmm || exit 1

zfs create -o mountpoint=/srv/virt/mach \
    -o utf8only=on \
    -o sharenfs=off \
    -o reservation=32M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=256K \
    -o dedup=sha256,verify \
    -o compression=zstd-2 \
    -o encryption=aes-256-gcm \
    -o keyformat=raw \
    -o keylocation=file:///root/.zfs/keys/optane-aics.zfs.key \
    optane-aics/virt/mach || exit 1

Adding DevFS Rules

/etc/devfs.rules

[devfsrules_usb_devices=20]
add path 'ttyU*' mode 0660 user operator group dialer
add path 'cuaU*' mode 0660 user operator group dialer
add path 'ugen*' mode 0660 user operator group operator

Additions to Boot Loader Config

  1. Add the following to /boot/loader.conf or /boot/loader.conf.local
  2. This example shows a secondary GPU being set to PCIe Passthrough for dedicated VM usage.
  3. Note: if you’re specifying PCIe Passthrough features then a reboot will be required.
# Virtualization
#-----------------------------------------------------#
# Ref: man vmm(4)
# https://wiki.freebsd.org/bhyve/pci_passthru
#-----------------------------------------------------#
kern.racct.enable=1
hw.vmm.maxcpu="16"
vmm_load="YES"
nmdm_load="YES"
if_bridge_load="YES"
bridgestp_load="YES"
#kqemu_load="YES"

#-----------------------------------------------------#
# VM Passthrough GPU: RTX A4000 @ upper pcie slot
# - slot spec: gen4 x16
# - slot dev: vgapci1@pci0:24:0:0 == 24/0/0
#-----------------------------------------------------#
pptdevs="24/0/0"

Configuring VM-Bhyve

RC System Additions

vm_enable="YES"
vm_dir="zfs:optane-aics/virt/vmm"

Download FreeBSD VM Images + Linux ISOs

cat<< EOF> /tmp/vm-bhyve-img-iso.download.sh
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/amd64/Latest/FreeBSD-14.1-RELEASE-amd64-BASIC-CLOUDINIT-ufs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/amd64/Latest/FreeBSD-14.1-RELEASE-amd64-BASIC-CLOUDINIT-zfs.raw.xz

vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/aarch64/Latest/FreeBSD-14.1-RELEASE-arm64-aarch64-ufs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/aarch64/Latest/FreeBSD-14.1-RELEASE-arm64-aarch64-zfs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/aarch64/Latest/FreeBSD-14.1-RELEASE-arm64-aarch64.raw.xz

vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/amd64/Latest/FreeBSD-14.1-RELEASE-amd64-ufs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/amd64/Latest/FreeBSD-14.1-RELEASE-amd64-zfs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/amd64/Latest/FreeBSD-14.1-RELEASE-amd64.raw.xz

vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/i386/Latest/FreeBSD-14.1-RELEASE-i386-ufs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/i386/Latest/FreeBSD-14.1-RELEASE-i386-zfs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/i386/Latest/FreeBSD-14.1-RELEASE-i386.raw.xz

vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/riscv64/Latest/FreeBSD-14.1-RELEASE-riscv-riscv64-ufs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/riscv64/Latest/FreeBSD-14.1-RELEASE-riscv-riscv64-zfs.raw.xz
vm img https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/riscv64/Latest/FreeBSD-14.1-RELEASE-riscv-riscv64.raw.xz

vm iso http://mirrors.ocf.berkeley.edu/debian-cd/12.5.0/amd64/iso-dvd/debian-12.5.0-amd64-DVD-1.iso
vm iso http://mirrors.ocf.berkeley.edu/fedora/fedora/linux/releases/40/Server/x86_64/iso/Fedora-Server-dvd-x86_64-40-1.14.iso
vm iso http://mirrors.ocf.berkeley.edu/ubuntu-releases/20.04/ubuntu-20.04.6-live-server-amd64.iso
vm iso http://mirrors.ocf.berkeley.edu/ubuntu-releases/22.04.4/ubuntu-22.04.4-live-server-amd64.iso
EOF

Execute Download Script

This process could take a while, depending on the speed of your network ingress.

sudo mkdir /srv/virt/vmm/.scripts
sudo cp /tmp/vm-bhyve-img-iso.download.sh /srv/virt/vmm/.scripts/
sudo /srv/virt/vmm/.scripts/vm-bhyve-img-iso.download.sh

Configuration Sequence Commands

sudo cp /usr/local/share/examples/vm-bhyve/* /srv/virt/vmm/.templates/

sudo vm switch create public
sudo vm switch add public lagg2

sudo vm create testnode
sudo vm install -f testnode FreeBSD-14.1-RELEASE-amd64-bootonly.iso

Using CloudInit Images

Note that SSH pub key and network settings can be applied during initialization.

vm create -t alpine -m 1024 -C -k /root/.ssh/authorized_keys \
    -n "ip=192.168.87.221/24;gateway=192.168.87.1;nameservers=192.168.87.70" \
    -i nocloud_alpine-3.20.1-x86_64-uefi-cloudinit-r0.qcow2 \
    VM_NAME

Setting up a GPU Passthrough VM

vm datastore add mach zfs:optane-aics/virt/mach
vm create -d mach -t linux-iommu-zvol -s 32G testnode-gpu-iommu

Sample Config

These settings are WIP and some flags are not validated for use with grub loader. 

loader="grub"
cpu=8
memory=32G
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_dev="sparse-zvol"
disk0_name="disk0"
uefi_vars="yes"
wired_memory="yes"
bhyve_options=""
utctime="yes"
debug="false"
virt_random="true"
priority="-10"
limit_pcpu="20"
virt_console0="io.rfc1918.byhve-agent"
passthru0="24/0/0=2:0"
xhci_mouse="yes"
graphics="yes"
graphics_res="1024x768"
graphics_port="1788"
graphics_listen="172.16.17.88"
graphics_wait="no"
console="tmux"
uuid="ebc194e5-271f-11ef-af67-3cecefdcc312"
network0_mac="58:9c:fc:0a:7e:2a"

Using LibVirt

sysrc libvirtd_enable="YES"

Notes

The libvirt port does not come with networking configuration enabled.
The 'default' network definition is available at:
  /usr/local/share/examples/libvirt/networks/default.xml

To enable this network please do the following:
  cp /usr/local/share/examples/libvirt/networks/default.xml /usr/local/etc/libvirt/qemu/networks

To configure this network for autostart, execute the following:
  ln -s ../default.xml /usr/local/etc/libvirt/qemu/networks/autostart/default.xml