FreeBSD 14 - Deploying Poudriere + CCache + NGINX on LX2160a¶

posted	revised
April 16, 2024 10:56:48	May 15, 2024 11:56:37

Basic Setup¶

Configure Latest for FreeBSD Repos¶

1. Create local repos dir
2. Copy generic quarterly config file to latest w/ naming convention in place
3. Disable generic quarterly from polling
4. Programmatically set latest config to use the desired polling
5. Update pkg repo index
6. Upgrade existing packages to latest

mkdir /usr/local/etc/pkg/repos
cp /etc/pkg/FreeBSD.conf /usr/local/etc/pkg/FreeBSD-latest.conf
echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf
sed -i '' -e 's|quarterly|latest|g' /usr/local/etc/pkg/FreeBSD-latest.conf
pkg update -f
pkg upgrade -y

Dependent Packages¶

We’ll use these at various stages, either explicitly in commands or implicitly during system administration.

pkg install -y anacron bash bash-completion beadm cmdwatch doas git git-lfs node_exporter pciutils rsync sudo tmux

Optional Packages¶

The following packages are general purpose tooling and services which are installed on most systems in my infrastructure.

pkg install -y autossh bat eza fusefs-sshfs grc htop hwloc2 hwstat ipmitool
pkg install -y ipsumdump lsof micro nmap pv tree usbutils vim

Enable and Configure Sudo¶

We’ll enable the wheel group to obtain sudo privs without a password.

\cat<< EOF> /usr/local/etc/sudoers.d/wheel
%wheel ALL=(ALL:ALL) NOPASSWD: ALL
EOF

Poudriere ZFS Datasets¶

Encrypted Dataset Support¶

Enable ZFS encryption key loading at boot.

sysrc zfskeys_enable=YES

Required Datasets¶

The following are created as separate datasets for various reasons:

1. Some datasets benefit from block level deduplication
2. Some datasets benefit from smaller or larger ZFS record sizes
3. Some datasets benefit from greater or lesser compression rates
4. Responsible dataset security (GBAC,AAA) implies dataset segmentation
5. Responsible dataset management requires block-level workload isolation
6. Advanced observability + analytics require workload segmented mount points

zfs create -o mountpoint=/usr/local/poudriere \
    -o reservation=256M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=128K \
    -o dedup=sha256,verify \
    -o compression=zstd-1 \
    zroot/poudriere || exit 11

zfs create -o mountpoint=/usr/local/poudriere/ports \
    -o reservation=256M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=128K \
    -o dedup=sha256,verify \
    -o compression=zstd-1 \
    zroot/poudriere/ports || exit 12

zfs create -o mountpoint=/usr/local/poudriere/distfiles \
    -o reservation=256M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=128K \
    -o dedup=sha256,verify \
    -o compression=zstd-1 \
    zroot/poudriere/distfiles || exit 13

zfs create -o mountpoint=/var/cache \
    -o reservation=256M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=128K \
    -o dedup=sha256,verify \
    -o compression=zstd-1 \
    zroot/var/cache || exit 14

zfs create -o mountpoint=/var/cache/ccache \
    -o reservation=256M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=64K \
    -o dedup=off \
    -o compression=zstd-1 \
    zroot/var/cache/ccache || exit 15

Network Time Improvements¶

Standard NTPd is mostly awful compared to Chrony, so we’ll install and configure that service here.

pkg inst chrony

cat<< EOF> /tmp/chrony.conf
# Chronyd config -- RFC1918.host v1.1.0 -- FreeBSD 14.0-RELEASE-p4 -- 2023-1219
#-----------------------------------------------------------------------------------------------------------------------
# This file is managed by Ansible
#-----------------------------------------------------------------------------------------------------------------------
## Use public servers from the pool.ntp.org project
pool 0.pool.ntp.org iburst minpoll 6 maxpoll 6
pool 1.pool.ntp.org iburst minpoll 6 maxpoll 6
pool 2.pool.ntp.org iburst minpoll 6 maxpoll 6
pool 3.pool.ntp.org iburst minpoll 6 maxpoll 6

## Record the rate at which the system clock gains/losses time
driftfile  /var/db/chrony/drift

## Location for NTS files
# ntsdumpdir /var/db/chrony
# keyfile /usr/local/etc/chrony.keys

## Allow system clock to be stepped in 3x updates if offset > 1s
makestep 1.0 3

## Enable kernel synchronization of the real-time clock (RTC)
rtconutc
rtcsync

## Increase the min sources required to adjust system clock
minsources 3

## Get TAI-UTC offset and leap seconds from the system tz database
leapsectz right/UTC

## Specify directory for log files
logdir /var/log/chrony
dumpdir /var/db/chrony

## Select which information is logged
logchange 0.5
log measurements statistics tracking rtc
EOF

mv /usr/local/etc/chrony.conf /usr/local/etc/chrony.conf.dist
mv /tmp/chrony.conf /usr/local/etc/chrony.conf

sysrc ntpd_enable=NO
sysrc ntpd_sync_on_start=NO
sysrc chronyd_enable=YES
service chronyd start

Poudriere Config¶

We’ll configure this here, but we’ll also have a Jinja2 template for it automation purposes. Note that automake is essential to Poudriere and without it there are many packages which will fail to build.

pkg install -y poudriere-devel automake

PARA_JOBS=$(expr $(sysctl -n hw.ncpu) - 4)
PREP_JOBS=$(expr ${PARA_JOBS} + 1)

mv /usr/local/etc/poudriere.conf /usr/local/etc/poudriere.conf.dist

cat <<\EOF >/usr/local/etc/poudriere.conf
ZPOOL=zroot
ZROOTFS=/poudriere
#
FREEBSD_HOST=https://download.freebsd.org
RESOLV_CONF=/etc/resolv.conf
#
# BASEFS=/usr/local/poudriere
BASEFS=/srv/jails/containers/poudriere
#
POUDRIERE_DATA=${BASEFS}/data
USE_PORTLINT=yes
USE_TMPFS=yes
TMPFS_BLACKLIST_TMPDIR=${BASEFS}/data/cache/tmp
#
# DISTFILES_CACHE=/usr/local/poudriere/ports/distfiles
DISTFILES_CACHE=${BASEFS}/ports/distfiles
#
GIT_BASEURL=git.freebsd.org/src.git
GIT_PORTSURL=git.freebsd.org/ports.git
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
BAD_PKGNAME_DEPS_ARE_FATAL=no
CCACHE_DIR=/var/cache/ccache
CCACHE_DIR_NON_ROOT_SAFE=yes
CCACHE_GROUP=ccache
CCACHE_GID=65531
PARALLEL_JOBS=${PARA_JOBS}
PREPARE_PARALLEL_JOBS=${PREP_JOBS}
SAVE_WRKDIR=yes
WRKDIR_ARCHIVE_FORMAT=txz
NOLINUX=yes
ALLOW_MAKE_JOBS=yes
ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py*"
TIMESTAMP_LOGS=yes
MAX_EXECUTION_TIME=86400
NOHANG_TIME=7200
ATOMIC_PACKAGE_REPOSITORY=yes
COMMIT_PACKAGES_ON_FAILURE=no
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=1
PORTTESTING_FATAL=yes
BUILDER_HOSTNAME=${HOSTNAME}.rfc1918.host
PRESERVE_TIMESTAMP=yes
BUILD_AS_NON_ROOT=yes
PORTBUILD_USER=nobody
PORTBUILD_GROUP=nobody
PORTBUILD_UID=65532
PORTBUILD_GID=65532
BUILDNAME_FORMAT="%FT%T%z"
DURATION_FORMAT="%H:%M:%S"
USE_COLORS=yes
TRIM_ORPHANED_BUILD_DEPS=yes
DELETE_UNKNOWN_FILES=yes
DELETE_UNQUEUED_PACKAGES=no
URL_BASE=http://${HOSTNAME}.rfc1918.host
HTML_TYPE="hosted"
HTML_TRACK_REMAINING=yes
DETERMINE_BUILD_FAILURE_REASON=yes
MAKEWORLDARGS="WITHOUT_LLVM_ASSERTIONS=yes WITH_MALLOC_PRODUCTION=yes -DMALLOC_PRODUCTION"
PACKAGE_FETCH_BRANCH=latest
PACKAGE_FETCH_URL=pkg+https://pkg.freebsd.org/\${ABI}
# END
EOF

Setup CCache Perms and Dirs¶

Create groups for Portbuild and CCache¶

pw groupadd portbuild -g 65532
pw useradd portbuild -u 65532 -g portbuild -d /nonexistent -s /usr/sbin/nologin
pw groupmod -n portbuild -m root

pw groupadd ccache -g 65531
pw groupmod -n ccache -m root,portbuild

Create dirs and set perms¶

# this tmpdir could be created as a tmpfs memory backed directory via fstab
#  perhaps later...
mkdir /var/cache/ccache/{debug,tmp}

find /var/cache/ccache/ -type d -exec chmod 2770 {} +
find /var/cache/ccache/ -type f -exec chmod 0660 {} +

chmod 1770 /var/cache/ccache/tmp
chmod -R u+s /var/cache/ccache
chmod -R g+s /var/cache/ccache
chown -R root:ccache /var/cache/ccache

Write CCache config files¶

echo "umask = 0002" >> /var/cache/ccache/ccache.conf

cat <<EOF >/usr/local/etc/ccache.conf
base_dir       = /var/cache/ccache
cache_dir      = /var/cache/ccache
debug_dir      = /var/cache/ccache/debug
temporary_dir  = /var/cache/ccache/tmp
EOF

Install and Configure NGINX for Web Access¶

Package additions¶

pkg inst nginx nginx-prometheus-exporter modsecurity3-nginx

Generating Crypto Requirements¶

The following OpenSSL command will take some time to complete, so you may want to run it in a different terminal pane from the others, and let it run until it’s finished.

openssl dhparam -out /usr/local/etc/nginx/dhparam.pem 4096

Service configurations¶

Enable separate group for TLS certs¶

pw group add -g 644 -M root,www -n pki

mkdir -p /usr/local/etc/pki
chown -R root:pki /usr/local/etc/pki
chmod 750 /usr/local/etc/pki

Nginx Base Config¶

mkdir /usr/local/etc/nginx/conf.d
chown -R www:www /usr/local/etc/nginx
chmod -R u+s /usr/local/etc/nginx
chmod -R g+s /usr/local/etc/nginx

# adding a log mime-type allows nginx to serve log files in the browser
sed -i '' -E 's|text/plain[\t\ ]*txt|text/plain txt log|g' /usr/local/etc/nginx/mime.types

cat <<EOF >/usr/local/etc/nginx/nginx.conf
## nginx base config for poudriere
user www;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
include /usr/local/etc/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';

  access_log          /var/log/nginx/access.log  main;
  sendfile            on;
  tcp_nopush          on;
  tcp_nodelay         on;
  keepalive_timeout   65;
  types_hash_max_size 2048;
  include             /usr/local/etc/nginx/mime.types;
  default_type        application/octet-stream;

  include /usr/local/etc/nginx/conf.d/*.conf;
}
## END
EOF

Nginx Site Config¶

The following SSL cert references are specific to this install, as it’s using a Comodo SSL Wildcard. You will want to adjust the file names accordingly to your own SSL cert files.

SRVC_DOMAIN="rfc1918.host"
SRVC_HOST="sys-rfc100-arm64-hclx2n2"
SRVC_FQDN="${SRVC_HOST}.${SRVC_DOMAIN}"

mkdir -p /usr/local/etc/pki/${SRVC_DOMAIN}

cat <<EOF >/usr/local/etc/nginx/conf.d/poudriere.conf
## nginx service config for poudriere
server {
  listen 80;
  server_name     ${SRVC_FQDN};
  access_log      /var/log/nginx/${SRVC_HOST}.access.log;
  error_log       /var/log/nginx/${SRVC_HOST}.error.log;
  location / {
    return 302 https://${SRVC_FQDN}/$request_uri;
  }
}

server {
  listen 443   ssl http2;
  server_name  ${SRVC_FQDN};
  set          $cache_uri $request_uri;
  set          $srvchost  "SRVC_HOST";

  # TLS
  resolver                   9.9.9.9 8.8.8.8 valid=300s;
  resolver_timeout           5s;
  ssl_ciphers                HIGH:!aNULL:!MD5:EECDH+AESGCM:EDH+AESGCM;
  ssl_dhparam                /usr/local/etc/nginx/dhparam.pem;
  ssl_ecdh_curve             secp384r1;
  ssl_prefer_server_ciphers  on;
  ssl_protocols              TLSv1.2 TLSv1.3;
  ssl_session_cache          shared:SSL:1m;
  ssl_session_tickets        off;
  ssl_session_timeout        10m;
  ssl_stapling               on;
  ssl_stapling_verify        on;
  ssl_certificate            /usr/local/etc/pki/${SRVC_DOMAIN}/${SRVC_DOMAIN}_ssl-comodo.ca-cert-chain.pem;
  ssl_certificate_key        /usr/local/etc/pki/${SRVC_DOMAIN}/${SRVC_DOMAIN}_ssl-comodo.key;

  # Allow gzipping js, css, log, svg and json files
  gzip on;
  gzip_buffers 16 8k;
  gzip_comp_level 6;
  gzip_http_version 1.0;
  gzip_min_length  1100;
  gzip_proxied any;
  gzip_vary on;
  gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/gif image/jpeg image/png application/json image/svg+xml;

  # Logging
  access_log           /var/log/nginx/${SRVC_HOST}.access.log;
  error_log            /var/log/nginx/${SRVC_HOST}.error.log;

  # Base html dir for poudriere site template
  root         /usr/local/share/poudriere/html;

  # Something probably useful
  location ~ /.well-known {
    allow all;
    }

  # Image file caching
  location ~* ^.+\.(jpg|jpeg|gif|png|ico|svg|woff|css|js|html)$ {
    add_header Cache-Control "public";
    expires 1d;
   }

  # Yep, packages
  location /pkg {
    autoindex on;
    alias /usr/local/poudriere/data/packages;
    }

  # WebDAV
  location /cache/ {
    # Where to store cache files
    # Enable needed HTTP methods
    # Allow creating subdirectories
    # Allow individual cache entries to be up to 100 MiB
    # Don't log 404 Not Found replies as errors
    # Enable WEBDAV reads
    dav_access user:rw group:rw all:r;
    alias /usr/local/poudriere/data/cache;
    log_not_found off;
    dav_methods PUT DELETE;
    create_full_put_path on;
    client_max_body_size 100M;
      }

    ## log data
    # Allow caching dynamic files but ensure they get rechecked
    location /data {
    alias /usr/local/poudriere/data/logs/bulk;
    location ~* ^.+\.(log|txz|tbz|bz2|gz)$ {
      add_header Cache-Control "public, must-revalidate, proxy-revalidate";
        }
      }

    # Don't log json requests as they come in frequently and ensure
    #   caching works as expected
    location ~* ^.+\.(json)$ {
      add_header Cache-Control "public, must-revalidate, proxy-revalidate";
      access_log off;
      log_not_found off;
      }

    # Allow indexing only in log dirs
    location ~ /data/?.*/(logs|latest-per-pkg)/ {
      autoindex on;
      }
    }

  ## Testing
  location /pkg/FreeBSD:14:arm64 {
    autoindex on;
    alias /usr/local/poudriere/data/packages/FBSD-140-ARM64j-FBSDp;
    }
}
## END

Enable and Start Nginx¶

sysrc nginx_enable=YES

Poudriere Operator Notes¶

Create two builder jails¶

Because reasons, we’ll create one FreeBSD 14 and one FreeBSD 15 jail… naming conventions subject to change…

poudriere jail -c -j FBSD-140-ARM64j -v 14.0-RELEASE -a arm64.aarch64
poudriere jail -c -j FBSD-150-ARM64j -v 15.0-CURRENT -a arm64.aarch64

Create port tree from git repo using main branch¶

poudriere ports -c -p FBSDp -m git+https -B main -U https://git.freebsd.org/ports.git

[1] Create port repo based pkglist¶

cd /usr/local/poudriere/ports/FBSDp
find ./ -type d -depth 2 \
  | awk -F/ '{print $2"/"$3}' \
  | sort \
  | grep -v Makefile \
  | egrep -v "Mk/"\
    |"Templates/Licenses"\
    |"Tools/scripts" \
  > /usr/local/etc/poudriere.d/FBSD-140-ports.repo-all.pkglist

[2] Create ‘installed’ query for pkglist (includes deps)¶

pkg query -e %a=0 %o | sort -h > /usr/local/etc/poudriere.d/FBSD-140-builder.query-all.pkglist

[2] Create ‘installed’ query for pkglist (without deps)¶

pkg prime-origins | sort -h > /usr/local/etc/poudriere.d/FBSD-140-builder.prime-origins.pkglist

Set options for a given port (if needed)¶

poudriere options -j JAIL-NAME -p FBSDp -n CATEGORY/PORTNAME

Set ‘make/config’ options for a package + no-alter of dependency options¶

poudriere options -p FBSDp -c -n CATEGORY/PORTNAME

Create build session with pkglist file¶

poudriere bulk -j JAIL-NAME -p FBSDp -f <pkglist file>

Create build session with all pkgs¶

poudriere bulk -j JAIL-NAME -p FBSDp -a

Poudriere Setup for HardenedBSD¶

HBSD Public Build Servers¶

Download Server for Installers¶

• Located at: https://installers.hardenedbsd.org/pub/keys/ssh.pub.txt
• PUB Key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPpi/BYqEClmw7VCmpPNrTtUgiL+Fc49qy3bgabDlf9F hbsd-os-build-01

Git Repo Server for OS Sources¶

• https://git.hardenedbsd.org/hardenedbsd/HardenedBSD.git
• Git source fork mapping: freebsd/14-stable/main -> hardened/14-stable/master

Git Repo Server for Ports¶

• https://git.hardenedbsd.org/hardenedbsd/ports.git
• Git source fork mapping: freebsd/main -> hardenedbsd/main

Jail Setup and Management¶

Create Jail via Git¶

poudriere jail -c -j HBSD-14-STABLE-GIT-AMD64j -v hardened/14-stable/master -a amd64 -m git+ssh -U git@git.hardenedbsd.org:hardenedbsd/HardenedBSD.git -K HARDENEDBSD

Create Jail via Tgz¶

Syntax for 14-STABLE¶

poudriere jail -c -j HBSD-14-STABLE-URL-AMD64j -v hardened/14-stable -a amd64 -m url=https://installers.hardenedbsd.org/pub/current/amd64/amd64/installer/LATEST/

Syntax for 13-STABLE¶

poudriere jail -c -j HBSD-13-STABLE-URL-AMD64j -v hardened/13-stable -a amd64 -m url=https://installers.hardenedbsd.org/pub/13-stable/amd64/amd64/installer/LATEST/

ARM64 Jails - Git only¶

Syntax for 14-STABLE¶

poudriere jail -c -j HBSD-14-STABLE-GIT-ARM64j -v hardened/14-stable/master -a arm64.aarch64 -m git+ssh -U git@git.hardenedbsd.org:hardenedbsd/HardenedBSD.git -K HARDENEDBSD

Ports Build Config¶

Create ports tree git+https (does not require ssh key on repo)¶

poudriere ports -c -p HBSDp -m git+https -B hardenedbsd/main -U https://git.hardenedbsd.org/hardenedbsd/ports.git

Create ports tree git+ssh (req ssh key added to repo)¶

poudriere ports -c -p HBSDp -m git+ssh -B hardenedbsd/main -U git@git.hardenedbsd.org:hardenedbsd/ports.git

Create build session with all pkgs from port repo¶

poudriere bulk -j HBSD-14-STABLE-URL-AMD64j -p HBSDp -a

(optional) Update the jail and build a selected kernel from src¶

poudriere jail -u -j HBSD-14-STABLE-GIT-AMD64j -p HBSDp -v hardened/14-stable/master -a amd64 -m git+ssh -U git@git.hardenedbsd.org:hardenedbsd/HardenedBSD.git -K HARDENEDBSD

References¶

• https://github.com/freebsd/poudriere/wiki/poudriere_in_jail