Jails for NetBSD

24 points by jaypatelani 8 hours ago on lobsters | 2 comments

Jails for NetBSD aims to bring lightweight, kernel-enforced isolation to NetBSD.

It is designed to fill the operational gap between simple chroot environments and full virtualization systems such as Xen.

The goal is to enable running multiple workloads on a single host with:

  • Strong process isolation
  • Per-jail resource control
  • Supervised runtime behavior
  • Unified lifecycle management
  • Centralized logging
  • Prometheus-compatible metrics

The system is intended to remain fully NetBSD-native. Isolation and policy enforcement are integrated into the kernel’s security framework rather than implemented in a separate runtime layer.

It does not aim to become a container platform. It does not aim to provide virtualization.

Instead, it focuses on explicit boundaries, predictable behavior, and a clear operational model that integrates naturally into NetBSD.

The implementation consists primarily of the following components:

Layer Model

  • secmodel_jail
    Kernel security model providing jail identity, policy enforcement, resource admission, and accounting.

  • jailctl
    Low-level control interface for jail lifecycle management, supervised execution, and runtime statistics export.

  • jailmgr
    Host-side orchestration layer for filesystem preparation, persistent configuration, and multi-jail operations.

Additionally, an optional inside-jail service runner (svcmgr) can be used for tightly-coupled multi-process workloads.

Jails for NetBSD focuses on practical, operator-friendly isolation.


Isolate Workloads Safely

Each jail defines a clear containment boundary.

Processes cannot inspect or signal processes in other jails.
Failures remain local to their workload.


Control Resource Usage

Apply limits per jail:

  • CPU quota
  • Memory ceilings
  • Maximum process count
  • File descriptor limits
  • Socket buffer accounting

You define predictable resource envelopes per workload.


Harden System Access

Apply security profiles to restrict system-level operations.

Administrative capabilities such as mounting filesystems or modifying host state can be controlled per jail.


Prevent Port Conflicts

Jails share the host network stack by design. No virtual network namespaces are introduced.

This keeps operational complexity low and avoids additional routing, firewalling, or interface management layers.

Listening ports can optionally be reserved per jail.

Port ownership is enforced by the kernel, preventing accidental or malicious conflicts while retaining a simple host-centric network model.


Supervise Services

Run services in foreground mode with host-side supervision.

  • Automatic restart with backoff
  • Deterministic lifecycle
  • No hidden control daemons

Centralized Logging & Metrics

  • stdout/stderr forwarded to host syslog
  • Single operational log plane
  • Runtime counters exposed for monitoring
  • Prometheus-compatible metrics export

Observability is built in, not bolted on.

This section demonstrates a minimal, reproducible workflow using jailmgr, jailctl, and NetBSD base components.

The example provisions a constrained HTTP service inside a jail, starts it, inspects runtime state, and exposes metrics.


1. Bootstrap

Prepare the host for jail operation.

This step:

  • Loads the secmodel_jail kernel module
  • Ensures it is configured for automatic loading on boot
  • Prepares base filesystem layers
  • Creates default configuration files
  • Initializes the jail data directory layout

2. Create Jail

Create a jail named web with:

  • HTTP daemon bound to port 8080
  • CPU quota
  • Memory limit
  • File descriptor limits
  • Socket buffer limits
  • Structured logging
vhost# jailmgr create -a -x '/usr/libexec/httpd -I 8080 -X -f -s /var/www/mysite' -q 200 -c 1000000 -m 536870912 -p 512 -d 8192 -s 134217728 -l medium -r 8080 -f local3 -o info -e err -t jail-web web
Created jail web
vhost#

3. Ephemeral Provisioning

Execute commands inside the jail from the host, without altering its persistent configuration.

The jail is entered temporarily, the commands are run, and control returns to the host.

vhost# jailmgr apply --ephemeral web <<'APPLY'
mkdir -p /var/www/mysite
echo "<html>Hello NetBSD!</html>" > /var/www/mysite/index.html
APPLY

4. Start All Jails

Each jail configuration contains an autostart setting that can be enabled or disabled individually. The --all option operates only on jails where autostart is active.

vhost# jailmgr start --all
Started jail web
vhost#

5. Supervision Model

When started in supervise mode, jailctl daemonizes itself on the host and becomes the parent process for the workload running inside the jail.

All processes started within the jail are descendants of the jailctl supervise process.

From the host perspective, these processes remain fully visible in the standard process table. There is no hidden runtime or separate process namespace.

Host process tree:

vhost# ps axd
....
8252 ?     Ss   0:00.00 |-- jailctl: jailctl supervise jail=web jid=3 
6488 ?     Ss   0:00.01 | `-- /usr/libexec/httpd -I 8080 -X -f -s /var/www/mysite 

Enter jail context:

vhost# jailctl exec web
vhost# whoami
root
vhost# ps axd
 PID TTY   STAT    TIME COMMAND
6488 ?     Is   0:00.01 /usr/libexec/httpd -I 8080 -X -f -s /var/www/mysite 
7117 pts/1 S    0:00.01 /bin/sh -i 
5572 pts/1 O+   0:00.00 - ps -axd 
vhost#

Inside the jail context, even the root user can only see processes belonging to that jail.

Cross-jail process visibility is denied by the kernel. The host context (jid 0) remains the only global view.


6. Runtime Statistics

Resource accounting per jail is maintained inside the kernel and exposed through the control interface.

vhost# jailctl stats
ID       NAME             CPU        PROC       FD         SOCKBUF    MEMORY     THROTTLE  
3        web              0          1          3          0          154025984  0

7. Prometheus-Compatible Metrics Endpoint

jailctl stats can emit Prometheus-compatible metrics.

The -P flag switches to Prometheus exposition format.
The -h flag prepends a minimal HTTP header.

This makes it possible to expose the metrics endpoint using only base system facilities (for example via inetd), without requiring a dedicated exporter daemon.

vhost# jailctl stats -P -h
HTTP/1.1 200 OK
Content-Type: text/plain

# TYPE jail_cpu_usage_ticks gauge
# TYPE jail_processes_current gauge
# TYPE jail_processes_max gauge
# TYPE jail_fd_current gauge
# TYPE jail_fd_max gauge
# TYPE jail_sockbuf_current gauge
# TYPE jail_sockbuf_max gauge
# TYPE jail_memory_current_bytes gauge
# TYPE jail_memory_max_bytes gauge
# TYPE jail_deny_process_total counter
# TYPE jail_deny_fd_total counter
# TYPE jail_deny_sockbuf_total counter
# TYPE jail_deny_memory_total counter
# TYPE jail_throttle_cpu_total counter
jail_cpu_usage_ticks{jid="3",name="web",root="/var/jailmgr/jails/web/root"} 0
jail_processes_current{jid="3",name="web",root="/var/jailmgr/jails/web/root"} 1
jail_processes_max{jid="3",name="web",root="/var/jailmgr/jails/web/root"} 512
jail_fd_current{jid="3",name="web",root="/var/jailmgr/jails/web/root"} 3
jail_fd_max{jid="3",name="web",root="/var/jailmgr/jails/web/root"} 8192
jail_sockbuf_current{jid="3",name="web",root="/var/jailmgr/jails/web/root"} 0
jail_sockbuf_max{jid="3",name="web",root="/var/jailmgr/jails/web/root"} 134217728
jail_memory_current_bytes{jid="3",name="web",root="/var/jailmgr/jails/web/root"} 154025984
jail_memory_max_bytes{jid="3",name="web",root="/var/jailmgr/jails/web/root"} 536870912
jail_deny_process_total{jid="3",name="web",root="/var/jailmgr/jails/web/root"} 0
jail_deny_fd_total{jid="3",name="web",root="/var/jailmgr/jails/web/root"} 0
jail_deny_sockbuf_total{jid="3",name="web",root="/var/jailmgr/jails/web/root"} 0
jail_deny_memory_total{jid="3",name="web",root="/var/jailmgr/jails/web/root"} 0
jail_throttle_cpu_total{jid="3",name="web",root="/var/jailmgr/jails/web/root"} 0

Result

  • Base-system HTTP daemon
  • Hard resource limits
  • Supervised execution
  • Ephemeral provisioning
  • Observable via structured metrics
  • No container runtime
  • No UID remapping
  • No full virtualization

This is process isolation as a disciplined, inspectable system primitive.

Jails for NetBSD is in active development.

The project provides both source code access and an experimental ISO build for evaluation and testing.


Source Code

The project is maintained as part of a dedicated source tree, currently as a development branch based on netbsd-10.

Repository:
https://github.com/MatthiasPetermann/netbsd-src/tree/feature/jails-v2-wip


Evaluation ISO (Unofficial Build)

An evaluation image based on NetBSD 10.1 (amd64) with integrated Jails support is available:

Download:
https://netbsd-jails.petermann-digital.de/downloads/NetBSD-10.1_STABLE_jailsv2.9-amd64.iso

Verify Checksum:

$ sha256sum NetBSD-10.1_STABLE_jailsv2.9-amd64.iso
57dc74a5eb65f56cb70492fe30d187891f07290c6979fac85f3b860dc9ae0f1c  NetBSD-10.1_STABLE_jailsv2.9-amd64.iso

Important Notice

This ISO is not an official NetBSD release.

It is an experimental build derived from the NetBSD 10.1 source tree with additional modifications for Jails support.

It is provided for evaluation and testing purposes.

For official NetBSD releases, please refer to:
https://www.netbsd.org/


Status

This project is currently in a technology preview stage.

It is suitable for development, evaluation, and architectural exploration. Interfaces, behavior, and internal structures may change as the design continues to evolve.

Parts of the implementation were developed with the assistance of AI-based development tools. All contributions are reviewed and integrated manually, but the project should be considered experimental.

It does not provide full virtualization guarantees and is not intended as a drop-in production replacement for established isolation or virtualization technologies.

Jails for NetBSD relates to earlier research and isolation efforts in the NetBSD ecosystem.

Project Focus Status
GAOLS (P3A, 2008) Jail-like process isolation via kauth(9) hooks Research prototype; never integrated
MULT (P5A, 2008) Resource isolation by instantiating full kernel subsystems Highly invasive research prototype
netbsd-sandbox Userland sandboxing via chroot, secmodel_sandbox, rlimits, capabilities, Lua policies Hardening tool; not a jail model
Systrace (NetBSD 2.0) / sysjail Syscall interposition–based process isolation Deprecated / removed

Jails for NetBSD focuses on jail-scoped enforcement integrated into the existing kauth security framework, without syscall interposition or full subsystem replication.

This website is operated in accordance with applicable German law.

Content is provided without warranty as to accuracy, completeness, or timeliness. External links are the responsibility of their respective operators. All content is subject to German copyright law.

NetBSD® is a registered trademark of The NetBSD Foundation, Inc.
All other product and service names are the trademarks of their respective owners and are used for identification purposes only.

For full provider information and detailed legal disclosures,
please refer to the complete Legal Notice.