High performance policy routed IPsec mesh SD-WAN

16 points by ysun a day ago on lobsters | 3 comments

              ..::----::..
          .:-++++++++++++++-:.
        :=+++-:..      ..:-+++=:
      :=++-.                .-+-.
    .=++-.       ::::::       .   .::.
   :++=.     :+#@@@@@@@@#+:      *@@@@=
  :++=     -#@@%*+-::-+*%@@#-   :@@@@@@
 .++=    .*@@#-          -#@@*. .#@@@@*
 -++.   .#@@=              =@@#.  -++:
.++-    *@@-      :++:      -@@*
:++.   .@@#     -%@@@@%-     #@@.   .--:
-++    .++:     %@@@@@@%     +@@:    ++-
-++             %@@@@@@%     +@@:    ++-
:++.    .=+-.   -%@@@@%-     #@@.   .++:
.++-   :%@@@%:    :++:      -@@*    -++.
 -++.  =@@@@@=             =@@#.   .++-
 .++=   +%@%+            -#@@*.    =++.
  :++=         =*+-::-+*%@@#-     =++:
   :++=.      .#@@@@@@@@#+:     .=++:
    .=++-.       ::::::       .-++=.
      :=++-.                .-++=:
        :=+++-:..      ..:-+++=:
          .:-++++++++++++++-:.
              ..::----::..

Why ranet3Protocols and routing

Why ranet3

Standard protocols

Peers only need standard IKEv2, ESP, Babel and SRv6.

Interoperability is checked against:

strongSwan: IKEv2 and ESP sessions with Ed25519 keys
BIRD: Babel routes with RTT metrics, matched route for route
Linux: Kernel seg6local acts on the SRv6 headers ranet3 writes

Any other router or switch that supports IKEv2 with Ed25519 keys, Babel and SRv6 should interoperate as well.

Common mesh VPN and SD-WAN tools (Tailscale, NetBird, ZeroTier, Nebula, etc.) admit only nodes that run their own agent, and ranet3 runs beside them on the same host.

Latency routing

Babel measures the round trip of every link and routes through a relay whenever the direct link is slower.

Chosen path through B, 80 msDirect link, 265 ms

RTT is measured continuously, and packet loss and outages count against a link. Traffic always takes the fastest path, moving back to the direct link once it is faster again.

Segment routing

A policy on source and destination prefix sends traffic through chosen waypoints and out at a chosen exit. Segment routing is implemented in userspace and works even on macOS, whose XNU kernel has no support for it.

Waypoints can come in any order the user specifies and may repeat, e.g. A->B->C->B->D with D as the exit.

BenchmarksMeasured on 2026-10-15

Benchmarks

95%

of the 10 GbE payload limit, 1 stream one way

8.39 of 8.84 Gbit/s

1.6-2.1x

kernel WireGuard throughput

All 6 tests

1.2-1.5x

wireguard-go throughput

All 6 tests

1.91-4.96 ms

RTT p99 under saturation

Kernel dataplanes 0.85-2.51 ms

Medians of 3 runs under saturation

Head to head

Each range is ranet3's result divided by the other dataplane's across the 6 tests below.

Kernel and userspace

Userspace dataplanes copy every packet into their own process and are usually expected to be slower than kernel ones.

* Plain TCP without tunnel

Kernel WireGuard stays at 5.1 to 5.4 Gbit/s one way.

Kernel IPsec stays at 4.0 to 4.7 Gbit/s because its single flow lands on 1 NIC receive queue.

Full results

Throughput, saturatedGbit/s receivedHigher is better

CPU cost at 3 Gbit/s offeredBusy cores per Gbit/s, both hostsLower is better

In the "both" rows kernel IPsec carried only 4.67 and 4.14 of the 6 Gbit/s offered.

Latency under saturationRTT p99, msLower is better

SetupHardware, software and method

Setup

Hardware

Hosts: 2 x Dell PowerEdge C6420
Processor: Intel Xeon Gold 6130 with 16 cores and 32 threads
Memory: 192 GiB
NIC: Intel X710
Connector: SFP+
Bandwidth: 10 Gbit/s each way
Switch: Both hosts on the same Dell S5296F-ON

Software

OS: Debian 13
Kernel: Linux 6.12.111
wireguard-go: 0.0.20250522
wireguard-tools: 1.0.20260223
iproute2: 7.1.0
iperf3: 3.21

Method

Traffic: TCP with cubic at inner MTU 1400
Streams: 1 and 8
Load: Saturated for throughput and latency and 3 Gbit/s each way for CPU cost
Direction: One way in each direction and both ways at once
Runs: Median of 3 runs of 10 seconds with the dataplanes interleaved
CPU cost: Busy cores from /proc/stat summed over both hosts with every dataplane unpinned
Line limit: 8.84 Gbit/s of TCP payload one way after ESP in UDP framing and 18.76 Gbit/s both ways without a tunnel