lima-docker-lean

Docker host on macOS: analysis and measurements

Русская версия

The goal: a Docker host on an Apple Silicon Mac that costs nothing when idle, meaning zero disk I/O, CPU as close to zero as possible, and RAM handed back to macOS when it is not needed. The host must still speak the full Docker Engine API. The project this was built for depends on testcontainers, several docker compose files and werf, so a runtime without docker.sock is not an option.

Test bench

   
Machine Apple M5 (4 performance + 6 efficiency cores), 16 GB RAM
Host macOS 26.6.2 (25G83)
VMM Lima 2.2.0, vmType: vz (Apple Virtualization.framework), 4 vCPU, 6 GiB, virtiofs, Rosetta on
Guests Alpine 3.23.4 cloud image (kernel 6.18.22 / 6.18.53) and alpine-lima std ISO v0.2.50 (Alpine 3.24.1, kernel 6.18.38)
Docker 29.5.2 from apk (cloud) / 29.8.1 static docker.com build (ISO); host CLI 29.5.2
Workloads postgres:17-alpine + pgbench, foundationdb/foundationdb:7.3.77 (ssd engine) + Python bindings 7.3.79
Date 2026-09-24

Each figure below comes from a single run, with no repetitions or error bars. Treat differences of a few percent as noise; see Limitations.

1. Runtime landscape

This section is desk research (release notes, docs, issue trackers), not a benchmark. Only Lima and Apple container were tested hands-on.

  Lima 2.2 (vz) OrbStack 2.2.3 Colima 0.10.3 Podman 6.1 (libkrun) Docker Desktop 4.92 Apple container 1.4.1
Real Docker Engine + docker.sock ✅ ✅ ✅ (it is Lima) ⚠️ API emulation ✅ ❌
Published ports on localhost ✅ guestagent ✅ ✅ ✅ ✅ ❌ for machine
Gives RAM back to macOS ❌ (#4220, #2789) ✅ “dynamic memory” (vendor claim) ❌ — partially —
Idle CPU ~1% of a core, measured below ~0.1% (vendor claim) as Lima — higher —
License Apache-2.0 proprietary, paid for commercial use MIT Apache-2.0 paid for large orgs Apache-2.0

Lima notes: 2.2.0 fixes a socket FD leak per forwarded connection (#5210, fix #5216). On a long-running instance it eventually hit RLIMIT_NOFILE and port forwarding died silently, which matters for testcontainers because it opens many connections. Lima 2.2 also dropped template:_images/alpine; templates must now name a version (_images/alpine-3.23).

2. Apple container machine as a Docker host

Tested with container 1.3.1, then re-checked against the 1.4.1 release notes. The setup was a custom Alpine image running dockerd on tcp://0.0.0.0:2375, driven through container machine.

What does not work, by design rather than by misconfiguration:

1.3.1 → 1.4.1. 1.4.0 was withdrawn. 1.4.1 brings two Containerization security fixes (symlink traversal in OCI image load; a sun_path length overflow), container clean, a richer container system status, unescaped slashes in JSON output, and Containerization 0.45.0. Nothing changes for machine networking. Open requests include user mounts (apple/container#1805, #2278) and machine stats (#1919).

3. LinuxKit / a hand-made microVM

The idea: build the guest like Apple’s vminit or Docker Desktop’s VM, with nothing but a kernel, an init and dockerd.

The closest thing Lima supports out of the box is the alpine-lima ISO, compared next.

4. Guest OS: immutable ISO vs cloud image

  ISO Cloud, stock Cloud, tuned (final template)
Empty idle, host CPU of the VZ process 0.96% of a core 1.31% 1.00–1.12%
Guest disk writes, idle, 180 s (one session) 0 B 86 KB in a 120 s window¹ 0 B
Guest RAM after boot (used) 133 MB + 76 MB tmpfs root 148 MB —
dockerd / containerd RSS 79 / 41 MB 85 / 47 MB —
limactl restart → READY ~12 s ~26 s ~14 s
Kernel command line editable ❌ baked into the ISO ✅ grub ✅ grub
Idle with FDB + Postgres running 3.88% 4.13% 3.77%
pgbench, 8 clients, tps 15,495 15,084 14,497
FDB write, keys/s 245k 246k 253k

¹ Measured across two separate limactl shell sessions, so the logins are included; not comparable to the one-session figures.

Verdict. After tuning, the two are equally quiet and equally fast. The deciding factor is the kernel command line: only the cloud image lets you set init_on_free=1, which roughly halves the host RAM the VM really holds after a load (§6). The ISO variant stays in the repo for anyone who values immutability more.

5. Idle: what costs, what does not

Disk: from ~25 KB/min to zero

The only disk writer in an idle guest is lima-guestagent. Every 10 s it logs SyncTime: system time synchronized with host (drift was 377.9ms). The drift never converges and the loop cannot be configured off (still true in Lima 2.2, where #5365 only quoted the string). About 600 B/min of log turns into ~25 KB/min of block writes through the ext4 journal. The same line also accumulates on the host in ~/.lima/<name>/ha.stderr.log (~900 KB/day, not rotated, reset on restart).

The fix is /var/log on tmpfs plus commit=60, applied in provision: mode: boot so that it happens before the guest agent opens its log. mode: system is too late, because the daemon keeps writing to the old inode. Result: 0 bytes in 180–240 s of idle. On the ISO the root is already a tmpfs, so there is no need for this.

CPU: small leaks in the cloud image

Guest wakeup profile over 30 s, empty idle, taken with bench/wakeups.sh:

  ISO Cloud, stock
arch_timer interrupts ~102/s ~157/s
lima-guestagent context switches ~22/s ~24/s
containerd ~17/s ~23/s
rcu_preempt ~1.3/s ~9.5/s
extras — log_proxy ~4/s, syslogd ~2/s, init ~2/s

Changes applied to the cloud image one by one, with the host CPU of the VZ process after each:

Step Host CPU
stock 1.31%
kernel 6.18.22 → 6.18.53 1.28%, so the kernel is not the cause
cloud-init-hotplugd off, syslogd to a ring buffer 1.27%
ttyAMA0 getty off 1.00%

The image’s inittab spawns getty on ttyAMA0, a device VZ does not have, so busybox init respawns it forever. That was the largest single drop we measured.

Also:

What remains, about 1% of one core, is mostly Go runtime timers in lima-guestagent and containerd. It cannot be tuned away with Lima’s settings. --tick (default 3 s) is only the port-polling interval.

top’s IDLEW column is cumulative since process start, so do not compare it across runs. Use a CPU-time delta (ps -o time) instead.

6. Memory: VZ never gives it back, and what to do about it

Observed. After one pgbench run the VZ process’s phys_footprint sits at the full 6 GiB. echo 3 > /proc/sys/vm/drop_caches in the guest frees 5.6 GB inside the guest, but the host footprint stays at 6158 MB. VZ has no free page reporting, and Lima does not drive the balloon.

How to measure it correctly. phys_footprint (and the Memory column in Activity Monitor) counts compressed pages at their uncompressed size, so it cannot show reclaim. For example, vmmap reported 4.9 GB of the VM’s 6 GB swapped_out (there is no swap here; this means compressed) while the footprint still showed 6 GB. The honest number is how much memory the host gets back when the VM stops: the change in free from vm_stat.

Experiment (bench/memtest.sh): pgbench + FDB load → drop_caches in the guest → 40 s of host memory pressure (memory_pressure -l warn) → limactl stop.

  Without init_on_free With init_on_free=1
Host RAM freed by stopping the VM (the VM’s real cost) 4,193 MB 2,054 MB
VM pages in the compressor → space they occupied 5,043 MB → 3,025 MB (1.7×) 4,327 MB → 238 MB (18×)
pgbench tps / FDB keys/s in the same run 15,084 / 246k 15,713 / 242k

Why it works: with init_on_free=1 the guest kernel zeroes every page it frees. The XNU compressor stores single-value pages at almost no cost, so once macOS needs memory, the VM’s free pages cost next to nothing. Page cache, however, is used memory and holds real data. The final template therefore runs a small cache-trim service that drops the clean page cache every 10 minutes, and only when the guest is idle (load1 < 0.2).

Side effects:

Without pressure macOS has no reason to reclaim anything, and nothing is lost by that.

7. Containers under load: Postgres and FoundationDB

Final template, from results/05-final-template-idle-and-load.txt:

   
pgbench -s 50, 8 clients × 60 s 14,497 tps, 0.55 ms avg latency
pgbench, 1 client × 20 s 3,331 tps, 0.30 ms avg latency
FDB, 400k × 100 B keys, 8 threads, 500 keys/txn 253k keys/s
FDB single-key commit latency p50 1.07 ms, p99 1.61 ms
FDB full range read 535k keys/s
Idle with both containers running 3.8% of a host core; ~33 KB/s of guest disk writes

8. Smaller findings

Limitations

Reproducing

limactl start --name=dev ./templates/minimal-docker.yaml && limactl restart dev
docker context create lima-dev --docker "host=unix://$HOME/.lima/dev/sock/docker.sock"
./bench/idle.sh dev 120 60        # empty idle
./bench/disk-writes.sh dev 180    # idle disk writes, one session
./bench/wakeups.sh dev 30         # who wakes the guest
./bench/load.sh dev               # FDB + Postgres: idle with containers, load, idle after
./bench/memtest.sh dev            # real RAM cost after load (stops the VM, applies host memory pressure)

The raw outputs of the runs quoted above are in results/.