funman300/OpenFUT
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ops(systemd): follow the anchor container's netns across recreation

Browse Source 
Recreating the Docker anchor left the supervised Core and host stranded in
the dead namespace while systemd still reported them active — serving
nobody, invisible to any monitoring that trusts unit state. Reproduced on
staging: netns 4026539938 -> 4026540033, both pids unchanged in the old
one, both units "active", traffic ConnectionResetError.

There is no systemd-native edge signal to bind to. Containers do appear as
units, but the scope name embeds the container ID (docker-<id>.scope), which
changes on every recreate, so BindsTo= has no stable target;
NetworkNamespacePath= resolves once at start; a .path unit on /run/netns
would watch the file this tooling maintains. So: a level-triggered reconcile
on a 10s timer, comparing the namespace the services are ACTUALLY in against
the anchor's CURRENT one, acting only on a real difference. That cannot miss
an event while the watcher restarts or dockerd is down, and needs no
debounce — a burst of three recreations produced exactly one rebind. The
trigger stays separable: a docker-events unit could invoke the same script.

Anchor absent stops the dependants rather than falling back to host
networking; docker unavailable logs once and retries on the next tick.

Two defects found while testing and fixed here:
- mount --bind STACKS when the old mount is busy, silently leaking nsfs
  entries; the bind helper now drains stale mounts in a loop.
- reconcile must stop -> rebind -> start, not rebind -> restart: a running
  service holds the old namespace open and makes the umount fail busy.

Staging also gained a faithful anchor container so the reproduction is
structural rather than mocked. Economy state was byte-identical across every
lifecycle test. Production units are templates only and remain uninstalled.
This commit is contained in:
funman300
2026-08-22 21:14:23 +00:00
parent 8ae432223a
commit 1e8d46b258
11 changed files with 409 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/systemd/README.md
+28 -11
View File
@@ -6,13 +6,18 @@ persistence and no supervisor-visible logs.
| file | scope | installed? |
|---|---|---|
| `openfut-staging-core.service` | staging Core, port 18081 | **yes** — proving ground |
| `openfut-staging-netns.service` | binds the staging anchor netns | **yes** — proving ground |
| `openfut-staging-core.service` | staging Core, port 18081 | **yes** |
| `openfut-staging-host.service` | staging host, port 8299 | **yes** |
| `openfut-netns.service` | publishes the container netns to `/run/netns/openfut` | template only |
| `openfut-staging-netns-reconcile.{service,timer}` | staging netns lifecycle | **yes** |
| `openfut-netns.service` | binds the production anchor netns to `/run/netns/openfut` | template only |
| `openfut-core.service` | production Core, port 18080 | template only |
| `openfut-host.service` | production host, port 8099 | template only |
| `openfut-netns-bind.sh` | resolves the container netns by NAME, idempotently | helper |
| `openfut-netns-reconcile.{service,timer}` | production netns lifecycle | template only |
| `openfut-netns-bind.sh` | resolves the anchor netns by NAME, idempotently, drains stale mounts | helper |
| `openfut-netns-reconcile.sh` | keeps services in the anchor's CURRENT netns | helper |
| `openfut-wait-tcp.sh` | bounded readiness gate | helper |
| `openfut-supervision-install.sh` | install / start / status / uninstall per environment | helper |
Production templates are **not installed**. Deploy only via the plan in
`OpenFUT-Vault/06 Operations/OpenFUT Service Supervision (staging-proven).md`.
@@ -42,20 +47,32 @@ interfaces, silently serving nobody. `openfut-netns-bind.sh` re-resolves by name
and refreshes a stale bind mount; `NetworkNamespacePath=` then enters it
declaratively.
> **Still open:** a container restart strands *already-running* services in the
> dead namespace. Re-running `openfut-netns.service` plus restarting Core/host
> repairs it, but nothing triggers that automatically yet. See the promotion
> plan's "Residual gap".
**4. Stale namespaces are repaired automatically (this closes the old gap).**
Recreating the anchor strands already-running services in the dead namespace,
and — the dangerous part — *systemd still reports them `active`*. Measured before
the fix: anchor `net:[4026539938] → net:[4026540033]`, Core and host unchanged in
the old one, both units `active`, traffic `ConnectionResetError`.
`openfut-netns-reconcile.sh` on a 10s timer compares the namespace the services
are **actually in** against the anchor's **current** one and, only on a real
difference, performs one stop → rebind → start cycle. It is level-triggered, so
it cannot miss an event and needs no debounce: a burst of three back-to-back
recreations produced exactly **one** rebind.
## Operating
```bash
systemctl status openfut-staging-core openfut-staging-host
journalctl -u openfut-staging-core -f
sudo systemctl restart openfut-staging-core # host survives and recovers
sudo systemctl stop openfut-staging-host openfut-staging-core
sudo ./openfut-supervision-install.sh status staging # units, anchor, netns agreement, mounts
journalctl -u openfut-staging-netns-reconcile -f # silent unless it acts
sudo systemctl restart openfut-staging-core # host survives and recovers
sudo ./openfut-supervision-install.sh install staging # copy + enable (does not start)
sudo ./openfut-supervision-install.sh uninstall staging # disable, stop, remove units
```
`status` is the one command worth knowing: it prints each unit's state, the
anchor's current netns, the namespace each service is *actually* in with an
`ok`/`MISMATCH` verdict, and the mount count (`1` healthy, `>1` a leaked stack).
Config lives in `EnvironmentFile`s (`…/systemd/core.env`, `host.env`), generated
from the live process environment so supervision changed *how* the processes
start and nothing about *what* they do. Binaries are immutable copies, so a