# infra/ — deployed infrastructure as code

> **Note (2026-07-30):** the legacy helper scripts this file references —
> `bin/ahel-agent-bootstrap`, `bin/ahel-agent-shadow`, `bin/ahel-claude-shim-credentials`,
> `bin/ahel-openclaw-credentials`, `bin/arkenstone-confirmation-key` — were deleted from the
> repo (zero code consumers; the platform runner installs via the orchestrator-served
> install.sh instead). Sections mentioning them describe machines as they were provisioned;
> recover the scripts from git history if a machine-side teardown ever needs them.

Canonical copies of every config deployed on one of the compute assets
(M1, M4a, M4b, Pi, MacBook) or on the box.
**Policy: edit here first, then deploy to the node.** If a file here and the
deployed copy disagree, the deployed copy is what's running — reconcile before
changing anything.

**Scope note (2026-07-29).** This directory covers the *local machines*. Arkenstone's own
manifests — the orchestrator Deployment, OpenViking, Postgres, JetStream, backups — live in the
arkenstone repo under `infra/k8s/`, and deploy through that repo's `arkenstone-deploy.sh`. Nothing here
deploys the orchestrator.

**Naming note (2026-07-29 purge, refined).** Only Ahel *component* names (product keys, UI labels,
deployed cloud identifiers) ban the old placeholder spelling; **lab-internal identifiers may stay
`erebor`**. Concretely: the host aliases (`ahel-m1`), the `~/.config/ahel/` and `~/.ahel/` paths,
`~/dev/ahel` (M1's own checkout path), and the `com.ahel.alloy` launchd label below are post-purge
names (compat symlinks on the machines keep both resolving), while the **`erebor_*` metric names and
the ntfy `erebor` topic stay as deployed** — they are lab wire values emitted by root daemons that
are not being redeployed, and a renamed query against an un-renamed emitter silently returns empty.
The migration is catalogued in `../../WORKSPACE.md` → "Legacy identifiers".

**Two different things are called `ahel-agent`.** The rows below for `agent/` are the *legacy* node
reporter and its M1 coordinator (Go, in this repo under `agent/`, shadow-installed only). The
platform runner that enrolls with Arkenstone, heartbeats and executes signed jobs is a **separate
repo**, `~/dev/ahel/ahel-agent`, and ships its own installers under its `install/`. Do not confuse
them when reading a plist.

**The pre-rename orchestrator identifiers are gone.** The 2026-07-29 fresh redeploy replaced the old
deployment wholesale — k8s namespace `arkenstone-orchestrator`, orchestrator NodePort 30040,
OpenViking NodePort 31940, laptop Keychain `arkenstone-token` — and this repo's asset-agent
templates were renamed to `ahel-asset-agent.sh` / `com.ahel.ahel-asset.plist` (parked, not
installed; the "Add machine" wizard onboards machines). The banned word survives only quoted in
dated history under `docs/superpowers/` and `docs/plans/`.

| Path | Deployed at | Deploy command |
|---|---|---|
| `macbook/launchd/com.ahel.openviking.plist` (RETIRED 2026-07-22 — knowledge moved to M1 realms; plist renamed *.retired) | `~/Library/LaunchAgents/` (MacBook) | `cp` + `launchctl bootstrap gui/501 <plist>` |
| `macbook/launchd/com.ahel.ov-summarizer.plist` (RETIRED — superseded by llama-swap, then the whole 4B path retired 2026-07-22) | `~/Library/LaunchAgents/` (MacBook) | same |
| `macbook/launchd/com.ahel.viking-tool.plist` (RETIRED 2026-07-22 — M1 runs viking-tool :8093) | `~/Library/LaunchAgents/` (MacBook) | same (runs `services/viking-tool/server.py`) |
| `macbook/launchd/com.ahel.subshim.plist` | `~/Library/LaunchAgents/` (MacBook, :8087, runs `services/subshim/server.py`) | `cp` + bootstrap gui/501 — OpenAI-compat shim for the claude/codex CLIs; auth held by the CLIs, runtime data in `~/.config/ahel/subshim` (not repo) |
| `m1/com.ahel.viking-fwd.plist` (RETIRED — M1-native viking-tool replaced the MacBook relay) | ahel-m1 `~/Library/LaunchAgents/` | scp + bootstrap on m1 |
| `m1/com.ahel.arkenstone-bridge.plist` | ahel-m1 `~/Library/LaunchAgents/` (:8095, runs `services/arkenstone-bridge/server.py` from `~/dev/ahel`) | scp both + `launchctl bootstrap gui/501` — bearer token in M1 `~/.arkenstone-bridge/env` (0600) + MacBook keychain `arkenstone-bridge-token`; confirmation verification key at M1 `/Users/admin/.arkenstone-bridge/confirmation.key` (0600), persistent state at `/Users/admin/.arkenstone-bridge/state.sqlite3` with database/WAL/SHM mode 0600 in a 0700 directory, and fixed Kubernetes context `contabo-arkenstone-prod-k8s` |
| `m1/com.ahel.claude-shim.plist` + `m1/claude-shim-run.sh` | ahel-m1 `~/Library/LaunchAgents/` + `~/dev/ahel/infra/m1/` (GUI-session LaunchAgent, fixed `192.168.8.168:8091`, runs `services/claude-shim/server.py`) | install the runner at its source-matching target, copy the plist, then `launchctl bootstrap gui/501` — dedicated token paths are provisioned separately by `bin/ahel-claude-shim-credentials` |
| `m1/com.ahel.arkenstone-pf.plist` + `m1/ahel-arkenstone-pf.sh` | ahel-m1 `/Library/LaunchDaemons/` + `~/dev/ahel/infra/m1/` (root daemon; kubectl port-forwards loopback 8020→svc/arkenstone-api, 9200→svc/opensearch) | scp + `sudo launchctl bootstrap system` |
| `m4a/pool-main.sh` | m4a `/usr/local/bin/pool-main.sh` (root) | scp via m1 + `sudo install` + `sudo launchctl kickstart -k system/com.ahel.pool-main` |
| `k8s/prometheus.yml` (Pi k3s legacy — deprecated, superseded by the systemd monitoring stack on the Pi) | Pi k3s `monitoring/prometheus-config` ConfigMap | `kubectl create cm prometheus-config --from-file=prometheus.yml -o yaml --dry-run=client \| kubectl apply -f -` + rollout restart |
| `k8s/chat-autologin-middleware.yaml` (Pi k3s legacy — deprecated, superseded by the systemd monitoring stack on the Pi) | Pi k3s `monitoring` ns (traefik Middleware) | `kubectl apply -f` |
| `k8s/homepage-config.yaml` (Pi k3s legacy — deprecated, superseded by the systemd monitoring stack on the Pi) | Pi k3s `default/homepage-config` ConfigMap | `kubectl apply -f` + rollout restart homepage |
| `grafana/build_dashboard.py` (RETIRED 2026-07-22 with the Grafana container; kept for un-retirement) | Grafana `ahel-overview` dashboard | `python3 build_dashboard.py` (POSTs via API; default `http://localhost:3000` anonymous — override with `GRAFANA_URL` / `GRAFANA_AUTH`) |
| `m1/prometheus.yml` | m1 `~/.ahel/prometheus.yml` | scp + `sudo launchctl kickstart -k system/com.ahel.prometheus` |
| `m1/com.ahel.prometheus.plist` | m1 `/Library/LaunchDaemons/` (ROOT daemon — macOS Local Network privacy blocks LAN dials from gui agents) | scp + `sudo launchctl bootstrap system` |
| `m1/com.ahel.node-exporter.plist` | m1 `~/Library/LaunchAgents/` | scp + bootstrap gui/501 |
| `m1/com.ahel.pushgateway.plist` | m1 `~/Library/LaunchAgents/` (binary from GitHub release → `/opt/homebrew/bin/pushgateway`) | scp + bootstrap gui/501 |
| `m1/ahel-speedtest.sh` | m1 `~/.ahel/ahel-speedtest.sh` (librespeed-cli from GitHub release; ex-Pi systemd timer, disabled there) | scp + chmod +x |
| `m1/com.ahel.speedtest.plist` | m1 `~/Library/LaunchAgents/` (30-min interval) | scp + bootstrap gui/501 |
| `m4a/com.ahel.node-exporter-full.plist` | m4a AND m4b `/Library/LaunchDaemons/` (:9102; root:wheel 644; GOTCHA: `bootstrap system` fails with error 5 on the m4s — use legacy `sudo launchctl load -w`) | scp via m1 + load |
| `macbook/ahel-grafana/` (Grafana container RETIRED 2026-07-22 — native Monitoring tab replaced it; compose kept for revert) | MacBook (in place) | `docker compose up -d` |
| `m1/bifrost-config.json` | m1 `~/.bifrost/config.json` | scp, then `launchctl bootout gui/501/com.ahel.bifrost`, `rm ~/.bifrost/config.db*`, `launchctl bootstrap gui/501 ~/Library/LaunchAgents/com.ahel.bifrost.plist` (json is imported into config.db only on fresh import). Key values are `env.*` refs resolved from `~/.bifrost/bifrost.env` (0600, NOT in repo) at request time — a key change needs only env edit + restart, no db wipe. ALWAYS verify a pool completion after restart. |
| `m1/bifrost-run-bifrost.sh` | m1 `~/.bifrost/run-bifrost.sh` | scp + chmod +x + restart agent |
| `macbook/omp/` (models.yml, config.yml, mcp.json, APPEND_SYSTEM.md) | MacBook `~/.omp/agent/` (omp binary at `~/.local/bin/omp`) | `cp <file> ~/.omp/agent/` — see `macbook/omp/README.md` |
| `macbook/omp/extensions/` (viking.ts, browse.ts) | MacBook `~/.omp/agent/extensions/` (omp auto-discovers `*.ts` there) | `cp <file> ~/.omp/agent/extensions/` |
| `macbook/launchd/com.ahel.viking-embed-tunnel.plist` (RETIRED 2026-07-22 with the MacBook OpenViking) | `~/Library/LaunchAgents/` (MacBook). ssh -L `127.0.0.1:18082` → m4b `:8082` via m1; works around macOS Local Network privacy denying LAN dials from the launchd-run openviking-server (homebrew Python; Apple-signed ssh is exempt). `~/.openviking/ov.conf` embedding api_base points at the tunnel. Retire if Python gets a Local Network grant. | `cp` + `launchctl bootstrap gui/501 <plist>` |
| `m1/ahel-asset-agent.sh` + `m1/com.ahel.ahel-asset.plist` | **PARKED TEMPLATE — NOT INSTALLED.** The fresh 2026-07-29 deployment starts with an empty asset registry; machines onboard through the in-app "Add machine" wizard. If the wizard installs it: ahel-m1 `/Users/admin/.ahel/ahel-asset-agent.sh` + `~/Library/LaunchAgents/` (gui/501 agent; 60s outbound register/heartbeat of M1 as a compute asset). **Interim** — the Go `ahel-agent` runner replaces it. | do NOT deploy by hand — bearer would live at M1 `~/.config/ahel/arkenstone-token` (0600, piped over ssh stdin from the MacBook keychain `arkenstone-token`, never echoed). GOTCHA: pool model ids come from the pool upstream `192.168.8.178:8080` (Bifrost `/v1/models` never lists custom providers) |
| `agent/com.ahel.agent-node.plist` (LEGACY node reporter — not the platform runner) | MacBook, m4a, and m4b `/Library/LaunchDaemons/` (shadow-only root daemon) | `bin/ahel-agent-shadow install` |
| `agent/com.ahel.agent-coordinator.plist` (LEGACY coordinator, M1 :8443) | ahel-m1 `/Library/LaunchDaemons/` (shadow-only root daemon) | `bin/ahel-agent-shadow install` |
| `pi/ahel-agent.service` (LEGACY node reporter on the Pi) | Pi `/etc/systemd/system/ahel-agent.service` | scp + `systemctl enable --now` |

## OpenClaw (RETIRED from the product 2026-07-29)

OpenClaw was removed from Ahel on 2026-07-29. Arkenstone's agent registry, task queue and the
`ahel-agent` runner replaced it; the Pi keeps only its monitoring role. `pi/openclaw/` and its
service files are kept **as the rollback path** and are not deployed to by any current procedure.
`bin/ahel-openclaw-credentials` is likewise retained, unrun.

The historical deployment note, kept for that rollback: the checked-in `pi/openclaw/openclaw.json` is
sanitized. A deployment would need a temporary host-local transform replacing `__GATEWAY_TOKEN__`,
`__CLAUDE_SHIM_TOKEN__`, and `__WEBHOOK_TOKEN__` from separate on-host credential files
(`gateway.token`, `claude-shim.token`, `openclaw-webhook.token`) without printing any value — write
the transformed file as mode `0600`, validate it, and remove the temporary file before the Gateway
restart. Never substitute via command-line arguments, logs, or shell tracing.

Not mirrored here (lives only on its node, intentionally):
- M1: fleet-tools (`~/.fleet-tools/`, the deployed service name), authproxy, cftunnel.
  (ahel-controller was RETIRED 2026-07-20: LaunchAgent booted out, plist renamed
  `*.retired`, installed code/data left on M1; its `controller/` source was removed
  from this repo — recover from git history if ever needed.)
- m4b: `/usr/local/etc/llama-swap.yaml` (embed/rerank), pool-worker.sh, qdrant launchd
- Pi k3s: grafana/open-webui/searxng/kuma/seaweed deployments (created 2026-07-19,
  no manifests in repo yet — export with `kubectl get deploy <x> -o yaml` if needed)
- Secrets: NEVER in this repo. Browserbase key `~/.config/ahel/browserbase.env` (MacBook),
  the fleet-tools sudo password in M1 `~/.fleet-tools/fleet.env`, OpenViking config `~/.openviking/ov.conf`.

The confirmation key is provisioned only with `bin/arkenstone-confirmation-key`, which supports
non-rotating `status`, `provision`, and `sync-existing` operations. Through `bin/ahel-secrets`,
it stores the Mac copy as Keychain service `ahel.arkenstone-confirmation-key` (helper name
`arkenstone-confirmation-key`) and installs the same base64 value into the M1 path
above without printing it. Running either write operation, copying the bridge or plist to M1,
creating the live state database, or changing launchd state requires a new exact execution-time
approval naming that operation and target; editing these canonical files is not deployment approval.

## Legacy node-agent shadow tooling

(The Go platform runner in `~/dev/ahel/ahel-agent` has nothing to do with this section.)

Bootstrap the legacy node-agent TLS and token material once with `bin/ahel-agent-bootstrap`; it refuses
to run if any target credential, CA, TLS, or operator-token target already exists. Then use
`bin/ahel-agent-shadow install`, `status`, `compare`, and `uninstall-shadow` for the code-only
shadow rollout and recovery path. `install` provisions only the binary, config, and new launchd
labels; `uninstall-shadow` only boots out those new labels and leaves binary/config/credentials/TLS/logs
recoverable in place. These scripts are committed unexecuted and unverified until the operator runs them.

## M1 Claude subscription shim

The source-controlled service consists of `services/claude-shim/server.py`,
`infra/m1/claude-shim-run.sh`, and `infra/m1/com.ahel.claude-shim.plist`.
Their M1 targets are `/Users/admin/dev/ahel/services/claude-shim/server.py`,
`/Users/admin/dev/ahel/infra/m1/claude-shim-run.sh`, and
`/Users/admin/Library/LaunchAgents/com.ahel.claude-shim.plist`. The LaunchAgent
must be bootstrapped in `gui/501` so the Claude CLI uses the admin GUI session;
the authenticated HTTP service listens only on `192.168.8.168:8091`.
Before starting `uv`, the runner creates the fixed
`/Users/admin/.config/ahel/claude-shim/workdir` at `0700` when absent. It
rejects a symlink, non-directory, or directory not owned by the LaunchAgent's
current user, and normalizes an owned directory to `0700`. Validation walks
each fixed component from `/Users/admin` with no-follow directory descriptors;
creation and chmod are descriptor-relative, followed by a final path/descriptor
identity check immediately before `uv` starts.

Run `bin/ahel-claude-shim-credentials` once from the MacBook before service
bootstrap. It generates one token and installs owner-only `0600` copies at
`/Users/admin/.config/ahel/claude-shim/token` on M1 and
`/home/admin/.openclaw/claude-shim.token` on the Pi. It refuses to replace
either path, including if a target appears after preflight. Final creation is
atomic and owner-only; if the second host fails, the helper removes the first
copy only while it still matches the token from that run. It does not print the
token and does not start or reconfigure a service. The plist and runner contain
paths only, never credential values.

For rollback, run
`launchctl bootout gui/501/com.ahel.claude-shim` on M1. Leave the two token
files and checked-in deployment files in place for a recoverable rollback; any
credential removal or replacement is a separate explicit operator action.
