Zero-SSH Observability: What 2RUN Ops Learned Watching Five Nodes from a Raspberry Pi
Edge agents over SSH polling, Tailscale MagicDNS pitfalls, and timeout sizing: the hard-won lessons from running a five-node self-hosted mesh dashboard.
Self-hosted infrastructure keeps failing in one specific way: every node is fine until the moment you need to know it is not. This week we shipped the next iteration of 2RUN Ops, our open internal operations dashboard, and it now watches a five-node Tailscale mesh — a Raspberry Pi gateway, two Hetzner VPSes, a media-server MacBook Pro, and the Probex production cluster — from a single screen with zero SSH sessions.
Why edge agents instead of SSH polling
Classic monitoring pulls: a central server opens SSH, runs commands, parses output. It works, but it scales badly and couples your observability to credential sprawl. 2RUN Ops inverts this. A ~30-line Node.js edge agent runs on each node, collects CPU, RAM, disk, load, and Docker container state, and pushes telemetry to the hub over a WebSocket with an HTTP fallback. The hub never dials out for metrics; the nodes report in.
The practical difference shows up in latency and resilience: telemetry arrives every 2.5 seconds, survives transient WebSocket drops, and the hub degrades gracefully to cached snapshots marked stale instead of failing the whole page.
Lessons from this week's rollout
Three real incidents shaped the current build:
- Stale identities. A node reported under an old hostname after a hardware swap, and the dashboard quietly showed zeros. Telemetry identity is now an explicit, reviewed configuration, and the UI shows the MagicDNS name of the node it actually talks to.
- Slow Docker daemons. On a busy macOS Docker Desktop, listing containers can take ~3 seconds. A 2-second timeout in the collector silently returned empty container lists. Timeouts are now sized to measured reality, not optimism.
- Container DNS amnesia. Docker's daemon-level DNS override broke Tailscale MagicDNS resolution inside containers. The fix was boring and correct: give the dashboard containers an explicit resolver order that starts with the Tailscale DNS resolver.
What this means for customers
The same pattern powers how we operate client platforms: observable, self-healing, and boring. When a queue stalls or a certificate nears expiry, we want the dashboard to say so before anyone notices. 2RUN Ops is our proof that a small team can run serious observability without a SaaS bill — and the playbook transfers directly to customer projects.
Want this level of visibility on your own stack? Talk to us at 2run.be.