system-droid: a watchdog that checks and heals my systems on its own

TL;DR

system-droid is a small Go program that launchd fires every two hours and after every wake from sleep. It walks a checklist across all my local services, keeps quiet while everything works, and messages Telegram only when something is actually broken. Once a day it sends a short all-green summary, so silence never means "unknown". And when something does break, it first tries to fix it on its own: a separate Claude (Opus) agent tracks down the cause and repairs it, calling me only if it fails.

◆ Усі системи в нормі · 10/10 · 2026-08-17 09:00

— OpenClaw / Resonance
◆ styletts2-ua (tts) — локальний український TTS :8123
◆ openclaw gateway (:18789) — ядро агента Resonance (launchd + порт)
◆ openclaw hooks — hooks-ендпоінт увімкнено

— Інфраструктура
◆ cli-proxy Claude auth — робочий Claude-токен (валідність, авто-оновлення)
◆ system-droid — сам вартовий (надсилає ці звіти)
◆ otel-block — корпоративна телеметрія заблокована у firewall
◆ vault git backup — git-копія Obsidian-сховища на GitHub

Why this exists

Over the past months I had quietly accumulated a whole fleet of local automation: an OpenClaw agent ("Resonance"), a Ukrainian StyleTTS2 voice server, an LLM proxy in front of Claude, daily note generators for Obsidian, a git copy of my vault, a firewall rule that blocks corporate telemetry. Each of these is useful exactly as long as it runs. And they always broke the same way — silently.

The morning brief simply didn't arrive, and I noticed only in the evening. The scheduler killed a process, yet its last run still looked "successful". An auth token expired, and generation could live on an emergency fallback for weeks. None of these failures announces itself: a service doesn't complain — it just disappears. I needed someone to notice for me.

How it works

The whole system is one binary plus a config.json with two kinds of checks: HTTP (the service answers /health) and command (a script exits with 0). launchd starts the droid on schedule; it runs the checks in parallel and decides whether I need to be bothered at all. Putting a new system under watch is one line in the config, no recompilation.

The core principle is economy of attention. No dashboards to open, no "all good" spam every two hours. There are exactly three kinds of messages: the daily all-green summary; "it broke — I fixed it"; and "it broke — I need you". Everything else is silence.

Self-healing

When a check fails, the droid doesn't run to me right away. First it launches a Claude agent with a restricted toolset and write access to specific directories only. Before every attempt it takes a git snapshot of all working trees, so anything the agent changes can be rolled back with one command — the snapshot hashes arrive right in the message:

🔧 system droid — полагоджено автоматично, 1 спроба (2026-08-02 07:41)

Було зламано:
• news digest freshness — застарілий дайджест: 2026-07-31 (2 дні тому)

✅ Зараз усі перевірки зелені.

↩️ Відкат змін агента (git reset --hard на знімок):
• ~/Developer/system-droid  @ f037cfb
• ~/Developer/obsidian-cron @ e98746c

And it is not a toy: the agent has genuinely regenerated a stale digest after a nighttime network outage, reloaded a wedged launchd job by itself, and correctly diagnosed that a token problem could not be fixed without me. In the message I see what broke, what the agent did, and how it ended.

What it has already caught

In seven weeks the droid has turned several silent failures — each of which could have lived for days — into messages I saw immediately:

• A killed brief generator. The macOS kernel shot the process right after a binary rebuild — the morning brief would simply never have gone out, and nobody would have noticed.
• An expired refresh token. The proxy stopped renewing its access token, and every LLM call came back 401. The droid said it plainly: this one needs a re-login — one command, two minutes.
• A placeholder instead of analysis. Generation finished "successfully", but the note contained fallback text instead of AI analysis. Checking the note's content, not the exit code, caught it on day one.
• A stuck vault backup. The git plugin kept committing locally while the push to GitHub had quietly died — the backup would have frozen, to be discovered at the worst possible moment.

What it deliberately won't fix

The most important decision in this system is not what to automate but what not to. Some checks are marked alert-only: everything that touches authentication, other systems' internals, and my own agent's sessions. An expired OAuth cannot be "fixed" by a script — a human has to reissue it in a browser. The agent is explicitly forbidden to touch tokens, restart my working services, or bend a check until it turns green.

There is also a physical limit: the healer is itself a Claude CLI, and when the network or auth goes down, it goes down with everything else. Whatever breaks the system also breaks the hands that fix it. So the honest behaviour there is not heroic retries but a clear "I need you", with instructions on exactly what to do.

The small things everything rested on

What took the most time wasn't the checks — it was the reliability of the watchdog itself. First: rebuilding a Go binary in place changes its signature, and launchd caches signing requirements — so the system simply kills the next scheduled run. A build now always ends with a job reload:

# Перезбирання бінарника на місці змінює його cdhash, а launchd кешує
# вимоги до підпису (LWCR) на момент bootstrap. Після збирання кеш
# застаріває — і наступний плановий запуск ядро вбиває (exit 9) або
# launchd відмовляється його запускати (exit 78 / EX_CONFIG).
# Тому збирання ЗАВЖДИ закінчується перезавантаженням задачі:
go build -o system-droid .
codesign --verify --strict system-droid
launchctl bootout  "gui/$(id -u)/com.oleksii.system-droid" 2>/dev/null || true
launchctl bootstrap "gui/$(id -u)" ~/Library/LaunchAgents/com.oleksii.system-droid.plist
./check-self.sh   # свіжий запуск має бути зеленим

Second: for KeepAlive services, the last run's exit code is a misleading metric. After every restart it reads SIGTERM, and a perfectly alive service looks broken. The right question is not "how did it exit" but "is it running right now":

# У KeepAlive-демона LastExitStatus після КОЖНОГО перезапуску
# ненульовий (15/SIGTERM) — тож перевірка "останній запуск успішний"
# хибно валить цілком живий сервіс. Правильне питання інше:
# чи працює він ПРЯМО ЗАРАЗ?
info=$(launchctl list "$1" 2>/dev/null) || { echo "not loaded"; exit 1; }
pid=$(printf '%s\n' "$info" | awk -F'= ' '/"PID"/{gsub(/[ ;]/,"",$2);print $2}')
[ -n "$pid" ] && [ "$pid" != "0" ] && { echo "running (pid $pid)"; exit 0; }
echo "loaded but not running"; exit 1

Third: check the exact object the system depends on. My first token check computed "issue date plus one year" — and glowed green while the real token had been returning 401 for a day:

# Календарна перевірка сяяла зеленим — річному токену ще жити й жити:
claude oauth token — OK: valid, 365d left

# А справжній робочий токен тим часом помер, і логи проксі це знали:
token refresh failed: {"error":"invalid_grant","error_description":"Refresh token expired"}
POST /v1/chat/completions → 401 "OAuth access token has expired. Re-authenticate."

And fourth: a one-cycle network blip at 3 a.m. is no reason to wake a human. The "intervention needed" alarm now fires only if the problem survives two consecutive runs. One-off failures dissolve on their own; real ones still get through — just two hours later:

// Тривога "потрібне втручання" — лише якщо перевірка провалилася
// два запуски поспіль. Нічний обрив мережі, демон посеред перезапуску,
// пробудження зі сну — все це минає само до наступного циклу,
// і будити людину через таке не можна. Успішне лікування,
// навпаки, повідомляється одразу — це добра новина, а не шум.
counts := loadHealCounts()          // ~/.system-droid-heal-state.json
for _, r := range stillFailing {
    counts[r.Name]++
}
var persistent []Result
for _, r := range stillFailing {
    if counts[r.Name] >= 2 {        // поріг ескалації
        persistent = append(persistent, r)
    }
}
// сповіщення — лише якщо len(persistent) > 0

Eyes for another agent

Besides Telegram, the droid also runs as an MCP server. That means my local agent can ask it about system health directly: instead of reading reports, I say "check that everything works" — the agent calls get_health, runs every check live and answers. Monitoring stopped being a page you look at and became a service you consult.

# Дроїд працює і як MCP-сервер — локальний агент питає його сам:
Resonance › виклич get_health у system-droid і скажи, чи все зелене

get_health →
overall: 🟢 OK — 10/10 green (2026-08-17 09:00)
🟢 openclaw gateway (:18789) — ядро агента Resonance (launchd + порт)
🟢 cli-proxy Claude auth — робочий Claude-токен (валідність, авто-оновлення)
...

Takeaways

Technically there is nothing complicated here: Go, launchd, a few shell scripts and a Telegram bot. But this simple bundle changed how the whole automation fleet feels: I stopped keeping a mental list of things to "remember to check". That is the droid's job now.

For seven weeks it has mostly been silent. Every message it did send was either good news about something already fixed, or a precise pointer to where my two minutes were needed. That is what convenience means: a system that guards other systems — and respects your silence.