Split monolithic status check.sh into modular check scripts

Replace the single-file prototype with independently-addable check
scripts (checks/*.sh) that each emit JSONL rows, a shared helper lib,
and one orchestrator/renderer (render.sh) that groups rows by section
and writes the static status page. A broken check script now shows up
as a down/warn row instead of crashing the whole render.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

10 files changed, 412 insertions, 0 deletions

diff --git a/.gitattributes b/.gitattributes
new file mode 100644
index 0000000..dfdb8b7
--- /dev/null
+++ b/.gitattributes
@@ -0,0 +1 @@
+*.sh text eol=lf
diff --git a/statuspage/README.md b/statuspage/README.md
new file mode 100644
index 0000000..3c2ae85
--- /dev/null
+++ b/statuspage/README.md
@@ -0,0 +1,109 @@
+# dandokmang status console
+
+Static status page for the dandokmang homelab server. A cron job runs
+`render.sh`, which runs every check script in `checks/`, collects their
+output, and writes a static `index.html` served by Caddy on the WireGuard
+interface only.
+
+## Layout
+
+```
+statuspage/
+  checks/       one script per metric or group of related metrics
+  lib/
+    common.sh   shared helpers (svc_status, port_check, jail_status, json_line)
+  render.sh     orchestrator + HTML renderer
+  README.md
+```
+
+## The check-script contract
+
+This is the part to remember when you come back in six months to add a
+metric: **every script in `checks/` is independent. `render.sh` never
+changes to support a new check.**
+
+A check script must:
+
+- Be executable (`chmod +x`).
+- Emit **one JSON object per line** (JSONL) to stdout, and nothing else.
+- Emit no output to stdout on failure (send diagnostics to stderr if you
+  want them, but they're discarded by the renderer).
+
+Each JSON object has exactly these fields:
+
+| field     | meaning                                                          |
+|-----------|-------------------------------------------------------------------|
+| `section` | groups rows under a heading: `hardware`, `host services`, `jails` (or a new section name - unrecognized sections are rendered after the known ones, in the order first seen) |
+| `label`   | display name for the row                                          |
+| `value`   | display value; use `""` for a plain up/down check with no value   |
+| `status`  | one of `ok`, `warn`, `down` - maps to a green/yellow/red square    |
+
+Example (`checks/wireguard.sh`):
+
+```json
+{"section": "host services", "label": "wireguard (wg0)", "value": "", "status": "ok"}
+```
+
+A script can emit multiple lines - `hw.sh` emits one line each for load
+avg, uptime, free mem, zroot usage, and zpool health.
+
+Use `json_line SECTION LABEL VALUE STATUS` from `lib/common.sh` instead of
+hand-rolling the JSON - it escapes backslashes, quotes, and embedded
+newlines (e.g. a multi-line `zpool status -x` when a pool is degraded) so
+the JSONL stays one object per line no matter what a command prints.
+
+### Adding a new check
+
+1. Create `checks/<section>-<name>.sh` (or reuse an existing section
+   script if the metric belongs with it).
+2. `. "$(dirname "$0")/../lib/common.sh"` and call `json_line` for each row.
+3. `chmod +x` it.
+
+That's it - no changes to `render.sh`, no template edits. It picks up any
+executable `checks/*.sh` file automatically and renders whatever sections
+it finds.
+
+## Failure isolation
+
+`render.sh` runs each check script and handles three failure modes without
+aborting the whole render:
+
+- **Non-zero exit** -> a `down` row labeled with the script's filename under
+  a synthetic `checks` section.
+- **No output** -> a `warn` row, same treatment.
+- **Output that doesn't look like our JSONL** -> a `warn` row, same
+  treatment.
+
+So a broken check shows up as a visible red/yellow row on the page instead
+of silently disappearing or crashing the render.
+
+## Dependencies
+
+Parsing is done with plain `awk`/`sed`/`grep` against the constrained JSONL
+shape our own `json_line` helper produces - no `jq` dependency. If checks
+ever need to emit richer JSON (nested objects, arrays), the awk parser in
+`render.sh` (`jsonval()`) will need to grow or be swapped for `jq`.
+
+## Deployment
+
+Cron (as root):
+
+```
+* * * * * /usr/local/etc/statuspage/render.sh
+```
+
+Caddyfile (bind to the WireGuard interface IP only):
+
+```
+172.16.0.1:8080 {
+    root * /usr/local/www/status
+    file_server
+}
+```
+
+Reload Caddy after editing: `service caddy reload`.
+Access from a WireGuard peer: `http://172.16.0.1:8080`.
+
+`render.sh` writes to `/usr/local/www/status/index.html` by default;
+override with the `STATUSPAGE_OUT` environment variable (useful for local
+testing without root).
diff --git a/statuspage/checks/caddy.sh b/statuspage/checks/caddy.sh
new file mode 100644
index 0000000..9e88a00
--- /dev/null
+++ b/statuspage/checks/caddy.sh
@@ -0,0 +1,10 @@
+#!/bin/sh
+# checks/caddy.sh
+# Host reverse proxy service.
+set -eu
+
+DIR=$(dirname "$0")
+. "$DIR/../lib/common.sh"
+
+STATUS=$(svc_status caddy)
+json_line "host services" "caddy" "" "$STATUS"
diff --git a/statuspage/checks/ddclient.sh b/statuspage/checks/ddclient.sh
new file mode 100644
index 0000000..361d252
--- /dev/null
+++ b/statuspage/checks/ddclient.sh
@@ -0,0 +1,10 @@
+#!/bin/sh
+# checks/ddclient.sh
+# Dynamic DNS updater service.
+set -eu
+
+DIR=$(dirname "$0")
+. "$DIR/../lib/common.sh"
+
+STATUS=$(svc_status ddclient)
+json_line "host services" "ddclient" "" "$STATUS"
diff --git a/statuspage/checks/hw.sh b/statuspage/checks/hw.sh
new file mode 100644
index 0000000..bc0c669
--- /dev/null
+++ b/statuspage/checks/hw.sh
@@ -0,0 +1,42 @@
+#!/bin/sh
+# checks/hw.sh
+# Hardware/host metrics: load avg, uptime, free mem, zroot usage, zpool health.
+set -eu
+
+DIR=$(dirname "$0")
+. "$DIR/../lib/common.sh"
+
+LOAD=$(sysctl -n vm.loadavg | tr -d '{}' | sed 's/^ *//;s/ *$//')
+json_line "hardware" "load avg" "$LOAD" "ok"
+
+UPTIME=$(uptime | sed 's/.*up //;s/,.*load.*//')
+json_line "hardware" "uptime" "$UPTIME" "ok"
+
+MEM_FREE=$(sysctl -n vm.stats.vm.v_free_count)
+MEM_PAGE=$(sysctl -n hw.pagesize)
+MEM_FREE_MB=$((MEM_FREE * MEM_PAGE / 1024 / 1024))
+json_line "hardware" "free mem" "${MEM_FREE_MB} MB" "ok"
+
+DISK=$(df -h /zroot 2>/dev/null | awk 'NR==2{print $5}')
+DISK_PCT=${DISK%\%}
+if [ -z "$DISK_PCT" ]; then
+    DISK_STATUS="warn"
+    DISK="unknown"
+elif [ "$DISK_PCT" -ge 90 ]; then
+    DISK_STATUS="down"
+elif [ "$DISK_PCT" -ge 75 ]; then
+    DISK_STATUS="warn"
+else
+    DISK_STATUS="ok"
+fi
+json_line "hardware" "zroot usage" "$DISK" "$DISK_STATUS"
+
+if ZPOOL=$(zpool status -x 2>&1); then
+    case "$ZPOOL" in
+        "all pools are healthy") ZPOOL_STATUS="ok" ;;
+        *) ZPOOL_STATUS="warn" ;;
+    esac
+else
+    ZPOOL_STATUS="down"
+fi
+json_line "hardware" "zpool" "$ZPOOL" "$ZPOOL_STATUS"
diff --git a/statuspage/checks/jail-cgit.sh b/statuspage/checks/jail-cgit.sh
new file mode 100644
index 0000000..aabeacf
--- /dev/null
+++ b/statuspage/checks/jail-cgit.sh
@@ -0,0 +1,13 @@
+#!/bin/sh
+# checks/jail-cgit.sh
+# cgit VNET jail: jail-up check, then nginx-inside-jail http check.
+set -eu
+
+DIR=$(dirname "$0")
+. "$DIR/../lib/common.sh"
+
+JAIL_STATUS=$(jail_status cgit)
+json_line "jails" "cgit jail" "" "$JAIL_STATUS"
+
+HTTP_STATUS=$(port_check 192.168.100.10 80)
+json_line "jails" "  -> http :80" "" "$HTTP_STATUS"
diff --git a/statuspage/checks/jail-www.sh b/statuspage/checks/jail-www.sh
new file mode 100644
index 0000000..7b6274e
--- /dev/null
+++ b/statuspage/checks/jail-www.sh
@@ -0,0 +1,13 @@
+#!/bin/sh
+# checks/jail-www.sh
+# www VNET jail: jail-up check, then nginx-inside-jail http check.
+set -eu
+
+DIR=$(dirname "$0")
+. "$DIR/../lib/common.sh"
+
+JAIL_STATUS=$(jail_status www)
+json_line "jails" "www jail" "" "$JAIL_STATUS"
+
+HTTP_STATUS=$(port_check 192.168.100.20 80)
+json_line "jails" "  -> http :80" "" "$HTTP_STATUS"
diff --git a/statuspage/checks/wireguard.sh b/statuspage/checks/wireguard.sh
new file mode 100644
index 0000000..3a899e5
--- /dev/null
+++ b/statuspage/checks/wireguard.sh
@@ -0,0 +1,10 @@
+#!/bin/sh
+# checks/wireguard.sh
+# wg0 interface: at least one recent handshake means the tunnel is alive.
+set -eu
+
+DIR=$(dirname "$0")
+. "$DIR/../lib/common.sh"
+
+STATUS=$(wg_status wg0)
+json_line "host services" "wireguard (wg0)" "" "$STATUS"
diff --git a/statuspage/lib/common.sh b/statuspage/lib/common.sh
new file mode 100644
index 0000000..d063e07
--- /dev/null
+++ b/statuspage/lib/common.sh
@@ -0,0 +1,49 @@
+#!/bin/sh
+# lib/common.sh
+# Shared helpers for statuspage check scripts. Source this, don't execute it.
+#
+#   . "$(dirname "$0")/../lib/common.sh"
+
+# json_line SECTION LABEL VALUE STATUS
+# Emits one JSONL row on stdout. Escapes double quotes and backslashes in
+# LABEL/VALUE so a stray '"' in e.g. `zpool status` output can't break the
+# line for the renderer's parser.
+json_line() {
+    section=$1
+    label=$2
+    value=$3
+    status=$4
+
+    # Order matters: backslashes first, then quotes, then newlines (zpool
+    # status output can be multi-line when a pool is degraded) so a JSONL
+    # line always stays exactly one line.
+    label=$(printf '%s' "$label" | sed 's/\\/\\\\/g; s/"/\\"/g' | tr '\n' '\1' | sed 's/\x01/\\n/g')
+    value=$(printf '%s' "$value" | sed 's/\\/\\\\/g; s/"/\\"/g' | tr '\n' '\1' | sed 's/\x01/\\n/g')
+
+    printf '{"section": "%s", "label": "%s", "value": "%s", "status": "%s"}\n' \
+        "$section" "$label" "$value" "$status"
+}
+
+# svc_status NAME
+# Prints "ok" or "down" depending on `service NAME status`.
+svc_status() {
+    service "$1" status >/dev/null 2>&1 && echo "ok" || echo "down"
+}
+
+# wg_status IFACE
+# Prints "ok" or "down" depending on whether wg reports handshakes for IFACE.
+wg_status() {
+    wg show "$1" latest-handshakes >/dev/null 2>&1 && echo "ok" || echo "down"
+}
+
+# port_check IP PORT
+# Prints "ok" or "down" depending on TCP connect success.
+port_check() {
+    nc -z -w2 "$1" "$2" >/dev/null 2>&1 && echo "ok" || echo "down"
+}
+
+# jail_status NAME
+# Prints "ok" or "down" depending on whether the jail is running.
+jail_status() {
+    jls -j "$1" >/dev/null 2>&1 && echo "ok" || echo "down"
+}
diff --git a/statuspage/render.sh b/statuspage/render.sh
new file mode 100644
index 0000000..22df5e7
--- /dev/null
+++ b/statuspage/render.sh
@@ -0,0 +1,155 @@
+#!/bin/sh
+# statuspage/render.sh
+# Orchestrator + renderer for the dandokmang status console.
+#
+# Runs every executable script in checks/, collects their JSONL rows, groups
+# them by section, and writes a static HTML status page. A check script
+# that fails or emits garbage is reported as a down/warn row instead of
+# aborting the whole render. See README.md for the JSONL contract that
+# check scripts must follow.
+set -u
+
+SELF_DIR=$(cd "$(dirname "$0")" && pwd)
+CHECKS_DIR="$SELF_DIR/checks"
+OUT=${STATUSPAGE_OUT:-/usr/local/www/status/index.html}
+HOST=$(hostname)
+NOW=$(date "+%Y-%m-%d %H:%M:%S %Z")
+
+. "$SELF_DIR/lib/common.sh"
+
+JSONL=$(mktemp)
+AWKFILE=$(mktemp)
+trap 'rm -f "$JSONL" "$AWKFILE"' EXIT INT TERM
+
+# valid_jsonl OUTPUT
+# Cheap structural check: every non-empty line must look like one of our
+# flat {"section":.., "label":.., "value":.., "status":..} objects. This
+# isn't a general JSON validator - it doesn't need to be, since the only
+# producer of this format is json_line() in lib/common.sh.
+valid_jsonl() {
+    bad=$(printf '%s\n' "$1" | grep -v '^$' | \
+        grep -Evc '^\{.*"section"[[:space:]]*:.*"label"[[:space:]]*:.*"value"[[:space:]]*:.*"status"[[:space:]]*:.*\}$')
+    [ "$bad" -eq 0 ]
+}
+
+for script in "$CHECKS_DIR"/*.sh; do
+    [ -e "$script" ] || continue
+    [ -x "$script" ] || continue
+    name=$(basename "$script")
+
+    output=$("$script" 2>/dev/null)
+    rc=$?
+
+    if [ "$rc" -ne 0 ]; then
+        json_line "checks" "$name" "check exited with status $rc" "down"
+    elif [ -z "$output" ]; then
+        json_line "checks" "$name" "no output" "warn"
+    elif ! valid_jsonl "$output"; then
+        json_line "checks" "$name" "malformed JSONL output" "warn"
+    else
+        printf '%s\n' "$output"
+    fi
+done >"$JSONL"
+
+# Render with awk: group rows by section (fixed order hardware/host
+# services/jails, then any other section in first-seen order - e.g.
+# "checks" for broken-script rows), preserving row order within a section.
+cat >"$AWKFILE" <<'AWKEOF'
+function jsonval(line, key,    pat, start, i, c, nc, res) {
+    pat = "\"" key "\"[ \t]*:[ \t]*\""
+    if (!match(line, pat)) return ""
+    start = RSTART + RLENGTH
+    res = ""
+    for (i = start; i <= length(line); i++) {
+        c = substr(line, i, 1)
+        if (c == "\\") {
+            nc = substr(line, i + 1, 1)
+            if (nc == "n") res = res "\n"
+            else res = res nc
+            i++
+            continue
+        }
+        if (c == "\"") break
+        res = res c
+    }
+    return res
+}
+
+function htmlesc(s) {
+    gsub(/&/, "\\&amp;", s)
+    gsub(/</, "\\&lt;", s)
+    gsub(/>/, "\\&gt;", s)
+    return s
+}
+
+function sqclass(status) {
+    if (status == "ok") return "green"
+    if (status == "warn") return "yellow"
+    return "red"
+}
+
+BEGIN {
+    order[1] = "hardware"; order[2] = "host services"; order[3] = "jails"
+    norder = 3
+    for (i = 1; i <= norder; i++) known[order[i]] = 1
+}
+
+{
+    if ($0 == "") next
+    section = jsonval($0, "section")
+    label   = jsonval($0, "label")
+    value   = jsonval($0, "value")
+    status  = jsonval($0, "status")
+    if (section == "") next
+
+    if (!(section in seen)) {
+        seen[section] = 1
+        if (!(section in known)) { norder++; order[norder] = section }
+    }
+    n = ++count[section]
+    L[section, n] = label
+    V[section, n] = value
+    S[section, n] = status
+}
+
+END {
+    print "<!DOCTYPE html>"
+    print "<html>"
+    print "<head>"
+    print "<title>" htmlesc(host) " status</title>"
+    print "<meta http-equiv=\"refresh\" content=\"30\">"
+    print "<style>"
+    print "body { background:#111; color:#ddd; font-family: monospace; padding: 2em; }"
+    print "h1 { color: #6ff; margin-bottom: 0; }"
+    print "h2 { color: #9cf; border-bottom: 1px solid #333; padding-bottom: 4px; }"
+    print "table { border-collapse: collapse; width: 100%; max-width: 600px; margin-bottom: 1.5em; }"
+    print "td { padding: 6px 10px; border-bottom: 1px solid #222; }"
+    print ".sq { display:inline-block; width:12px; height:12px; border-radius:2px; margin-right:6px; vertical-align:middle; }"
+    print ".green { background:#3c3; }"
+    print ".yellow { background:#cc3; }"
+    print ".red { background:#c33; }"
+    print ".dim { color:#888; font-size: 0.85em; }"
+    print "</style>"
+    print "</head>"
+    print "<body>"
+    print "<h1>" htmlesc(host) "</h1>"
+    print "<p class=\"dim\">updated " htmlesc(now) "</p>"
+
+    for (i = 1; i <= norder; i++) {
+        sect = order[i]
+        if (!(sect in count)) continue
+        print "<h2>" htmlesc(sect) "</h2>"
+        print "<table>"
+        for (j = 1; j <= count[sect]; j++) {
+            valcell = (V[sect, j] == "") ? "" : "<td>" htmlesc(V[sect, j]) "</td>"
+            print "<tr><td><span class=\"sq " sqclass(S[sect, j]) "\"></span>" htmlesc(L[sect, j]) "</td>" valcell "</tr>"
+        }
+        print "</table>"
+    }
+
+    print "</body>"
+    print "</html>"
+}
+AWKEOF
+
+awk -v host="$HOST" -v now="$NOW" -f "$AWKFILE" "$JSONL" >"$OUT"