1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
|
# AGENTS.md
Guidance for AI coding agents working in this repository.
## What this repo is
`statuspage/` generates a static status console for a FreeBSD homelab
server (`dandokmang.com`). Modular POSIX `/bin/sh` check scripts each emit
JSONL rows; an orchestrator (`render.sh`) runs them all, groups the rows by
section, and writes a static `index.html` (dark/monospace, stoplight-grid
style) that Caddy serves over WireGuard.
**Start with `statuspage/README.md`** for the full JSONL contract between
check scripts and the renderer, the jails-are-auto-discovered-not-scripted
design, failure-isolation behavior, and the current deployment recipe
(Caddy + DNS-01 cert via Porkbun). This file only covers what that one
doesn't: workflow and environment gotchas specific to developing here.
## Homelab topology
Context that isn't tied to any one check script but explains why they're
shaped the way they are:
- FreeBSD 15.0-RELEASE, single physical homelab server (`dandokmang.com`).
- WireGuard interface `wg0`, network `172.16.0.0/24` - the status page is
reachable only from this network (see `statuspage/README.md`'s Caddy
config).
- Caddy is the host's reverse proxy, fronting the jails below and (per its
own Caddyfile, not in this repo) the status page's `status.dandokmang.com`
vhost.
- ZFS pool `zroot`.
- VNET jails on `bridge0` / `192.168.100.0/24`: `cgit` (`192.168.100.10`)
and `www` (`192.168.100.20`), each running nginx on port 80.
- Host services outside the jails: `wg0`, `caddy`, `ddclient` (dynamic DNS,
since the box is on a residential/dynamic IP - this is also why
`checks/wg-watcher.sh` exists, to catch the WAN-change-reaction daemon
dying silently).
## Where code runs vs. where it's edited
This checkout is on Windows. None of the runtime commands the check
scripts depend on (`sysctl`, `jls`, `jexec`, `pfctl`, `zpool`, `wg`,
`service`, `ntpq`) exist here - they only exist on the target FreeBSD box.
- **Validate here**: `sh -n path/to/script.sh` for syntax, and dry-run
logic by putting small shell stubs for the real commands in a temp dir
and prepending it to `PATH` (e.g. a fake `pfctl` that echoes canned
output). This is the only way to exercise check-script logic without
the real box.
- **Deploy for real**: this repo's `origin` remote is
`ssh://cgit-jail/srv/git/console.git`. The FreeBSD server has its own
clone at `~/projects/console`, with `/usr/local/etc/statuspage`
symlinked to `~/projects/console/statuspage`. `render.sh` runs there via
root cron every minute, writing to `/usr/local/www/status/index.html`.
Changes only take effect after: commit here -> push -> `git pull` on the
server. There is no way to verify a change actually works against real
system state from this machine - say so rather than claiming success.
## Committing shell scripts
`core.fileMode` is `false` on this checkout, because this Windows
filesystem doesn't reliably preserve the executable bit - `chmod +x`
followed by a plain `git add` still stages a `.sh` file as `100644`.
`git config core.hooksPath .githooks` (already set in this checkout, but
**not** carried by a fresh clone - re-run it once after cloning elsewhere)
activates a pre-commit hook that force-sets `+x` on every staged `*.sh`
file automatically. A non-`.sh` executable (rare - the hook only globs
`*.sh`) needs `git update-index --chmod=+x <file>` by hand, and so does
the hook file itself if it's ever edited (it can't fix its own bit).
## Gotchas already hit once - don't re-discover these
- **`ln -s` on a directory, tested on this Windows box, silently falls
back to a real copy instead of a symlink.** Don't trust symlink
behavior verified here; it needs confirming on the real FreeBSD box.
- **Greedy regex substring traps**: `usec` contains `sec`, so
`sed 's/.*sec = \(...\).*/.../'` will match the *last* occurrence
(`usec`), not the first. Anchor tightly (e.g. match the literal `{ sec
= ` prefix) instead of relying on `.*` to stop at the right spot.
- **`ntpq`'s `rv 0 offset` prefixes non-negative values with a literal
`+`.** A sed capture class of `[-0-9.]` silently drops the `+` and
matches empty rather than failing loudly - always include `+` alongside
`-` in numeric-capture character classes.
- **`grep -c pattern` exits `1` when the count is `0`**, even though it
prints `0`. Guard command substitutions that end in `grep -c` with
`|| var=0`, or `set -e` will kill the whole script on a legitimate
zero-count case.
- **Most check commands (`pfctl`, `jls`, `wg`, `jexec`) need root.**
`render.sh` runs as root via cron, so this isn't an issue in production,
but manual testing on the box needs `doas`/`sudo`.
## Cosmetic conventions (render.sh)
- Status values are `ok` / `warn` / `down` (green/yellow/red) or `info`
(gray, for rows that aren't a health signal - e.g. `pf.sh`'s rule
listing). Don't repurpose the health colors for non-health rows.
- The font (`statuspage/fonts/scientifica.ttf`, a bitmap-style font) only
ships Arrows, Geometric Shapes, Box Drawings, Mathematical
Operators/Symbols-A, Misc Technical/Symbols, and PUA/Powerline glyphs -
notably **not** Dingbats (✓/✗). Check glyph coverage against that list
before adding new symbols to check-script output.
## Design preference: rows should fit on one line
The user has repeatedly pushed back on wrapped/overflowing row text -
this is a standing preference, not a one-off request. When a check emits
a label or value that could be long or variable-length (rule dumps, peer
lists, anything sourced from a system command's verbose output), assume
it needs to be compressed to fit a ~380px column at 16px, not just left
to wrap. `td.label`'s hanging indent (`text-indent`/`padding-left` in
`render.sh`) exists as a fallback for when wrapping is unavoidable, not
as the primary solution - reach for compression first.
`checks/pf.sh`'s `abbreviate_rule()` is the reference example for how
this compression was actually developed, worth following the same
approach for any future long-text row:
1. First attempt was the raw `pfctl -sr`/`-sn` output verbatim - overflowed
badly, multi-line wraps looked messy even with the hanging indent.
2. Stripped tokens that are boilerplate *for this specific ruleset*
(`quick`, `flags S/SA`) rather than guessing generically - checked
against the user's real `/etc/pf.conf` first rather than assuming.
3. Swapped `in`/`out` for `→`/`←`, generalized `from X to Y` to `X → Y`,
and replaced pfctl's own ASCII `->` with the same unicode arrow for
visual consistency - all using only the confirmed-supported Arrows
block (see the font gotcha above).
4. Re-sorted the rule list by (interface, direction) for readability.
Display-only - doesn't touch pf's real evaluation order or
`quick`/first-match semantics.
5. For the state-tracking flag (`no state`/`keep state`/`modulate
state`/`synproxy state` - four modes with real, different security
properties), the instinct was to find clever unicode from Mathematical
Operators/Geometric Shapes, but landed on plain bracket-letter tags
(`[N]`/`[K]`/`[M]`/`[S]`) instead: a distinction with real operational
meaning needs to be legible without a legend, and a "clever" symbol
that requires memorizing isn't actually more compact once you factor
in "what does this mean."
6. Every step was verified against a rendered preview (Artifact tool with
mocked check output), not just eyeballed character counts - monospace-
ish bitmap font wrapping doesn't line up with raw string length in an
obvious way.
## When cron + static HTML stops being enough
This project started as a refactor of a single monolithic cron script
(one `check.sh` that shelled out to various commands and heredoc'd a
static page) into the independently-addable checks/render.sh split
described in `statuspage/README.md`. The goal was always "modular static
page," not "monitoring platform" - if a future ask starts pushing on one
of these, say so explicitly rather than bolting a workaround onto the
current design:
- **Alerting / notification on state change.** The current design is
pull-only - a human has to load the page to see anything. There's no
mechanism to push "X just went down" anywhere. That needs a real daemon
(or at minimum a separate cron job comparing renders and calling out).
- **History or trends** (e.g. "graph load average over the last day").
Each render overwrites the last; nothing persists prior states. That
needs a time-series store, which is a different project.
- **Faster-than-cron or push-based updates.** Cron's practical floor is
about a minute, and the page currently self-refreshes via `<meta
http-equiv="refresh">` (a full reload), not a live push. If sub-minute
latency or in-place updates without a full reload actually matter, that
means WebSockets/SSE and a long-running process, not this architecture.
- **Check execution time approaching the 1-minute cron interval.** This
one's an operational red flag rather than a feature request - if
`render.sh` starts taking close to 60s to run, cron ticks will begin
overlapping/queuing. Worth watching as more checks get added.
|
