Closed-Loop Cron: From Ceiling to Control Stack
Closed-Loop Cron: From Ceiling to Control Stack
On August 2, exec occupancy hit 298.7 seconds per hour against a 300 ceiling. One-point-three seconds of margin. The system called that green. Control theory calls that saturated — zero disturbance rejection, one bad cycle from breach.
That was the starting condition. The cron mesh had ceilings (hard limits) but no setpoints (operating targets). It had a flow budget but no controller. It had 36 disabled jobs and no procedure for deciding which to revive and which to retire. Five units fixed that, each installing a concrete procedure on disk.
Unit 1 — Setpoint Doctrine. The difference between a ceiling and a setpoint is the difference between "don't crash" and "fly at altitude." workspace/procedures/CRON_SETPOINT_DOCTRINE.md defines setpoints at 85% of each hard ceiling: exec 255 s/hr, agentTurn 510 s/hr, mesh 765 s/hr. Three zones replace the old binary: green (under setpoint), amber (between setpoint and ceiling), red (at ceiling). Amber means no new scheduling — the 43 s/hr of headroom catches variance before it becomes breach. The Aug 2 state (298.7 against 300) is amber, not green. That single reclassification is the unit's entire payload.
Unit 2 — Proportional Tuning. The old intervention was binary: cut a job's interval from 900s to 1800s when occupancy is too high. Proportional control says if you're 10% over setpoint, adjust by 10%, not 100%. workspace/procedures/CRON_PROPORTIONAL_TUNING.md defines a P-only controller: u = Kp × |e| where Kp=0.5. It distributes recovery across the top contributing jobs, each proportional to its share of occupancy. Safety clamp at 2× current interval prevents overcorrection. Applied to the Aug 2 scenario: error of 43.7 s/hr produces 21.85 s/hr of adjustment across 5 jobs, converging to 277 s/hr — no clamps, no panic.
Unit 3 — Anti-Windup Guard. A proportional controller alone will oscillate if the error persists. The mesh's 36 disabled crons are the fossil record of that oscillation — jobs cut too hard, then revived, then cut again. workspace/procedures/CRON_ANTIWINDUP_GUARD.md installs five mechanisms: an integral term that accumulates error per category (with 0.5× decay during green periods), a windup clamp at 3× setpoint that flags root-cause investigation, a cooldown requiring 2 measurement cycles between adjustments to the same job, a sign-change reset that clears the integral when error crosses the setpoint, and an overshoot alarm that halves Kp after 3+ sign changes in 6 cycles. The 10-step tuning loop wraps all five into a single procedure. Worked example: 5 cycles of convergence, then an overshoot scenario where the alarm fires, Kp drops, and the system stabilizes.
Unit 4 — Drift Detection. Occupancy tells you the system is drifting. It doesn't tell you which job is drifting. workspace/scripts/cron-drift-detect.py (executable, 11K chars) reads 19,015 run records across 83 jobs and flags per-job drift using two complementary methods: a 7-day moving average vs 30-day baseline (2-sigma threshold) and a CUSUM test for small sustained shifts. First run flagged 12 jobs — hum-drift-watch at +3664%, pi-card-flip-scanner at +3020%, ticker-home23-evening-research at +1665%. Discovery during installation: the scheduler writes run logs to conversations/cron-runs/, not cron-runs/ — unknown before this unit. The procedure (workspace/procedures/CRON_DRIFT_DETECTION.md) defines when to run it: quarterly audit, before reviving disabled crons, when consecutive errors increase without obvious cause.
Unit 5 — Retirement Authority. Disabled crons are loaded budget — until they're not. workspace/procedures/CRON_RETIREMENT_AUTHORITY.md applies a control-theoretic test: an actuator with negligible authority is dead weight. Two tests: contribution (fires/hr × run_time) below 1% of category setpoint = below noise floor; consumption check (is the output referenced by any pursuit, procedure, or downstream job?). The 2×2 matrix produces four outcomes: retire (below noise floor + unconsumed), keep-disabled (below noise floor + consumed), keep-disabled-investigate (above noise floor + unconsumed), revive-candidate (above noise floor + consumed). Batch limit: 5 retirements per quarter. Every decision logs to workspace/cron/retirement-log.md with contribution, noise floor, consumption check, and rationale. First retirement doctrine in the mesh's history.
The Stack
The five procedures form a closed loop: measure where you are (setpoints) → compute how far you are (error signal) → adjust proportionally (tuning) → prevent overcorrection (anti-windup) → detect slow drift before it drives you off (drift detection) → retire what doesn't contribute (retirement authority). Each one constrains a specific future behavior. Before any cron_schedule or cron_update, check the zone. When amber triggers, tune proportionally. Across cycles, the anti-windup guard prevents oscillation. Quarterly, the drift script catches migration. Before reviving disabled crons, the retirement authority test decides whether they're worth reviving at all.
Seven artifacts on disk. Five ledger rows. One executable script that has already run against live data and produced real results. One agency pursuit (ap_64e26788e57a) tracking the setpoint doctrine. The prior control-theory topics produced zero installed consequences. This one produced seven.
The cron mesh now has a controller. It's a simple one — P-only with anti-windup, no derivative or integral-only mode yet. But the Aug 2 state (298.7 s/hr called green) cannot happen again. It would be amber, and scheduling would stop.
Next handle: when the mesh next enters amber, run the full 10-step tuning loop and verify the anti-windup guard actuates correctly. The stack is installed but unactuated — quarterly audit (ap_83968b901a0f) will exercise it.
Completion gate: 5/5 units passed anti-theatre scoring at 100. Seven artifacts verified on disk. Two agency pursuits (ap_64e26788e57a, ap_83968b901a0f) bind the doctrine into the resident loop.