Nightly update pulls for coolify/immich kept hitting transient ghcr.io
toomanyrequests errors and getting skipped, because the timer fired at
exactly 02:00:00 Europe/Amsterdam (the same instant countless other
hosts' cron/timer jobs also hit registries) and the retry loop only
spanned ~15s (3 attempts, 5/10/15s backoff) — not enough to ride out
the burst.
- Widen pull retries to 5 attempts with 20/40/60/80s backoff (~3+ min).
- Add RandomizedDelaySec=1800 to the update timer (and its install
template) so this host's pulls no longer land on the exact
top-of-hour thundering herd.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XmtksEEYSBZqUacw3qF1A9
Two opposite failure modes in `update run`:
- pulsenode was reported "updated" every night though its image never
changed: the pre-pull image-ID snapshot came back empty (the awk
fallback used \s, which mawk silently ignores), and empty != populated
was read as "changed".
- coolify was never recreated despite a new image: its nightly pull kept
hitting a transient `toomanyrequests`, and a failed pull only warned
before falling through to the comparison, which then saw no change.
Changes:
- _stack_image_ids: POSIX [[:space:]] instead of \s; return non-zero when
images can't be resolved so callers can tell "unknown" from "unchanged".
- Retry `docker compose pull` 3x with backoff; if it still fails, add the
stack to a new `skipped` list and skip the recreate instead of treating
it as up to date.
- Detect changes by comparing running containers' image IDs against the
post-pull resolved config, so a transient empty read can't be mistaken
for "everything changed".
- Notify with an amber "finished with warnings" summary listing skipped
stacks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
update run silently swallowed pull errors and never logged what actually
differed, making a recurring false "updates found" for pulsenode (no
registry-side change since Aug 19) impossible to diagnose.
--no-pager causes journalctl to disable ANSI colors. Setting SYSTEMD_PAGER=cat
achieves the same effect (no interactive pager) while preserving color output.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
systemctl and journalctl use their own color detection (SYSTEMD_COLORS env
var) independently of the terminal — child processes don't always inherit
it. Force colors for status and logs subcommands of update and backup.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The update and backup services ran without STACKS_DIR since systemd
doesn't source ~/.env or ~/.zshrc. Capture the value at install time
and write it as Environment= in the generated unit files.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Makes stacks work like arty — discovers stacks in the current directory.
Explicit STACKS_DIR env var still takes precedence.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Drop .sh extension — scripts are now invocable directly as arty and stacks.
Remove redundant aliases from .zshrc since ~/bin is already in PATH.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
valknarandClaude Sonnet 5