valknar/stacks
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: absorb _backup and _update into stacks.sh

Browse Source 
- Inline update logic (pull → compare digests → up -d → prune → notify)
- Inline backup logic with dynamic Postgres detection: any running
  <stack>_db container is dumped using the <stack>/<stack> convention
- Systemd unit files are now generated on `install` from embedded
  heredocs pointing at stacks.sh itself — no external scripts needed
- Root .env (WEBHOOK_URL, RESTIC_REPOSITORY, RESTIC_PASSWORD) replaces
  the per-service .env files in _backup/ and _update/
- Remove _backup/ and _update/ directories entirely
- Update README accordingly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Sebastian Krüger
2026-06-16 20:56:04 +02:00
parent e3cd2df372
commit fcff6f3298
11 changed files with 238 additions and 253 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+6
View File
@@ -0,0 +1,6 @@
# Webhook for update/backup notifications (n8n → Telegram)
WEBHOOK_URL=https://n8n.example.com/webhook/your-webhook-id
# Restic backup repository and password
RESTIC_REPOSITORY=/mnt/hidrive/users/valknar/Backup/stacks
RESTIC_PASSWORD=changeme
README.md
+22 -31
View File
@@ -20,11 +20,11 @@ Each stack is independently deployable with its own `compose.yml` and `.env`. Al
## Tools
| File/Directory | Description |
| File | Description |
|---|---|
| `stacks.sh` | CLI to manage stacks, services, and scaffolding |
| `_backup` | Daily restic backups to HiDrive (host script + systemd timer) |
| `_update` | Nightly image update check + prune (host script + systemd timer) |
| `stacks.sh` | CLI to manage stacks, services, scaffolding, updates, and backups |
| `.env` | Root config: `WEBHOOK_URL`, `RESTIC_REPOSITORY`, `RESTIC_PASSWORD` (gitignored) |
| `.env.example` | Template for the root `.env` |
## stacks.sh
@@ -50,16 +50,16 @@ Each stack is independently deployable with its own `compose.yml` and `.env`. Al
./stacks.sh run passbolt passbolt bin/cake passbolt healthcheck
```
**Service management:**
**Service management** (reads `WEBHOOK_URL`, `RESTIC_REPOSITORY`, `RESTIC_PASSWORD` from root `.env`):
```bash
./stacks.sh update install # link & enable systemd update timer
./stacks.sh update install # write & enable systemd update timer
./stacks.sh update run # run update now
./stacks.sh update status # show timer/service status
./stacks.sh update logs # show journal logs
./stacks.sh backup install # link & enable systemd backup timer
./stacks.sh backup run # run backup now
./stacks.sh backup install # write & enable systemd backup timer
./stacks.sh backup run # run backup now (auto-detects <stack>_db containers)
./stacks.sh backup snapshots # list restic snapshots
```
@@ -105,38 +105,29 @@ All stacks share the external `falcon_network` Docker network for inter-service
## Backup
The `_backup` stack runs a daily restic backup at 3:00 AM. It dumps all Postgres databases, then backs up the entire `.data/` directory to HiDrive. Retention: 7 daily, 4 weekly, 6 monthly snapshots. Notifications go to Telegram via n8n.
Runs daily at 3:00 AM via a systemd timer. Detects Postgres databases automatically by convention (`<stack>_db` container, user `<stack>`, database `<stack>`), dumps each one, then runs a full restic backup of `.data/`. Retention: 7 daily, 4 weekly, 6 monthly. Notifications go to Telegram via n8n.
```bash
# Deploy backup stack
rsync -avz _backup/ vps:~/stacks/_backup/
# First-time setup on VPS
cp .env.example .env && $EDITOR .env # set RESTIC_REPOSITORY, RESTIC_PASSWORD, WEBHOOK_URL
restic init # initialise restic repo (uses vars from .env)
# Initialize restic repo (first time only)
ssh vps 'source ~/stacks/_backup/.env && restic init -r /mnt/hidrive/users/valknar/Backup/stacks'
# Install systemd units (or use stacks.sh on the VPS)
ssh vps '~/stacks/stacks.sh backup install'
# Manual run / status
ssh vps '~/stacks/stacks.sh backup run'
ssh vps '~/stacks/stacks.sh backup status'
ssh vps '~/stacks/stacks.sh backup snapshots'
./stacks.sh backup install # write & enable systemd unit + timer
./stacks.sh backup run # test run
./stacks.sh backup snapshots # list snapshots
./stacks.sh backup status # timer/service status
./stacks.sh backup logs # journald logs
```
## Updates
The `_update` script runs nightly at 2:00 AM. It pulls the latest image for every stack, recreates any containers whose image changed, prunes dangling images, and sends a Telegram notification via n8n.
Runs nightly at 2:00 AM via a systemd timer. Pulls the latest image for every stack, recreates any container whose image changed, prunes dangling images, and sends a Telegram notification via n8n.
```bash
# Deploy update stack
rsync -avz _update/ vps:~/stacks/_update/
# Install systemd units (or use stacks.sh on the VPS)
ssh vps '~/stacks/stacks.sh update install'
# Manual run / status
ssh vps '~/stacks/stacks.sh update run'
ssh vps '~/stacks/stacks.sh update status'
./stacks.sh update install # write & enable systemd unit + timer
./stacks.sh update run # test run
./stacks.sh update status # timer/service status
./stacks.sh update logs # journald logs
```
## Notifications
_backup/.env.example
-3
View File
@@ -1,3 +0,0 @@
RESTIC_REPOSITORY=/mnt/hidrive/users/valknar/Backup/stacks
RESTIC_PASSWORD=change_me
WEBHOOK_URL=https://n8n.example.com/webhook/change_me
_backup/backup.sh
-83
View File
@@ -1,83 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
STACKS_DIR="$(dirname "$SCRIPT_DIR")"
DATA_DIR="$STACKS_DIR/.data"
DUMP_DIR="$DATA_DIR/backup/dumps"
LOG_FILE="$SCRIPT_DIR/backup.log"
# Load environment
set -a
source "$SCRIPT_DIR/.env"
set +a
export RESTIC_REPOSITORY RESTIC_PASSWORD
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }
notify() {
local color="$1" text="$2"
curl -sf -o /dev/null -X POST "$WEBHOOK_URL" \
-H 'Content-Type: application/json' \
-d "{\"message\":\"$text\",\"color\":\"$color\"}"
}
# Truncate log on each run
: > "$LOG_FILE"
log "Starting backup"
# --- Postgres dumps ---
mkdir -p "$DUMP_DIR"
declare -A DATABASES=(
[umami_db]="umami:umami"
[gitea_db]="gitea:gitea"
[n8n_db]="n8n:n8n"
[immich_db]="immich:immich"
[coolify_db]="coolify:coolify"
[passbolt_db]="passbolt:passbolt"
)
dump_errors=()
for container in "${!DATABASES[@]}"; do
IFS=: read -r user db <<< "${DATABASES[$container]}"
log "Dumping $db from $container"
if docker exec "$container" pg_dump -U "$user" "$db" > "$DUMP_DIR/$db.sql" 2>>"$LOG_FILE"; then
log " OK ($(du -h "$DUMP_DIR/$db.sql" | cut -f1))"
else
log " FAILED: $container"
dump_errors+=("$db")
fi
done
if [ ${#dump_errors[@]} -gt 0 ]; then
log "WARNING: Failed dumps: ${dump_errors[*]}"
fi
# --- Restic backup ---
log "Running restic backup"
if ! restic backup "$DATA_DIR" --tag stacks 2>&1 | tee -a "$LOG_FILE"; then
log "FATAL: restic backup failed"
notify "#cc0000" "❌ **Backup failed** — restic backup error. Check \`$LOG_FILE\`."
exit 1
fi
# --- Restic prune ---
log "Pruning old snapshots"
if ! restic forget --prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 2>&1 | tee -a "$LOG_FILE"; then
log "WARNING: restic forget failed"
fi
# --- Summary ---
snapshot_info=$(restic snapshots --latest 1 --compact 2>/dev/null | tail -3 | head -1)
repo_stats=$(restic stats 2>/dev/null | grep "Total Size" || true)
summary="✅ **Backup complete**"
[ ${#dump_errors[@]} -gt 0 ] && summary+="\n⚠️ Failed dumps: ${dump_errors[*]}"
summary+="\nLatest: \`$snapshot_info\`"
[ -n "$repo_stats" ] && summary+="\nRepo: $repo_stats"
notify "#36a64f" "$summary"
log "Backup complete"
_backup/stacks-backup.service
-10
View File
@@ -1,10 +0,0 @@
[Unit]
Description=Backup stacks data to HiDrive via restic
After=network-online.target docker.service
Wants=network-online.target
[Service]
Type=oneshot
User=root
ExecStart=/root/stacks/_backup/backup.sh
Environment=HOME=/root
_backup/stacks-backup.timer
-9
View File
@@ -1,9 +0,0 @@
[Unit]
Description=Daily stacks backup at 3:00 AM
[Timer]
OnCalendar=*-*-* 03:00:00 Europe/Amsterdam
Persistent=true
[Install]
WantedBy=timers.target
_update/.env.example
-1
View File
@@ -1 +0,0 @@
WEBHOOK_URL=https://n8n.example.com/webhook/change_me
_update/stacks-update.service
-10
View File
@@ -1,10 +0,0 @@
[Unit]
Description=Docker stacks image update
After=network-online.target docker.service
Wants=network-online.target
[Service]
Type=oneshot
User=root
ExecStart=/root/stacks/_update/update.sh
Environment=HOME=/root
_update/stacks-update.timer
-9
View File
@@ -1,9 +0,0 @@
[Unit]
Description=Daily Docker stacks image update at 2:00 AM
[Timer]
OnCalendar=*-*-* 02:00:00 Europe/Amsterdam
Persistent=true
[Install]
WantedBy=timers.target
_update/update.sh
-62
View File
@@ -1,62 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
STACKS_DIR="$(dirname "$SCRIPT_DIR")"
LOG_FILE="$SCRIPT_DIR/update.log"
set -a; source "$SCRIPT_DIR/.env"; set +a
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }
notify() {
local color="$1" text="$2"
curl -sf -o /dev/null -X POST "$WEBHOOK_URL" \
-H 'Content-Type: application/json' \
-d "{\"message\":\"$text\",\"color\":\"$color\"}"
}
: > "$LOG_FILE"
log "Starting image update check"
updated=(); failed=(); checked=0
for stack_dir in "$STACKS_DIR"/*/; do
stack=$(basename "$stack_dir")
[[ "$stack" == _* ]] && continue
[[ ! -f "$stack_dir/compose.yml" ]] && continue
log "Checking $stack"
cd "$stack_dir"
checked=$((checked + 1))
before=$(docker compose images -q 2>/dev/null | sort)
docker compose pull 2>&1 | tee -a "$LOG_FILE"
after=$(docker compose images -q 2>/dev/null | sort)
if [ "$before" != "$after" ]; then
log " Updates found — recreating containers"
if docker compose up -d 2>&1 | tee -a "$LOG_FILE"; then
updated+=("$stack")
else
log " FAILED to recreate $stack"
failed+=("$stack")
fi
else
log " Up to date"
fi
done
log "Pruning dangling images"
docker image prune -f 2>&1 | tee -a "$LOG_FILE"
date_str=$(date '+%Y-%m-%d %H:%M')
if [ ${#failed[@]} -gt 0 ]; then
notify "#cc0000" "❌ **Update failed** — $date_str\nUpdated: ${updated[*]:-none}\nFailed: ${failed[*]}"
elif [ ${#updated[@]} -gt 0 ]; then
notify "#36a64f" "✅ **Updates applied** — $date_str\nStacks updated (${#updated[@]}/$checked): ${updated[*]}"
else
notify "#36a64f" "✅ **All up to date** — $date_str ($checked stacks checked)"
fi
log "Update check complete"
stacks.sh
+210 -35
View File
@@ -7,8 +7,7 @@ set -euo pipefail
SCRIPT_PATH="$(realpath "${BASH_SOURCE[0]}")"
SCRIPT_NAME="$(basename "$SCRIPT_PATH")"
STACKS_DIR="$(cd "$(dirname "$SCRIPT_PATH")" && pwd)"
UPDATE_DIR="$STACKS_DIR/_update"
BACKUP_DIR="$STACKS_DIR/_backup"
ROOT_ENV="$STACKS_DIR/.env"
VERSION="1.0.0"
# ─── Colors ───────────────────────────────────────────────────────────────────
@@ -33,6 +32,30 @@ die() { err "$*"; exit 1; }
header() { [[ $QUIET == true ]] && return; echo -e "\n${BOLD}${BLUE}━━━ $* ━━━${RESET}"; }
sep() { [[ $QUIET == true ]] && return; echo -e "${DIM}────────────────────────────────────────${RESET}"; }
# ─── Root env + Notifications ────────────────────────────────────────────────
load_root_env() {
if [[ -f "$ROOT_ENV" ]]; then
# shellcheck disable=SC1090
set -a; source "$ROOT_ENV"; set +a
else
warn "No root .env found at $ROOT_ENV — create it from .env.example"
fi
}
_notify() {
local color="$1" text="$2"
local webhook="${WEBHOOK_URL:-}"
if [[ -z "$webhook" ]]; then
warn "WEBHOOK_URL not set — skipping notification"
return 0
fi
curl -sf -o /dev/null -X POST "$webhook" \
-H 'Content-Type: application/json' \
-d "{\"message\":\"$text\",\"color\":\"$color\"}" \
|| warn "Webhook notification failed"
}
# ─── Global Flags ─────────────────────────────────────────────────────────────
DRY_RUN=false
@@ -356,25 +379,95 @@ cmd_compose() {
# ─── Update Service ───────────────────────────────────────────────────────────
_update_run() {
load_root_env
local -a all_stacks=()
mapfile -t all_stacks < <(list_all_stacks)
local -a updated=() failed=()
local -i checked=0
log "Starting image update check (${#all_stacks[@]} stacks)"
for stack in "${all_stacks[@]}"; do
local stack_dir="$STACKS_DIR/$stack"
local -a env_flag=()
[[ -f "$stack_dir/.env" ]] && env_flag=(--env-file .env)
info "Checking ${BOLD}$stack${RESET}"
((checked++)) || true
local before after
before=$(cd "$stack_dir" && docker compose "${env_flag[@]}" images -q 2>/dev/null | sort || true)
(cd "$stack_dir" && docker compose "${env_flag[@]}" pull 2>&1) || true
after=$(cd "$stack_dir" && docker compose "${env_flag[@]}" images -q 2>/dev/null | sort || true)
if [[ "$before" != "$after" ]]; then
info " Updates found — recreating $stack"
if (cd "$stack_dir" && docker compose "${env_flag[@]}" up -d 2>&1); then
updated+=("$stack")
ok " $stack updated"
else
err " Failed to recreate $stack"
failed+=("$stack")
fi
else
info " Up to date"
fi
done
log "Pruning dangling images"
docker image prune -f
local date_str
date_str="$(date '+%Y-%m-%d %H:%M')"
if [[ ${#failed[@]} -gt 0 ]]; then
_notify "#cc0000" "❌ **Update failed** — $date_str\nUpdated: ${updated[*]:-none}\nFailed: ${failed[*]}"
return 1
elif [[ ${#updated[@]} -gt 0 ]]; then
_notify "#36a64f" "✅ **Updates applied** — $date_str\nStacks updated (${#updated[@]}/$checked): ${updated[*]}"
else
_notify "#36a64f" "✅ **All up to date** — $date_str ($checked stacks checked)"
fi
log "Update check complete"
}
cmd_update() {
local sub="${1:-help}"; shift || true
case "$sub" in
install)
[[ -d "$UPDATE_DIR" ]] || die "Update dir not found: $UPDATE_DIR"
local svc="$UPDATE_DIR/stacks-update.service"
local tmr="$UPDATE_DIR/stacks-update.timer"
[[ -f "$svc" ]] || die "Missing: $svc"
[[ -f "$tmr" ]] || die "Missing: $tmr"
log "Installing update service"
if $DRY_RUN; then
info "[dry-run] ln -sf $svc /etc/systemd/system/"
info "[dry-run] ln -sf $tmr /etc/systemd/system/"
info "[dry-run] Would write /etc/systemd/system/stacks-update.{service,timer}"
info "[dry-run] systemctl enable --now stacks-update.timer"
return
fi
ln -sf "$svc" /etc/systemd/system/stacks-update.service
ln -sf "$tmr" /etc/systemd/system/stacks-update.timer
cat > /etc/systemd/system/stacks-update.service << EOF
[Unit]
Description=Docker stacks image update
After=network-online.target docker.service
Wants=network-online.target
[Service]
Type=oneshot
User=root
ExecStart=${SCRIPT_PATH} update run
Environment=HOME=/root
EOF
cat > /etc/systemd/system/stacks-update.timer << 'EOF'
[Unit]
Description=Daily Docker stacks image update at 2:00 AM
[Timer]
OnCalendar=*-*-* 02:00:00 Europe/Amsterdam
Persistent=true
[Install]
WantedBy=timers.target
EOF
systemctl daemon-reload
systemctl enable --now stacks-update.timer
ok "Update service installed and enabled"
@@ -391,10 +484,9 @@ cmd_update() {
;;
run)
[[ -f "$UPDATE_DIR/update.sh" ]] || die "Missing: $UPDATE_DIR/update.sh"
log "Running update now"
if $DRY_RUN; then info "[dry-run] bash $UPDATE_DIR/update.sh"; return; fi
bash "$UPDATE_DIR/update.sh"
if $DRY_RUN; then info "[dry-run] Would run image update across all stacks"; return; fi
_update_run
;;
status)
@@ -403,9 +495,8 @@ cmd_update() {
;;
logs)
journalctl -u stacks-update.service "$@" --no-pager 2>/dev/null \
|| cat "$UPDATE_DIR/update.log" 2>/dev/null \
|| warn "No update logs found"
journalctl -u stacks-update.service "${@}" --no-pager 2>/dev/null \
|| warn "No update logs found (is the service installed?)"
;;
next)
@@ -426,25 +517,110 @@ cmd_update() {
# ─── Backup Service ───────────────────────────────────────────────────────────
_backup_run() {
load_root_env
[[ -n "${RESTIC_REPOSITORY:-}" ]] || die "RESTIC_REPOSITORY not set in $ROOT_ENV"
[[ -n "${RESTIC_PASSWORD:-}" ]] || die "RESTIC_PASSWORD not set in $ROOT_ENV"
export RESTIC_REPOSITORY RESTIC_PASSWORD
local data_dir="$STACKS_DIR/.data"
local dump_dir="$data_dir/backup/dumps"
log "Starting backup"
mkdir -p "$dump_dir"
# --- Postgres dumps: detect <stack>_db containers by convention ---
local -a all_stacks=()
mapfile -t all_stacks < <(list_all_stacks)
local -a dump_errors=()
local -i dumped=0
for stack in "${all_stacks[@]}"; do
local container="${stack}_db"
# Check the container exists and is running
if docker ps --filter "name=^/${container}$" --filter "status=running" -q 2>/dev/null | grep -q .; then
info "Dumping ${BOLD}$stack${RESET} DB ($container, user: $stack, db: $stack)"
if docker exec "$container" pg_dump -U "$stack" "$stack" \
> "$dump_dir/${stack}.sql" 2>/dev/null; then
local size
size="$(du -h "$dump_dir/${stack}.sql" 2>/dev/null | cut -f1)"
ok " $stack DB dumped ($size)"
((dumped++)) || true
else
err " Dump failed: $container"
dump_errors+=("$stack")
rm -f "$dump_dir/${stack}.sql"
fi
fi
done
[[ $dumped -gt 0 || ${#dump_errors[@]} -gt 0 ]] \
&& info "DB dumps: $dumped ok, ${#dump_errors[@]} failed"
# --- Restic backup ---
log "Running restic backup of $data_dir"
if ! restic backup "$data_dir" --tag stacks 2>&1; then
err "FATAL: restic backup failed"
_notify "#cc0000" "❌ **Backup failed** — restic error. Check \`journalctl -u stacks-backup\`."
return 1
fi
# --- Prune ---
log "Pruning old snapshots"
restic forget --prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 2>&1 \
|| warn "restic forget failed"
# --- Summary ---
local snapshot_info repo_stats summary
snapshot_info="$(restic snapshots --latest 1 --compact 2>/dev/null | tail -3 | head -1 || true)"
repo_stats="$(restic stats 2>/dev/null | grep "Total Size" || true)"
summary="✅ **Backup complete** — $(date '+%Y-%m-%d %H:%M')"
[[ $dumped -gt 0 ]] && summary+="\nDB dumps: $dumped stacks"
[[ ${#dump_errors[@]} -gt 0 ]] && summary+="\n⚠️ Failed dumps: ${dump_errors[*]}"
[[ -n "$snapshot_info" ]] && summary+="\nLatest: \`$snapshot_info\`"
[[ -n "$repo_stats" ]] && summary+="\nRepo: $repo_stats"
_notify "#36a64f" "$summary"
log "Backup complete"
}
cmd_backup() {
local sub="${1:-help}"; shift || true
case "$sub" in
install)
[[ -d "$BACKUP_DIR" ]] || die "Backup dir not found: $BACKUP_DIR"
local svc="$BACKUP_DIR/stacks-backup.service"
local tmr="$BACKUP_DIR/stacks-backup.timer"
[[ -f "$svc" ]] || die "Missing: $svc"
[[ -f "$tmr" ]] || die "Missing: $tmr"
log "Installing backup service"
if $DRY_RUN; then
info "[dry-run] ln -sf $svc /etc/systemd/system/"
info "[dry-run] ln -sf $tmr /etc/systemd/system/"
info "[dry-run] Would write /etc/systemd/system/stacks-backup.{service,timer}"
info "[dry-run] systemctl enable --now stacks-backup.timer"
return
fi
ln -sf "$svc" /etc/systemd/system/stacks-backup.service
ln -sf "$tmr" /etc/systemd/system/stacks-backup.timer
cat > /etc/systemd/system/stacks-backup.service << EOF
[Unit]
Description=Backup stacks data to HiDrive via restic
After=network-online.target docker.service
Wants=network-online.target
[Service]
Type=oneshot
User=root
ExecStart=${SCRIPT_PATH} backup run
Environment=HOME=/root
EOF
cat > /etc/systemd/system/stacks-backup.timer << 'EOF'
[Unit]
Description=Daily stacks backup at 3:00 AM
[Timer]
OnCalendar=*-*-* 03:00:00 Europe/Amsterdam
Persistent=true
[Install]
WantedBy=timers.target
EOF
systemctl daemon-reload
systemctl enable --now stacks-backup.timer
ok "Backup service installed and enabled"
@@ -461,10 +637,9 @@ cmd_backup() {
;;
run)
[[ -f "$BACKUP_DIR/backup.sh" ]] || die "Missing: $BACKUP_DIR/backup.sh"
log "Running backup now"
if $DRY_RUN; then info "[dry-run] bash $BACKUP_DIR/backup.sh"; return; fi
bash "$BACKUP_DIR/backup.sh"
if $DRY_RUN; then info "[dry-run] Would run restic backup with dynamic DB detection"; return; fi
_backup_run
;;
status)
@@ -473,9 +648,8 @@ cmd_backup() {
;;
logs)
journalctl -u stacks-backup.service "$@" --no-pager 2>/dev/null \
|| cat "$BACKUP_DIR/backup.log" 2>/dev/null \
|| warn "No backup logs found"
journalctl -u stacks-backup.service "${@}" --no-pager 2>/dev/null \
|| warn "No backup logs found (is the service installed?)"
;;
next)
@@ -484,9 +658,10 @@ cmd_backup() {
;;
snapshots)
[[ -f "$BACKUP_DIR/.env" ]] || die "Missing env: $BACKUP_DIR/.env"
# shellcheck disable=SC1090
set -a; source "$BACKUP_DIR/.env"; set +a
load_root_env
[[ -n "${RESTIC_REPOSITORY:-}" ]] || die "RESTIC_REPOSITORY not set in $ROOT_ENV"
[[ -n "${RESTIC_PASSWORD:-}" ]] || die "RESTIC_PASSWORD not set in $ROOT_ENV"
export RESTIC_REPOSITORY RESTIC_PASSWORD
restic snapshots 2>/dev/null || die "restic not installed or repo not initialised"
;;