refactor: absorb _backup and _update into stacks.sh
- Inline update logic (pull → compare digests → up -d → prune → notify) - Inline backup logic with dynamic Postgres detection: any running <stack>_db container is dumped using the <stack>/<stack> convention - Systemd unit files are now generated on `install` from embedded heredocs pointing at stacks.sh itself — no external scripts needed - Root .env (WEBHOOK_URL, RESTIC_REPOSITORY, RESTIC_PASSWORD) replaces the per-service .env files in _backup/ and _update/ - Remove _backup/ and _update/ directories entirely - Update README accordingly Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,6 @@
|
|||||||
|
# Webhook for update/backup notifications (n8n → Telegram)
|
||||||
|
WEBHOOK_URL=https://n8n.example.com/webhook/your-webhook-id
|
||||||
|
|
||||||
|
# Restic backup repository and password
|
||||||
|
RESTIC_REPOSITORY=/mnt/hidrive/users/valknar/Backup/stacks
|
||||||
|
RESTIC_PASSWORD=changeme
|
||||||
@@ -20,11 +20,11 @@ Each stack is independently deployable with its own `compose.yml` and `.env`. Al
|
|||||||
|
|
||||||
## Tools
|
## Tools
|
||||||
|
|
||||||
| File/Directory | Description |
|
| File | Description |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `stacks.sh` | CLI to manage stacks, services, and scaffolding |
|
| `stacks.sh` | CLI to manage stacks, services, scaffolding, updates, and backups |
|
||||||
| `_backup` | Daily restic backups to HiDrive (host script + systemd timer) |
|
| `.env` | Root config: `WEBHOOK_URL`, `RESTIC_REPOSITORY`, `RESTIC_PASSWORD` (gitignored) |
|
||||||
| `_update` | Nightly image update check + prune (host script + systemd timer) |
|
| `.env.example` | Template for the root `.env` |
|
||||||
|
|
||||||
## stacks.sh
|
## stacks.sh
|
||||||
|
|
||||||
@@ -50,16 +50,16 @@ Each stack is independently deployable with its own `compose.yml` and `.env`. Al
|
|||||||
./stacks.sh run passbolt passbolt bin/cake passbolt healthcheck
|
./stacks.sh run passbolt passbolt bin/cake passbolt healthcheck
|
||||||
```
|
```
|
||||||
|
|
||||||
**Service management:**
|
**Service management** (reads `WEBHOOK_URL`, `RESTIC_REPOSITORY`, `RESTIC_PASSWORD` from root `.env`):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./stacks.sh update install # link & enable systemd update timer
|
./stacks.sh update install # write & enable systemd update timer
|
||||||
./stacks.sh update run # run update now
|
./stacks.sh update run # run update now
|
||||||
./stacks.sh update status # show timer/service status
|
./stacks.sh update status # show timer/service status
|
||||||
./stacks.sh update logs # show journal logs
|
./stacks.sh update logs # show journal logs
|
||||||
|
|
||||||
./stacks.sh backup install # link & enable systemd backup timer
|
./stacks.sh backup install # write & enable systemd backup timer
|
||||||
./stacks.sh backup run # run backup now
|
./stacks.sh backup run # run backup now (auto-detects <stack>_db containers)
|
||||||
./stacks.sh backup snapshots # list restic snapshots
|
./stacks.sh backup snapshots # list restic snapshots
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -105,38 +105,29 @@ All stacks share the external `falcon_network` Docker network for inter-service
|
|||||||
|
|
||||||
## Backup
|
## Backup
|
||||||
|
|
||||||
The `_backup` stack runs a daily restic backup at 3:00 AM. It dumps all Postgres databases, then backs up the entire `.data/` directory to HiDrive. Retention: 7 daily, 4 weekly, 6 monthly snapshots. Notifications go to Telegram via n8n.
|
Runs daily at 3:00 AM via a systemd timer. Detects Postgres databases automatically by convention (`<stack>_db` container, user `<stack>`, database `<stack>`), dumps each one, then runs a full restic backup of `.data/`. Retention: 7 daily, 4 weekly, 6 monthly. Notifications go to Telegram via n8n.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Deploy backup stack
|
# First-time setup on VPS
|
||||||
rsync -avz _backup/ vps:~/stacks/_backup/
|
cp .env.example .env && $EDITOR .env # set RESTIC_REPOSITORY, RESTIC_PASSWORD, WEBHOOK_URL
|
||||||
|
restic init # initialise restic repo (uses vars from .env)
|
||||||
|
|
||||||
# Initialize restic repo (first time only)
|
./stacks.sh backup install # write & enable systemd unit + timer
|
||||||
ssh vps 'source ~/stacks/_backup/.env && restic init -r /mnt/hidrive/users/valknar/Backup/stacks'
|
./stacks.sh backup run # test run
|
||||||
|
./stacks.sh backup snapshots # list snapshots
|
||||||
# Install systemd units (or use stacks.sh on the VPS)
|
./stacks.sh backup status # timer/service status
|
||||||
ssh vps '~/stacks/stacks.sh backup install'
|
./stacks.sh backup logs # journald logs
|
||||||
|
|
||||||
# Manual run / status
|
|
||||||
ssh vps '~/stacks/stacks.sh backup run'
|
|
||||||
ssh vps '~/stacks/stacks.sh backup status'
|
|
||||||
ssh vps '~/stacks/stacks.sh backup snapshots'
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Updates
|
## Updates
|
||||||
|
|
||||||
The `_update` script runs nightly at 2:00 AM. It pulls the latest image for every stack, recreates any containers whose image changed, prunes dangling images, and sends a Telegram notification via n8n.
|
Runs nightly at 2:00 AM via a systemd timer. Pulls the latest image for every stack, recreates any container whose image changed, prunes dangling images, and sends a Telegram notification via n8n.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Deploy update stack
|
./stacks.sh update install # write & enable systemd unit + timer
|
||||||
rsync -avz _update/ vps:~/stacks/_update/
|
./stacks.sh update run # test run
|
||||||
|
./stacks.sh update status # timer/service status
|
||||||
# Install systemd units (or use stacks.sh on the VPS)
|
./stacks.sh update logs # journald logs
|
||||||
ssh vps '~/stacks/stacks.sh update install'
|
|
||||||
|
|
||||||
# Manual run / status
|
|
||||||
ssh vps '~/stacks/stacks.sh update run'
|
|
||||||
ssh vps '~/stacks/stacks.sh update status'
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Notifications
|
## Notifications
|
||||||
|
|||||||
@@ -1,3 +0,0 @@
|
|||||||
RESTIC_REPOSITORY=/mnt/hidrive/users/valknar/Backup/stacks
|
|
||||||
RESTIC_PASSWORD=change_me
|
|
||||||
WEBHOOK_URL=https://n8n.example.com/webhook/change_me
|
|
||||||
@@ -1,83 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
|
||||||
STACKS_DIR="$(dirname "$SCRIPT_DIR")"
|
|
||||||
DATA_DIR="$STACKS_DIR/.data"
|
|
||||||
DUMP_DIR="$DATA_DIR/backup/dumps"
|
|
||||||
LOG_FILE="$SCRIPT_DIR/backup.log"
|
|
||||||
|
|
||||||
# Load environment
|
|
||||||
set -a
|
|
||||||
source "$SCRIPT_DIR/.env"
|
|
||||||
set +a
|
|
||||||
|
|
||||||
export RESTIC_REPOSITORY RESTIC_PASSWORD
|
|
||||||
|
|
||||||
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }
|
|
||||||
|
|
||||||
notify() {
|
|
||||||
local color="$1" text="$2"
|
|
||||||
curl -sf -o /dev/null -X POST "$WEBHOOK_URL" \
|
|
||||||
-H 'Content-Type: application/json' \
|
|
||||||
-d "{\"message\":\"$text\",\"color\":\"$color\"}"
|
|
||||||
}
|
|
||||||
|
|
||||||
# Truncate log on each run
|
|
||||||
: > "$LOG_FILE"
|
|
||||||
|
|
||||||
log "Starting backup"
|
|
||||||
|
|
||||||
# --- Postgres dumps ---
|
|
||||||
mkdir -p "$DUMP_DIR"
|
|
||||||
|
|
||||||
declare -A DATABASES=(
|
|
||||||
[umami_db]="umami:umami"
|
|
||||||
[gitea_db]="gitea:gitea"
|
|
||||||
[n8n_db]="n8n:n8n"
|
|
||||||
[immich_db]="immich:immich"
|
|
||||||
[coolify_db]="coolify:coolify"
|
|
||||||
[passbolt_db]="passbolt:passbolt"
|
|
||||||
)
|
|
||||||
|
|
||||||
dump_errors=()
|
|
||||||
for container in "${!DATABASES[@]}"; do
|
|
||||||
IFS=: read -r user db <<< "${DATABASES[$container]}"
|
|
||||||
log "Dumping $db from $container"
|
|
||||||
if docker exec "$container" pg_dump -U "$user" "$db" > "$DUMP_DIR/$db.sql" 2>>"$LOG_FILE"; then
|
|
||||||
log " OK ($(du -h "$DUMP_DIR/$db.sql" | cut -f1))"
|
|
||||||
else
|
|
||||||
log " FAILED: $container"
|
|
||||||
dump_errors+=("$db")
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
if [ ${#dump_errors[@]} -gt 0 ]; then
|
|
||||||
log "WARNING: Failed dumps: ${dump_errors[*]}"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# --- Restic backup ---
|
|
||||||
log "Running restic backup"
|
|
||||||
if ! restic backup "$DATA_DIR" --tag stacks 2>&1 | tee -a "$LOG_FILE"; then
|
|
||||||
log "FATAL: restic backup failed"
|
|
||||||
notify "#cc0000" "❌ **Backup failed** — restic backup error. Check \`$LOG_FILE\`."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# --- Restic prune ---
|
|
||||||
log "Pruning old snapshots"
|
|
||||||
if ! restic forget --prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 2>&1 | tee -a "$LOG_FILE"; then
|
|
||||||
log "WARNING: restic forget failed"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# --- Summary ---
|
|
||||||
snapshot_info=$(restic snapshots --latest 1 --compact 2>/dev/null | tail -3 | head -1)
|
|
||||||
repo_stats=$(restic stats 2>/dev/null | grep "Total Size" || true)
|
|
||||||
|
|
||||||
summary="✅ **Backup complete**"
|
|
||||||
[ ${#dump_errors[@]} -gt 0 ] && summary+="\n⚠️ Failed dumps: ${dump_errors[*]}"
|
|
||||||
summary+="\nLatest: \`$snapshot_info\`"
|
|
||||||
[ -n "$repo_stats" ] && summary+="\nRepo: $repo_stats"
|
|
||||||
|
|
||||||
notify "#36a64f" "$summary"
|
|
||||||
log "Backup complete"
|
|
||||||
@@ -1,10 +0,0 @@
|
|||||||
[Unit]
|
|
||||||
Description=Backup stacks data to HiDrive via restic
|
|
||||||
After=network-online.target docker.service
|
|
||||||
Wants=network-online.target
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
Type=oneshot
|
|
||||||
User=root
|
|
||||||
ExecStart=/root/stacks/_backup/backup.sh
|
|
||||||
Environment=HOME=/root
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
[Unit]
|
|
||||||
Description=Daily stacks backup at 3:00 AM
|
|
||||||
|
|
||||||
[Timer]
|
|
||||||
OnCalendar=*-*-* 03:00:00 Europe/Amsterdam
|
|
||||||
Persistent=true
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=timers.target
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
WEBHOOK_URL=https://n8n.example.com/webhook/change_me
|
|
||||||
@@ -1,10 +0,0 @@
|
|||||||
[Unit]
|
|
||||||
Description=Docker stacks image update
|
|
||||||
After=network-online.target docker.service
|
|
||||||
Wants=network-online.target
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
Type=oneshot
|
|
||||||
User=root
|
|
||||||
ExecStart=/root/stacks/_update/update.sh
|
|
||||||
Environment=HOME=/root
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
[Unit]
|
|
||||||
Description=Daily Docker stacks image update at 2:00 AM
|
|
||||||
|
|
||||||
[Timer]
|
|
||||||
OnCalendar=*-*-* 02:00:00 Europe/Amsterdam
|
|
||||||
Persistent=true
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=timers.target
|
|
||||||
@@ -1,62 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
|
||||||
STACKS_DIR="$(dirname "$SCRIPT_DIR")"
|
|
||||||
LOG_FILE="$SCRIPT_DIR/update.log"
|
|
||||||
|
|
||||||
set -a; source "$SCRIPT_DIR/.env"; set +a
|
|
||||||
|
|
||||||
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }
|
|
||||||
notify() {
|
|
||||||
local color="$1" text="$2"
|
|
||||||
curl -sf -o /dev/null -X POST "$WEBHOOK_URL" \
|
|
||||||
-H 'Content-Type: application/json' \
|
|
||||||
-d "{\"message\":\"$text\",\"color\":\"$color\"}"
|
|
||||||
}
|
|
||||||
|
|
||||||
: > "$LOG_FILE"
|
|
||||||
log "Starting image update check"
|
|
||||||
|
|
||||||
updated=(); failed=(); checked=0
|
|
||||||
|
|
||||||
for stack_dir in "$STACKS_DIR"/*/; do
|
|
||||||
stack=$(basename "$stack_dir")
|
|
||||||
[[ "$stack" == _* ]] && continue
|
|
||||||
[[ ! -f "$stack_dir/compose.yml" ]] && continue
|
|
||||||
|
|
||||||
log "Checking $stack"
|
|
||||||
cd "$stack_dir"
|
|
||||||
checked=$((checked + 1))
|
|
||||||
|
|
||||||
before=$(docker compose images -q 2>/dev/null | sort)
|
|
||||||
docker compose pull 2>&1 | tee -a "$LOG_FILE"
|
|
||||||
after=$(docker compose images -q 2>/dev/null | sort)
|
|
||||||
|
|
||||||
if [ "$before" != "$after" ]; then
|
|
||||||
log " Updates found — recreating containers"
|
|
||||||
if docker compose up -d 2>&1 | tee -a "$LOG_FILE"; then
|
|
||||||
updated+=("$stack")
|
|
||||||
else
|
|
||||||
log " FAILED to recreate $stack"
|
|
||||||
failed+=("$stack")
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
log " Up to date"
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
log "Pruning dangling images"
|
|
||||||
docker image prune -f 2>&1 | tee -a "$LOG_FILE"
|
|
||||||
|
|
||||||
date_str=$(date '+%Y-%m-%d %H:%M')
|
|
||||||
|
|
||||||
if [ ${#failed[@]} -gt 0 ]; then
|
|
||||||
notify "#cc0000" "❌ **Update failed** — $date_str\nUpdated: ${updated[*]:-none}\nFailed: ${failed[*]}"
|
|
||||||
elif [ ${#updated[@]} -gt 0 ]; then
|
|
||||||
notify "#36a64f" "✅ **Updates applied** — $date_str\nStacks updated (${#updated[@]}/$checked): ${updated[*]}"
|
|
||||||
else
|
|
||||||
notify "#36a64f" "✅ **All up to date** — $date_str ($checked stacks checked)"
|
|
||||||
fi
|
|
||||||
|
|
||||||
log "Update check complete"
|
|
||||||
@@ -7,8 +7,7 @@ set -euo pipefail
|
|||||||
SCRIPT_PATH="$(realpath "${BASH_SOURCE[0]}")"
|
SCRIPT_PATH="$(realpath "${BASH_SOURCE[0]}")"
|
||||||
SCRIPT_NAME="$(basename "$SCRIPT_PATH")"
|
SCRIPT_NAME="$(basename "$SCRIPT_PATH")"
|
||||||
STACKS_DIR="$(cd "$(dirname "$SCRIPT_PATH")" && pwd)"
|
STACKS_DIR="$(cd "$(dirname "$SCRIPT_PATH")" && pwd)"
|
||||||
UPDATE_DIR="$STACKS_DIR/_update"
|
ROOT_ENV="$STACKS_DIR/.env"
|
||||||
BACKUP_DIR="$STACKS_DIR/_backup"
|
|
||||||
VERSION="1.0.0"
|
VERSION="1.0.0"
|
||||||
|
|
||||||
# ─── Colors ───────────────────────────────────────────────────────────────────
|
# ─── Colors ───────────────────────────────────────────────────────────────────
|
||||||
@@ -33,6 +32,30 @@ die() { err "$*"; exit 1; }
|
|||||||
header() { [[ $QUIET == true ]] && return; echo -e "\n${BOLD}${BLUE}━━━ $* ━━━${RESET}"; }
|
header() { [[ $QUIET == true ]] && return; echo -e "\n${BOLD}${BLUE}━━━ $* ━━━${RESET}"; }
|
||||||
sep() { [[ $QUIET == true ]] && return; echo -e "${DIM}────────────────────────────────────────${RESET}"; }
|
sep() { [[ $QUIET == true ]] && return; echo -e "${DIM}────────────────────────────────────────${RESET}"; }
|
||||||
|
|
||||||
|
# ─── Root env + Notifications ────────────────────────────────────────────────
|
||||||
|
|
||||||
|
load_root_env() {
|
||||||
|
if [[ -f "$ROOT_ENV" ]]; then
|
||||||
|
# shellcheck disable=SC1090
|
||||||
|
set -a; source "$ROOT_ENV"; set +a
|
||||||
|
else
|
||||||
|
warn "No root .env found at $ROOT_ENV — create it from .env.example"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
_notify() {
|
||||||
|
local color="$1" text="$2"
|
||||||
|
local webhook="${WEBHOOK_URL:-}"
|
||||||
|
if [[ -z "$webhook" ]]; then
|
||||||
|
warn "WEBHOOK_URL not set — skipping notification"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
curl -sf -o /dev/null -X POST "$webhook" \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d "{\"message\":\"$text\",\"color\":\"$color\"}" \
|
||||||
|
|| warn "Webhook notification failed"
|
||||||
|
}
|
||||||
|
|
||||||
# ─── Global Flags ─────────────────────────────────────────────────────────────
|
# ─── Global Flags ─────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
DRY_RUN=false
|
DRY_RUN=false
|
||||||
@@ -356,25 +379,95 @@ cmd_compose() {
|
|||||||
|
|
||||||
# ─── Update Service ───────────────────────────────────────────────────────────
|
# ─── Update Service ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
_update_run() {
|
||||||
|
load_root_env
|
||||||
|
|
||||||
|
local -a all_stacks=()
|
||||||
|
mapfile -t all_stacks < <(list_all_stacks)
|
||||||
|
local -a updated=() failed=()
|
||||||
|
local -i checked=0
|
||||||
|
|
||||||
|
log "Starting image update check (${#all_stacks[@]} stacks)"
|
||||||
|
|
||||||
|
for stack in "${all_stacks[@]}"; do
|
||||||
|
local stack_dir="$STACKS_DIR/$stack"
|
||||||
|
local -a env_flag=()
|
||||||
|
[[ -f "$stack_dir/.env" ]] && env_flag=(--env-file .env)
|
||||||
|
|
||||||
|
info "Checking ${BOLD}$stack${RESET}"
|
||||||
|
((checked++)) || true
|
||||||
|
|
||||||
|
local before after
|
||||||
|
before=$(cd "$stack_dir" && docker compose "${env_flag[@]}" images -q 2>/dev/null | sort || true)
|
||||||
|
(cd "$stack_dir" && docker compose "${env_flag[@]}" pull 2>&1) || true
|
||||||
|
after=$(cd "$stack_dir" && docker compose "${env_flag[@]}" images -q 2>/dev/null | sort || true)
|
||||||
|
|
||||||
|
if [[ "$before" != "$after" ]]; then
|
||||||
|
info " Updates found — recreating $stack"
|
||||||
|
if (cd "$stack_dir" && docker compose "${env_flag[@]}" up -d 2>&1); then
|
||||||
|
updated+=("$stack")
|
||||||
|
ok " $stack updated"
|
||||||
|
else
|
||||||
|
err " Failed to recreate $stack"
|
||||||
|
failed+=("$stack")
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
info " Up to date"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
log "Pruning dangling images"
|
||||||
|
docker image prune -f
|
||||||
|
|
||||||
|
local date_str
|
||||||
|
date_str="$(date '+%Y-%m-%d %H:%M')"
|
||||||
|
|
||||||
|
if [[ ${#failed[@]} -gt 0 ]]; then
|
||||||
|
_notify "#cc0000" "❌ **Update failed** — $date_str\nUpdated: ${updated[*]:-none}\nFailed: ${failed[*]}"
|
||||||
|
return 1
|
||||||
|
elif [[ ${#updated[@]} -gt 0 ]]; then
|
||||||
|
_notify "#36a64f" "✅ **Updates applied** — $date_str\nStacks updated (${#updated[@]}/$checked): ${updated[*]}"
|
||||||
|
else
|
||||||
|
_notify "#36a64f" "✅ **All up to date** — $date_str ($checked stacks checked)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Update check complete"
|
||||||
|
}
|
||||||
|
|
||||||
cmd_update() {
|
cmd_update() {
|
||||||
local sub="${1:-help}"; shift || true
|
local sub="${1:-help}"; shift || true
|
||||||
|
|
||||||
case "$sub" in
|
case "$sub" in
|
||||||
install)
|
install)
|
||||||
[[ -d "$UPDATE_DIR" ]] || die "Update dir not found: $UPDATE_DIR"
|
|
||||||
local svc="$UPDATE_DIR/stacks-update.service"
|
|
||||||
local tmr="$UPDATE_DIR/stacks-update.timer"
|
|
||||||
[[ -f "$svc" ]] || die "Missing: $svc"
|
|
||||||
[[ -f "$tmr" ]] || die "Missing: $tmr"
|
|
||||||
log "Installing update service"
|
log "Installing update service"
|
||||||
if $DRY_RUN; then
|
if $DRY_RUN; then
|
||||||
info "[dry-run] ln -sf $svc /etc/systemd/system/"
|
info "[dry-run] Would write /etc/systemd/system/stacks-update.{service,timer}"
|
||||||
info "[dry-run] ln -sf $tmr /etc/systemd/system/"
|
|
||||||
info "[dry-run] systemctl enable --now stacks-update.timer"
|
info "[dry-run] systemctl enable --now stacks-update.timer"
|
||||||
return
|
return
|
||||||
fi
|
fi
|
||||||
ln -sf "$svc" /etc/systemd/system/stacks-update.service
|
cat > /etc/systemd/system/stacks-update.service << EOF
|
||||||
ln -sf "$tmr" /etc/systemd/system/stacks-update.timer
|
[Unit]
|
||||||
|
Description=Docker stacks image update
|
||||||
|
After=network-online.target docker.service
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
User=root
|
||||||
|
ExecStart=${SCRIPT_PATH} update run
|
||||||
|
Environment=HOME=/root
|
||||||
|
EOF
|
||||||
|
cat > /etc/systemd/system/stacks-update.timer << 'EOF'
|
||||||
|
[Unit]
|
||||||
|
Description=Daily Docker stacks image update at 2:00 AM
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnCalendar=*-*-* 02:00:00 Europe/Amsterdam
|
||||||
|
Persistent=true
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
|
EOF
|
||||||
systemctl daemon-reload
|
systemctl daemon-reload
|
||||||
systemctl enable --now stacks-update.timer
|
systemctl enable --now stacks-update.timer
|
||||||
ok "Update service installed and enabled"
|
ok "Update service installed and enabled"
|
||||||
@@ -391,10 +484,9 @@ cmd_update() {
|
|||||||
;;
|
;;
|
||||||
|
|
||||||
run)
|
run)
|
||||||
[[ -f "$UPDATE_DIR/update.sh" ]] || die "Missing: $UPDATE_DIR/update.sh"
|
|
||||||
log "Running update now"
|
log "Running update now"
|
||||||
if $DRY_RUN; then info "[dry-run] bash $UPDATE_DIR/update.sh"; return; fi
|
if $DRY_RUN; then info "[dry-run] Would run image update across all stacks"; return; fi
|
||||||
bash "$UPDATE_DIR/update.sh"
|
_update_run
|
||||||
;;
|
;;
|
||||||
|
|
||||||
status)
|
status)
|
||||||
@@ -403,9 +495,8 @@ cmd_update() {
|
|||||||
;;
|
;;
|
||||||
|
|
||||||
logs)
|
logs)
|
||||||
journalctl -u stacks-update.service "$@" --no-pager 2>/dev/null \
|
journalctl -u stacks-update.service "${@}" --no-pager 2>/dev/null \
|
||||||
|| cat "$UPDATE_DIR/update.log" 2>/dev/null \
|
|| warn "No update logs found (is the service installed?)"
|
||||||
|| warn "No update logs found"
|
|
||||||
;;
|
;;
|
||||||
|
|
||||||
next)
|
next)
|
||||||
@@ -426,25 +517,110 @@ cmd_update() {
|
|||||||
|
|
||||||
# ─── Backup Service ───────────────────────────────────────────────────────────
|
# ─── Backup Service ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
_backup_run() {
|
||||||
|
load_root_env
|
||||||
|
|
||||||
|
[[ -n "${RESTIC_REPOSITORY:-}" ]] || die "RESTIC_REPOSITORY not set in $ROOT_ENV"
|
||||||
|
[[ -n "${RESTIC_PASSWORD:-}" ]] || die "RESTIC_PASSWORD not set in $ROOT_ENV"
|
||||||
|
export RESTIC_REPOSITORY RESTIC_PASSWORD
|
||||||
|
|
||||||
|
local data_dir="$STACKS_DIR/.data"
|
||||||
|
local dump_dir="$data_dir/backup/dumps"
|
||||||
|
|
||||||
|
log "Starting backup"
|
||||||
|
mkdir -p "$dump_dir"
|
||||||
|
|
||||||
|
# --- Postgres dumps: detect <stack>_db containers by convention ---
|
||||||
|
local -a all_stacks=()
|
||||||
|
mapfile -t all_stacks < <(list_all_stacks)
|
||||||
|
|
||||||
|
local -a dump_errors=()
|
||||||
|
local -i dumped=0
|
||||||
|
|
||||||
|
for stack in "${all_stacks[@]}"; do
|
||||||
|
local container="${stack}_db"
|
||||||
|
# Check the container exists and is running
|
||||||
|
if docker ps --filter "name=^/${container}$" --filter "status=running" -q 2>/dev/null | grep -q .; then
|
||||||
|
info "Dumping ${BOLD}$stack${RESET} DB ($container, user: $stack, db: $stack)"
|
||||||
|
if docker exec "$container" pg_dump -U "$stack" "$stack" \
|
||||||
|
> "$dump_dir/${stack}.sql" 2>/dev/null; then
|
||||||
|
local size
|
||||||
|
size="$(du -h "$dump_dir/${stack}.sql" 2>/dev/null | cut -f1)"
|
||||||
|
ok " $stack DB dumped ($size)"
|
||||||
|
((dumped++)) || true
|
||||||
|
else
|
||||||
|
err " Dump failed: $container"
|
||||||
|
dump_errors+=("$stack")
|
||||||
|
rm -f "$dump_dir/${stack}.sql"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
[[ $dumped -gt 0 || ${#dump_errors[@]} -gt 0 ]] \
|
||||||
|
&& info "DB dumps: $dumped ok, ${#dump_errors[@]} failed"
|
||||||
|
|
||||||
|
# --- Restic backup ---
|
||||||
|
log "Running restic backup of $data_dir"
|
||||||
|
if ! restic backup "$data_dir" --tag stacks 2>&1; then
|
||||||
|
err "FATAL: restic backup failed"
|
||||||
|
_notify "#cc0000" "❌ **Backup failed** — restic error. Check \`journalctl -u stacks-backup\`."
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- Prune ---
|
||||||
|
log "Pruning old snapshots"
|
||||||
|
restic forget --prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 2>&1 \
|
||||||
|
|| warn "restic forget failed"
|
||||||
|
|
||||||
|
# --- Summary ---
|
||||||
|
local snapshot_info repo_stats summary
|
||||||
|
snapshot_info="$(restic snapshots --latest 1 --compact 2>/dev/null | tail -3 | head -1 || true)"
|
||||||
|
repo_stats="$(restic stats 2>/dev/null | grep "Total Size" || true)"
|
||||||
|
|
||||||
|
summary="✅ **Backup complete** — $(date '+%Y-%m-%d %H:%M')"
|
||||||
|
[[ $dumped -gt 0 ]] && summary+="\nDB dumps: $dumped stacks"
|
||||||
|
[[ ${#dump_errors[@]} -gt 0 ]] && summary+="\n⚠️ Failed dumps: ${dump_errors[*]}"
|
||||||
|
[[ -n "$snapshot_info" ]] && summary+="\nLatest: \`$snapshot_info\`"
|
||||||
|
[[ -n "$repo_stats" ]] && summary+="\nRepo: $repo_stats"
|
||||||
|
|
||||||
|
_notify "#36a64f" "$summary"
|
||||||
|
log "Backup complete"
|
||||||
|
}
|
||||||
|
|
||||||
cmd_backup() {
|
cmd_backup() {
|
||||||
local sub="${1:-help}"; shift || true
|
local sub="${1:-help}"; shift || true
|
||||||
|
|
||||||
case "$sub" in
|
case "$sub" in
|
||||||
install)
|
install)
|
||||||
[[ -d "$BACKUP_DIR" ]] || die "Backup dir not found: $BACKUP_DIR"
|
|
||||||
local svc="$BACKUP_DIR/stacks-backup.service"
|
|
||||||
local tmr="$BACKUP_DIR/stacks-backup.timer"
|
|
||||||
[[ -f "$svc" ]] || die "Missing: $svc"
|
|
||||||
[[ -f "$tmr" ]] || die "Missing: $tmr"
|
|
||||||
log "Installing backup service"
|
log "Installing backup service"
|
||||||
if $DRY_RUN; then
|
if $DRY_RUN; then
|
||||||
info "[dry-run] ln -sf $svc /etc/systemd/system/"
|
info "[dry-run] Would write /etc/systemd/system/stacks-backup.{service,timer}"
|
||||||
info "[dry-run] ln -sf $tmr /etc/systemd/system/"
|
|
||||||
info "[dry-run] systemctl enable --now stacks-backup.timer"
|
info "[dry-run] systemctl enable --now stacks-backup.timer"
|
||||||
return
|
return
|
||||||
fi
|
fi
|
||||||
ln -sf "$svc" /etc/systemd/system/stacks-backup.service
|
cat > /etc/systemd/system/stacks-backup.service << EOF
|
||||||
ln -sf "$tmr" /etc/systemd/system/stacks-backup.timer
|
[Unit]
|
||||||
|
Description=Backup stacks data to HiDrive via restic
|
||||||
|
After=network-online.target docker.service
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
User=root
|
||||||
|
ExecStart=${SCRIPT_PATH} backup run
|
||||||
|
Environment=HOME=/root
|
||||||
|
EOF
|
||||||
|
cat > /etc/systemd/system/stacks-backup.timer << 'EOF'
|
||||||
|
[Unit]
|
||||||
|
Description=Daily stacks backup at 3:00 AM
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnCalendar=*-*-* 03:00:00 Europe/Amsterdam
|
||||||
|
Persistent=true
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
|
EOF
|
||||||
systemctl daemon-reload
|
systemctl daemon-reload
|
||||||
systemctl enable --now stacks-backup.timer
|
systemctl enable --now stacks-backup.timer
|
||||||
ok "Backup service installed and enabled"
|
ok "Backup service installed and enabled"
|
||||||
@@ -461,10 +637,9 @@ cmd_backup() {
|
|||||||
;;
|
;;
|
||||||
|
|
||||||
run)
|
run)
|
||||||
[[ -f "$BACKUP_DIR/backup.sh" ]] || die "Missing: $BACKUP_DIR/backup.sh"
|
|
||||||
log "Running backup now"
|
log "Running backup now"
|
||||||
if $DRY_RUN; then info "[dry-run] bash $BACKUP_DIR/backup.sh"; return; fi
|
if $DRY_RUN; then info "[dry-run] Would run restic backup with dynamic DB detection"; return; fi
|
||||||
bash "$BACKUP_DIR/backup.sh"
|
_backup_run
|
||||||
;;
|
;;
|
||||||
|
|
||||||
status)
|
status)
|
||||||
@@ -473,9 +648,8 @@ cmd_backup() {
|
|||||||
;;
|
;;
|
||||||
|
|
||||||
logs)
|
logs)
|
||||||
journalctl -u stacks-backup.service "$@" --no-pager 2>/dev/null \
|
journalctl -u stacks-backup.service "${@}" --no-pager 2>/dev/null \
|
||||||
|| cat "$BACKUP_DIR/backup.log" 2>/dev/null \
|
|| warn "No backup logs found (is the service installed?)"
|
||||||
|| warn "No backup logs found"
|
|
||||||
;;
|
;;
|
||||||
|
|
||||||
next)
|
next)
|
||||||
@@ -484,9 +658,10 @@ cmd_backup() {
|
|||||||
;;
|
;;
|
||||||
|
|
||||||
snapshots)
|
snapshots)
|
||||||
[[ -f "$BACKUP_DIR/.env" ]] || die "Missing env: $BACKUP_DIR/.env"
|
load_root_env
|
||||||
# shellcheck disable=SC1090
|
[[ -n "${RESTIC_REPOSITORY:-}" ]] || die "RESTIC_REPOSITORY not set in $ROOT_ENV"
|
||||||
set -a; source "$BACKUP_DIR/.env"; set +a
|
[[ -n "${RESTIC_PASSWORD:-}" ]] || die "RESTIC_PASSWORD not set in $ROOT_ENV"
|
||||||
|
export RESTIC_REPOSITORY RESTIC_PASSWORD
|
||||||
restic snapshots 2>/dev/null || die "restic not installed or repo not initialised"
|
restic snapshots 2>/dev/null || die "restic not installed or repo not initialised"
|
||||||
;;
|
;;
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user