Compare commits
7 Commits
e613d766ec
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| d7829cb5ae | |||
| 8ae9c9e878 | |||
| fcff6f3298 | |||
| e3cd2df372 | |||
| 067d017ea6 | |||
| 3251f27edb | |||
| cf32e669cb |
@@ -0,0 +1,6 @@
|
||||
# Webhook for update/backup notifications (n8n → Telegram)
|
||||
WEBHOOK_URL=https://n8n.example.com/webhook/your-webhook-id
|
||||
|
||||
# Restic backup repository and password
|
||||
RESTIC_REPOSITORY=/mnt/hidrive/users/valknar/Backup/stacks
|
||||
RESTIC_PASSWORD=changeme
|
||||
@@ -20,10 +20,71 @@ Each stack is independently deployable with its own `compose.yml` and `.env`. Al
|
||||
|
||||
## Tools
|
||||
|
||||
| Directory | Description |
|
||||
| File | Description |
|
||||
|---|---|
|
||||
| `_backup` | Daily restic backups to HiDrive (host script + systemd timer) |
|
||||
| `_update` | Nightly image update check + prune (host script + systemd timer) |
|
||||
| `stacks.sh` | CLI to manage stacks, services, scaffolding, updates, and backups |
|
||||
| `.env` | Root config: `WEBHOOK_URL`, `RESTIC_REPOSITORY`, `RESTIC_PASSWORD` (gitignored) |
|
||||
| `.env.example` | Template for the root `.env` |
|
||||
|
||||
## stacks.sh
|
||||
|
||||
`stacks.sh` is the primary management CLI. It wraps `docker compose` with glob-based multi-stack targeting, manages the update and backup systemd services, scaffolds new stacks, and generates shell completions.
|
||||
|
||||
```bash
|
||||
./stacks.sh help
|
||||
```
|
||||
|
||||
**Stack commands** — all accept one or more stack names or glob patterns (omit for all stacks):
|
||||
|
||||
```bash
|
||||
./stacks.sh ls # list all stacks with live container status
|
||||
./stacks.sh ps gitea # container status table
|
||||
./stacks.sh up # start all stacks
|
||||
./stacks.sh up gitea traefik # start specific stacks
|
||||
./stacks.sh down 'g*' # stop stacks matching glob
|
||||
./stacks.sh restart 'g*,traefik' # glob + exact name, comma-separated
|
||||
./stacks.sh pull --parallel # pull all images in parallel
|
||||
./stacks.sh logs -f gitea # follow logs
|
||||
./stacks.sh logs -n 100 gitea n8n # tail multiple stacks
|
||||
./stacks.sh exec gitea gitea gitea admin user list # exec in container
|
||||
./stacks.sh run passbolt passbolt bin/cake passbolt healthcheck
|
||||
```
|
||||
|
||||
**Service management** (reads `WEBHOOK_URL`, `RESTIC_REPOSITORY`, `RESTIC_PASSWORD` from root `.env`):
|
||||
|
||||
```bash
|
||||
./stacks.sh update install # write & enable systemd update timer
|
||||
./stacks.sh update run # run update now
|
||||
./stacks.sh update status # show timer/service status
|
||||
./stacks.sh update logs # show journal logs
|
||||
|
||||
./stacks.sh backup install # write & enable systemd backup timer
|
||||
./stacks.sh backup run # run backup now (auto-detects <stack>_db containers)
|
||||
./stacks.sh backup snapshots # list restic snapshots
|
||||
```
|
||||
|
||||
**Scaffold a new stack:**
|
||||
|
||||
```bash
|
||||
./stacks.sh new myapp # basic stack with Traefik labels
|
||||
./stacks.sh new myapp --db postgres # with Postgres service
|
||||
./stacks.sh new myapp --db postgres --redis # with Postgres + Redis
|
||||
./stacks.sh new myapp --no-traefik # expose port instead of Traefik
|
||||
```
|
||||
|
||||
Generates `compose.yml` (with healthchecks, `../.data/` volumes, Traefik labels) and `.env.example`.
|
||||
|
||||
**Shell completion:**
|
||||
|
||||
```bash
|
||||
# Dynamic — discovers stacks at tab-complete time (default)
|
||||
./stacks.sh completion zsh --install
|
||||
|
||||
# Static — bakes current stack list in; useful on the VPS
|
||||
./stacks.sh completion zsh --static --install
|
||||
```
|
||||
|
||||
**Global flags:** `--dry-run`, `--parallel`, `--verbose`, `--quiet`
|
||||
|
||||
## Deployment
|
||||
|
||||
@@ -44,60 +105,36 @@ All stacks share the external `falcon_network` Docker network for inter-service
|
||||
|
||||
## Backup
|
||||
|
||||
The `_backup` stack runs a daily restic backup at 3:00 AM. It dumps all Postgres databases, then backs up the entire `.data/` directory to HiDrive. Retention: 7 daily, 4 weekly, 6 monthly snapshots. Notifications go to Telegram via n8n.
|
||||
Runs daily at 3:00 AM via a systemd timer. Detects Postgres databases automatically by convention (`<stack>_db` container, user `<stack>`, database `<stack>`), dumps each one, then runs a full restic backup of `.data/`. Retention: 7 daily, 4 weekly, 6 monthly. Notifications go to Telegram via n8n.
|
||||
|
||||
```bash
|
||||
# Deploy backup stack
|
||||
rsync -avz _backup/ vps:~/stacks/_backup/
|
||||
# First-time setup on VPS
|
||||
cp .env.example .env && $EDITOR .env # set RESTIC_REPOSITORY, RESTIC_PASSWORD, WEBHOOK_URL
|
||||
source .env && restic init # initialise restic repo
|
||||
|
||||
# Initialize restic repo (first time only)
|
||||
ssh vps 'source ~/stacks/_backup/.env && restic init -r /mnt/hidrive/users/valknar/Backup/stacks'
|
||||
|
||||
# Install systemd units
|
||||
ssh vps 'ln -sf ~/stacks/_backup/stacks-backup.service /etc/systemd/system/ && \
|
||||
ln -sf ~/stacks/_backup/stacks-backup.timer /etc/systemd/system/ && \
|
||||
systemctl daemon-reload && systemctl enable --now stacks-backup.timer'
|
||||
|
||||
# Manual test run
|
||||
ssh vps '~/stacks/_backup/backup.sh'
|
||||
|
||||
# Check timer status
|
||||
ssh vps 'systemctl status stacks-backup.timer'
|
||||
|
||||
# View snapshots
|
||||
ssh vps 'source ~/stacks/_backup/.env && restic -r /mnt/hidrive/users/valknar/Backup/stacks snapshots'
|
||||
./stacks.sh backup install # write & enable systemd unit + timer
|
||||
./stacks.sh backup run # test run
|
||||
./stacks.sh backup snapshots # list snapshots
|
||||
./stacks.sh backup status # timer/service status
|
||||
./stacks.sh backup logs # journald logs
|
||||
```
|
||||
|
||||
## Updates
|
||||
|
||||
The `_update` script runs nightly at 2:00 AM. It pulls the latest image for every stack, recreates any containers whose image changed, prunes dangling images, and sends a Telegram notification via n8n.
|
||||
Runs nightly at 2:00 AM via a systemd timer. Pulls the latest image for every stack, recreates any container whose image changed, prunes dangling images, and sends a Telegram notification via n8n.
|
||||
|
||||
```bash
|
||||
# Deploy update stack
|
||||
rsync -avz _update/ vps:~/stacks/_update/
|
||||
ssh vps 'chmod +x ~/stacks/_update/update.sh'
|
||||
|
||||
# Install systemd units
|
||||
ssh vps 'ln -sf ~/stacks/_update/stacks-update.service /etc/systemd/system/ && \
|
||||
ln -sf ~/stacks/_update/stacks-update.timer /etc/systemd/system/ && \
|
||||
systemctl daemon-reload && systemctl enable --now stacks-update.timer'
|
||||
|
||||
# Manual test run
|
||||
ssh vps '~/stacks/_update/update.sh'
|
||||
|
||||
# Check timer status
|
||||
ssh vps 'systemctl status stacks-update.timer'
|
||||
./stacks.sh update install # write & enable systemd unit + timer
|
||||
./stacks.sh update run # test run
|
||||
./stacks.sh update status # timer/service status
|
||||
./stacks.sh update logs # journald logs
|
||||
```
|
||||
|
||||
## Notifications
|
||||
|
||||
The update script and the backup script both POST to an n8n webhook, which forwards messages to Telegram.
|
||||
Both update and backup POST to an n8n webhook on completion, which forwards the message to Telegram.
|
||||
|
||||
The webhook URL is set in:
|
||||
- `_backup/.env` → `WEBHOOK_URL`
|
||||
- `_update/.env` → `WEBHOOK_URL`
|
||||
|
||||
Both point to the same n8n workflow at `https://n8n.pivoine.art`. The workflow accepts `{ "message": "..." }` and forwards it to Telegram.
|
||||
The webhook URL is set via `WEBHOOK_URL` in the root `.env`. Both services point to the same n8n workflow at `https://n8n.pivoine.art`, which accepts `{ "message": "...", "color": "..." }` and forwards it to Telegram.
|
||||
|
||||
## Data
|
||||
|
||||
|
||||
@@ -1,3 +0,0 @@
|
||||
RESTIC_REPOSITORY=/mnt/hidrive/users/valknar/Backup/stacks
|
||||
RESTIC_PASSWORD=change_me
|
||||
WEBHOOK_URL=https://n8n.example.com/webhook/change_me
|
||||
@@ -1,83 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
STACKS_DIR="$(dirname "$SCRIPT_DIR")"
|
||||
DATA_DIR="$STACKS_DIR/.data"
|
||||
DUMP_DIR="$DATA_DIR/backup/dumps"
|
||||
LOG_FILE="$SCRIPT_DIR/backup.log"
|
||||
|
||||
# Load environment
|
||||
set -a
|
||||
source "$SCRIPT_DIR/.env"
|
||||
set +a
|
||||
|
||||
export RESTIC_REPOSITORY RESTIC_PASSWORD
|
||||
|
||||
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }
|
||||
|
||||
notify() {
|
||||
local color="$1" text="$2"
|
||||
curl -sf -o /dev/null -X POST "$WEBHOOK_URL" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"message\":\"$text\",\"color\":\"$color\"}"
|
||||
}
|
||||
|
||||
# Truncate log on each run
|
||||
: > "$LOG_FILE"
|
||||
|
||||
log "Starting backup"
|
||||
|
||||
# --- Postgres dumps ---
|
||||
mkdir -p "$DUMP_DIR"
|
||||
|
||||
declare -A DATABASES=(
|
||||
[umami_db]="umami:umami"
|
||||
[gitea_db]="gitea:gitea"
|
||||
[n8n_db]="n8n:n8n"
|
||||
[immich_db]="immich:immich"
|
||||
[coolify_db]="coolify:coolify"
|
||||
[passbolt_db]="passbolt:passbolt"
|
||||
)
|
||||
|
||||
dump_errors=()
|
||||
for container in "${!DATABASES[@]}"; do
|
||||
IFS=: read -r user db <<< "${DATABASES[$container]}"
|
||||
log "Dumping $db from $container"
|
||||
if docker exec "$container" pg_dump -U "$user" "$db" > "$DUMP_DIR/$db.sql" 2>>"$LOG_FILE"; then
|
||||
log " OK ($(du -h "$DUMP_DIR/$db.sql" | cut -f1))"
|
||||
else
|
||||
log " FAILED: $container"
|
||||
dump_errors+=("$db")
|
||||
fi
|
||||
done
|
||||
|
||||
if [ ${#dump_errors[@]} -gt 0 ]; then
|
||||
log "WARNING: Failed dumps: ${dump_errors[*]}"
|
||||
fi
|
||||
|
||||
# --- Restic backup ---
|
||||
log "Running restic backup"
|
||||
if ! restic backup "$DATA_DIR" --tag stacks 2>&1 | tee -a "$LOG_FILE"; then
|
||||
log "FATAL: restic backup failed"
|
||||
notify "#cc0000" "❌ **Backup failed** — restic backup error. Check \`$LOG_FILE\`."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- Restic prune ---
|
||||
log "Pruning old snapshots"
|
||||
if ! restic forget --prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 2>&1 | tee -a "$LOG_FILE"; then
|
||||
log "WARNING: restic forget failed"
|
||||
fi
|
||||
|
||||
# --- Summary ---
|
||||
snapshot_info=$(restic snapshots --latest 1 --compact 2>/dev/null | tail -3 | head -1)
|
||||
repo_stats=$(restic stats 2>/dev/null | grep "Total Size" || true)
|
||||
|
||||
summary="✅ **Backup complete**"
|
||||
[ ${#dump_errors[@]} -gt 0 ] && summary+="\n⚠️ Failed dumps: ${dump_errors[*]}"
|
||||
summary+="\nLatest: \`$snapshot_info\`"
|
||||
[ -n "$repo_stats" ] && summary+="\nRepo: $repo_stats"
|
||||
|
||||
notify "#36a64f" "$summary"
|
||||
log "Backup complete"
|
||||
@@ -1,10 +0,0 @@
|
||||
[Unit]
|
||||
Description=Backup stacks data to HiDrive via restic
|
||||
After=network-online.target docker.service
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=root
|
||||
ExecStart=/root/stacks/_backup/backup.sh
|
||||
Environment=HOME=/root
|
||||
@@ -1,9 +0,0 @@
|
||||
[Unit]
|
||||
Description=Daily stacks backup at 3:00 AM
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 03:00:00 Europe/Amsterdam
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -1 +0,0 @@
|
||||
WEBHOOK_URL=https://n8n.example.com/webhook/change_me
|
||||
@@ -1,10 +0,0 @@
|
||||
[Unit]
|
||||
Description=Docker stacks image update
|
||||
After=network-online.target docker.service
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=root
|
||||
ExecStart=/root/stacks/_update/update.sh
|
||||
Environment=HOME=/root
|
||||
@@ -1,9 +0,0 @@
|
||||
[Unit]
|
||||
Description=Daily Docker stacks image update at 2:00 AM
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 02:00:00 Europe/Amsterdam
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -1,62 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
STACKS_DIR="$(dirname "$SCRIPT_DIR")"
|
||||
LOG_FILE="$SCRIPT_DIR/update.log"
|
||||
|
||||
set -a; source "$SCRIPT_DIR/.env"; set +a
|
||||
|
||||
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }
|
||||
notify() {
|
||||
local color="$1" text="$2"
|
||||
curl -sf -o /dev/null -X POST "$WEBHOOK_URL" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"message\":\"$text\",\"color\":\"$color\"}"
|
||||
}
|
||||
|
||||
: > "$LOG_FILE"
|
||||
log "Starting image update check"
|
||||
|
||||
updated=(); failed=(); checked=0
|
||||
|
||||
for stack_dir in "$STACKS_DIR"/*/; do
|
||||
stack=$(basename "$stack_dir")
|
||||
[[ "$stack" == _* ]] && continue
|
||||
[[ ! -f "$stack_dir/compose.yml" ]] && continue
|
||||
|
||||
log "Checking $stack"
|
||||
cd "$stack_dir"
|
||||
checked=$((checked + 1))
|
||||
|
||||
before=$(docker compose images -q 2>/dev/null | sort)
|
||||
docker compose pull 2>&1 | tee -a "$LOG_FILE"
|
||||
after=$(docker compose images -q 2>/dev/null | sort)
|
||||
|
||||
if [ "$before" != "$after" ]; then
|
||||
log " Updates found — recreating containers"
|
||||
if docker compose up -d 2>&1 | tee -a "$LOG_FILE"; then
|
||||
updated+=("$stack")
|
||||
else
|
||||
log " FAILED to recreate $stack"
|
||||
failed+=("$stack")
|
||||
fi
|
||||
else
|
||||
log " Up to date"
|
||||
fi
|
||||
done
|
||||
|
||||
log "Pruning dangling images"
|
||||
docker image prune -f 2>&1 | tee -a "$LOG_FILE"
|
||||
|
||||
date_str=$(date '+%Y-%m-%d %H:%M')
|
||||
|
||||
if [ ${#failed[@]} -gt 0 ]; then
|
||||
notify "#cc0000" "❌ **Update failed** — $date_str\nUpdated: ${updated[*]:-none}\nFailed: ${failed[*]}"
|
||||
elif [ ${#updated[@]} -gt 0 ]; then
|
||||
notify "#36a64f" "✅ **Updates applied** — $date_str\nStacks updated (${#updated[@]}/$checked): ${updated[*]}"
|
||||
else
|
||||
notify "#36a64f" "✅ **All up to date** — $date_str ($checked stacks checked)"
|
||||
fi
|
||||
|
||||
log "Update check complete"
|
||||
@@ -3,5 +3,3 @@ NETWORK_NAME=falcon_network
|
||||
RUNNER_TOKEN=change_me
|
||||
EMAIL_FROM=gitea@pivoine.art
|
||||
EMAIL_USER=Gitea
|
||||
UMAMI_SRC=https://umami.example.com/script.js
|
||||
UMAMI_ID=
|
||||
|
||||
@@ -36,8 +36,6 @@ services:
|
||||
GITEA__ui__THEMES: gitea-auto,gitea-light,gitea-dark,arc-green,edge-auto,edge-dark,edge-light,everforest-auto,everforest-dark,everforest-light,gruvbox-auto,gruvbox-dark,gruvbox-light,gruvbox-material-auto,gruvbox-material-dark,gruvbox-material-light,nord,palenight,soft-era,sonokai,sonokai-andromeda,sonokai-atlantis,sonokai-espresso,sonokai-maia,sonokai-shusia
|
||||
GITEA__ui__DEFAULT_THEME: nord
|
||||
GITEA__ui__ENABLE_FEED: "true"
|
||||
UMAMI_SRC: ${UMAMI_SRC}
|
||||
UMAMI_ID: ${UMAMI_ID}
|
||||
ports:
|
||||
- "2222:2222"
|
||||
volumes:
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
{{if and (eq .RunMode "prod") (not .IsSigned)}}
|
||||
<script defer src="{{.UmamiSrc}}" data-website-id="{{.UmamiId}}"></script>
|
||||
{{if and .RunModeIsProd (not .IsSigned)}}
|
||||
<script defer src="https://umami.pivoine.art/script.js" data-website-id="eee7e810-116b-408a-a46d-f51dcec217c2"></script>
|
||||
{{end}}
|
||||
|
||||
Reference in New Issue
Block a user