Compare commits
9 Commits
735adfabea
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| d7829cb5ae | |||
| 8ae9c9e878 | |||
| fcff6f3298 | |||
| e3cd2df372 | |||
| 067d017ea6 | |||
| 3251f27edb | |||
| cf32e669cb | |||
| e613d766ec | |||
| 53608a7195 |
@@ -0,0 +1,6 @@
|
|||||||
|
# Webhook for update/backup notifications (n8n → Telegram)
|
||||||
|
WEBHOOK_URL=https://n8n.example.com/webhook/your-webhook-id
|
||||||
|
|
||||||
|
# Restic backup repository and password
|
||||||
|
RESTIC_REPOSITORY=/mnt/hidrive/users/valknar/Backup/stacks
|
||||||
|
RESTIC_PASSWORD=changeme
|
||||||
@@ -20,10 +20,71 @@ Each stack is independently deployable with its own `compose.yml` and `.env`. Al
|
|||||||
|
|
||||||
## Tools
|
## Tools
|
||||||
|
|
||||||
| Directory | Description |
|
| File | Description |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `_backup` | Daily restic backups to HiDrive (host script + systemd timer) |
|
| `stacks.sh` | CLI to manage stacks, services, scaffolding, updates, and backups |
|
||||||
| `_update` | Nightly image update check + prune (host script + systemd timer) |
|
| `.env` | Root config: `WEBHOOK_URL`, `RESTIC_REPOSITORY`, `RESTIC_PASSWORD` (gitignored) |
|
||||||
|
| `.env.example` | Template for the root `.env` |
|
||||||
|
|
||||||
|
## stacks.sh
|
||||||
|
|
||||||
|
`stacks.sh` is the primary management CLI. It wraps `docker compose` with glob-based multi-stack targeting, manages the update and backup systemd services, scaffolds new stacks, and generates shell completions.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./stacks.sh help
|
||||||
|
```
|
||||||
|
|
||||||
|
**Stack commands** — all accept one or more stack names or glob patterns (omit for all stacks):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./stacks.sh ls # list all stacks with live container status
|
||||||
|
./stacks.sh ps gitea # container status table
|
||||||
|
./stacks.sh up # start all stacks
|
||||||
|
./stacks.sh up gitea traefik # start specific stacks
|
||||||
|
./stacks.sh down 'g*' # stop stacks matching glob
|
||||||
|
./stacks.sh restart 'g*,traefik' # glob + exact name, comma-separated
|
||||||
|
./stacks.sh pull --parallel # pull all images in parallel
|
||||||
|
./stacks.sh logs -f gitea # follow logs
|
||||||
|
./stacks.sh logs -n 100 gitea n8n # tail multiple stacks
|
||||||
|
./stacks.sh exec gitea gitea gitea admin user list # exec in container
|
||||||
|
./stacks.sh run passbolt passbolt bin/cake passbolt healthcheck
|
||||||
|
```
|
||||||
|
|
||||||
|
**Service management** (reads `WEBHOOK_URL`, `RESTIC_REPOSITORY`, `RESTIC_PASSWORD` from root `.env`):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./stacks.sh update install # write & enable systemd update timer
|
||||||
|
./stacks.sh update run # run update now
|
||||||
|
./stacks.sh update status # show timer/service status
|
||||||
|
./stacks.sh update logs # show journal logs
|
||||||
|
|
||||||
|
./stacks.sh backup install # write & enable systemd backup timer
|
||||||
|
./stacks.sh backup run # run backup now (auto-detects <stack>_db containers)
|
||||||
|
./stacks.sh backup snapshots # list restic snapshots
|
||||||
|
```
|
||||||
|
|
||||||
|
**Scaffold a new stack:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./stacks.sh new myapp # basic stack with Traefik labels
|
||||||
|
./stacks.sh new myapp --db postgres # with Postgres service
|
||||||
|
./stacks.sh new myapp --db postgres --redis # with Postgres + Redis
|
||||||
|
./stacks.sh new myapp --no-traefik # expose port instead of Traefik
|
||||||
|
```
|
||||||
|
|
||||||
|
Generates `compose.yml` (with healthchecks, `../.data/` volumes, Traefik labels) and `.env.example`.
|
||||||
|
|
||||||
|
**Shell completion:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Dynamic — discovers stacks at tab-complete time (default)
|
||||||
|
./stacks.sh completion zsh --install
|
||||||
|
|
||||||
|
# Static — bakes current stack list in; useful on the VPS
|
||||||
|
./stacks.sh completion zsh --static --install
|
||||||
|
```
|
||||||
|
|
||||||
|
**Global flags:** `--dry-run`, `--parallel`, `--verbose`, `--quiet`
|
||||||
|
|
||||||
## Deployment
|
## Deployment
|
||||||
|
|
||||||
@@ -44,60 +105,36 @@ All stacks share the external `falcon_network` Docker network for inter-service
|
|||||||
|
|
||||||
## Backup
|
## Backup
|
||||||
|
|
||||||
The `_backup` stack runs a daily restic backup at 3:00 AM. It dumps all Postgres databases, then backs up the entire `.data/` directory to HiDrive. Retention: 7 daily, 4 weekly, 6 monthly snapshots. Notifications go to Telegram via n8n.
|
Runs daily at 3:00 AM via a systemd timer. Detects Postgres databases automatically by convention (`<stack>_db` container, user `<stack>`, database `<stack>`), dumps each one, then runs a full restic backup of `.data/`. Retention: 7 daily, 4 weekly, 6 monthly. Notifications go to Telegram via n8n.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Deploy backup stack
|
# First-time setup on VPS
|
||||||
rsync -avz _backup/ vps:~/stacks/_backup/
|
cp .env.example .env && $EDITOR .env # set RESTIC_REPOSITORY, RESTIC_PASSWORD, WEBHOOK_URL
|
||||||
|
source .env && restic init # initialise restic repo
|
||||||
|
|
||||||
# Initialize restic repo (first time only)
|
./stacks.sh backup install # write & enable systemd unit + timer
|
||||||
ssh vps 'source ~/stacks/_backup/.env && restic init -r /mnt/hidrive/users/valknar/Backup/stacks'
|
./stacks.sh backup run # test run
|
||||||
|
./stacks.sh backup snapshots # list snapshots
|
||||||
# Install systemd units
|
./stacks.sh backup status # timer/service status
|
||||||
ssh vps 'ln -sf ~/stacks/_backup/stacks-backup.service /etc/systemd/system/ && \
|
./stacks.sh backup logs # journald logs
|
||||||
ln -sf ~/stacks/_backup/stacks-backup.timer /etc/systemd/system/ && \
|
|
||||||
systemctl daemon-reload && systemctl enable --now stacks-backup.timer'
|
|
||||||
|
|
||||||
# Manual test run
|
|
||||||
ssh vps '~/stacks/_backup/backup.sh'
|
|
||||||
|
|
||||||
# Check timer status
|
|
||||||
ssh vps 'systemctl status stacks-backup.timer'
|
|
||||||
|
|
||||||
# View snapshots
|
|
||||||
ssh vps 'source ~/stacks/_backup/.env && restic -r /mnt/hidrive/users/valknar/Backup/stacks snapshots'
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Updates
|
## Updates
|
||||||
|
|
||||||
The `_update` script runs nightly at 2:00 AM. It pulls the latest image for every stack, recreates any containers whose image changed, prunes dangling images, and sends a Telegram notification via n8n.
|
Runs nightly at 2:00 AM via a systemd timer. Pulls the latest image for every stack, recreates any container whose image changed, prunes dangling images, and sends a Telegram notification via n8n.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Deploy update stack
|
./stacks.sh update install # write & enable systemd unit + timer
|
||||||
rsync -avz _update/ vps:~/stacks/_update/
|
./stacks.sh update run # test run
|
||||||
ssh vps 'chmod +x ~/stacks/_update/update.sh'
|
./stacks.sh update status # timer/service status
|
||||||
|
./stacks.sh update logs # journald logs
|
||||||
# Install systemd units
|
|
||||||
ssh vps 'ln -sf ~/stacks/_update/stacks-update.service /etc/systemd/system/ && \
|
|
||||||
ln -sf ~/stacks/_update/stacks-update.timer /etc/systemd/system/ && \
|
|
||||||
systemctl daemon-reload && systemctl enable --now stacks-update.timer'
|
|
||||||
|
|
||||||
# Manual test run
|
|
||||||
ssh vps '~/stacks/_update/update.sh'
|
|
||||||
|
|
||||||
# Check timer status
|
|
||||||
ssh vps 'systemctl status stacks-update.timer'
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Notifications
|
## Notifications
|
||||||
|
|
||||||
The update script and the backup script both POST to an n8n webhook, which forwards messages to Telegram.
|
Both update and backup POST to an n8n webhook on completion, which forwards the message to Telegram.
|
||||||
|
|
||||||
The webhook URL is set in:
|
The webhook URL is set via `WEBHOOK_URL` in the root `.env`. Both services point to the same n8n workflow at `https://n8n.pivoine.art`, which accepts `{ "message": "...", "color": "..." }` and forwards it to Telegram.
|
||||||
- `_backup/.env` → `WEBHOOK_URL`
|
|
||||||
- `_update/.env` → `WEBHOOK_URL`
|
|
||||||
|
|
||||||
Both point to the same n8n workflow at `https://n8n.pivoine.art`. The workflow accepts `{ "message": "..." }` and forwards it to Telegram.
|
|
||||||
|
|
||||||
## Data
|
## Data
|
||||||
|
|
||||||
|
|||||||
@@ -1,3 +0,0 @@
|
|||||||
RESTIC_REPOSITORY=/mnt/hidrive/users/valknar/Backup/stacks
|
|
||||||
RESTIC_PASSWORD=change_me
|
|
||||||
WEBHOOK_URL=https://n8n.example.com/webhook/change_me
|
|
||||||
@@ -1,83 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
|
||||||
STACKS_DIR="$(dirname "$SCRIPT_DIR")"
|
|
||||||
DATA_DIR="$STACKS_DIR/.data"
|
|
||||||
DUMP_DIR="$DATA_DIR/backup/dumps"
|
|
||||||
LOG_FILE="$SCRIPT_DIR/backup.log"
|
|
||||||
|
|
||||||
# Load environment
|
|
||||||
set -a
|
|
||||||
source "$SCRIPT_DIR/.env"
|
|
||||||
set +a
|
|
||||||
|
|
||||||
export RESTIC_REPOSITORY RESTIC_PASSWORD
|
|
||||||
|
|
||||||
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }
|
|
||||||
|
|
||||||
notify() {
|
|
||||||
local color="$1" text="$2"
|
|
||||||
curl -sf -o /dev/null -X POST "$WEBHOOK_URL" \
|
|
||||||
-H 'Content-Type: application/json' \
|
|
||||||
-d "{\"message\":\"$text\",\"color\":\"$color\"}"
|
|
||||||
}
|
|
||||||
|
|
||||||
# Truncate log on each run
|
|
||||||
: > "$LOG_FILE"
|
|
||||||
|
|
||||||
log "Starting backup"
|
|
||||||
|
|
||||||
# --- Postgres dumps ---
|
|
||||||
mkdir -p "$DUMP_DIR"
|
|
||||||
|
|
||||||
declare -A DATABASES=(
|
|
||||||
[umami_db]="umami:umami"
|
|
||||||
[gitea_db]="gitea:gitea"
|
|
||||||
[n8n_db]="n8n:n8n"
|
|
||||||
[immich_db]="immich:immich"
|
|
||||||
[coolify_db]="coolify:coolify"
|
|
||||||
[passbolt_db]="passbolt:passbolt"
|
|
||||||
)
|
|
||||||
|
|
||||||
dump_errors=()
|
|
||||||
for container in "${!DATABASES[@]}"; do
|
|
||||||
IFS=: read -r user db <<< "${DATABASES[$container]}"
|
|
||||||
log "Dumping $db from $container"
|
|
||||||
if docker exec "$container" pg_dump -U "$user" "$db" > "$DUMP_DIR/$db.sql" 2>>"$LOG_FILE"; then
|
|
||||||
log " OK ($(du -h "$DUMP_DIR/$db.sql" | cut -f1))"
|
|
||||||
else
|
|
||||||
log " FAILED: $container"
|
|
||||||
dump_errors+=("$db")
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
if [ ${#dump_errors[@]} -gt 0 ]; then
|
|
||||||
log "WARNING: Failed dumps: ${dump_errors[*]}"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# --- Restic backup ---
|
|
||||||
log "Running restic backup"
|
|
||||||
if ! restic backup "$DATA_DIR" --tag stacks 2>&1 | tee -a "$LOG_FILE"; then
|
|
||||||
log "FATAL: restic backup failed"
|
|
||||||
notify "#cc0000" "❌ **Backup failed** — restic backup error. Check \`$LOG_FILE\`."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# --- Restic prune ---
|
|
||||||
log "Pruning old snapshots"
|
|
||||||
if ! restic forget --prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 2>&1 | tee -a "$LOG_FILE"; then
|
|
||||||
log "WARNING: restic forget failed"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# --- Summary ---
|
|
||||||
snapshot_info=$(restic snapshots --latest 1 --compact 2>/dev/null | tail -3 | head -1)
|
|
||||||
repo_stats=$(restic stats 2>/dev/null | grep "Total Size" || true)
|
|
||||||
|
|
||||||
summary="✅ **Backup complete**"
|
|
||||||
[ ${#dump_errors[@]} -gt 0 ] && summary+="\n⚠️ Failed dumps: ${dump_errors[*]}"
|
|
||||||
summary+="\nLatest: \`$snapshot_info\`"
|
|
||||||
[ -n "$repo_stats" ] && summary+="\nRepo: $repo_stats"
|
|
||||||
|
|
||||||
notify "#36a64f" "$summary"
|
|
||||||
log "Backup complete"
|
|
||||||
@@ -1,10 +0,0 @@
|
|||||||
[Unit]
|
|
||||||
Description=Backup stacks data to HiDrive via restic
|
|
||||||
After=network-online.target docker.service
|
|
||||||
Wants=network-online.target
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
Type=oneshot
|
|
||||||
User=root
|
|
||||||
ExecStart=/root/stacks/_backup/backup.sh
|
|
||||||
Environment=HOME=/root
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
[Unit]
|
|
||||||
Description=Daily stacks backup at 3:00 AM
|
|
||||||
|
|
||||||
[Timer]
|
|
||||||
OnCalendar=*-*-* 03:00:00 Europe/Amsterdam
|
|
||||||
Persistent=true
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=timers.target
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
WEBHOOK_URL=https://n8n.example.com/webhook/change_me
|
|
||||||
@@ -1,10 +0,0 @@
|
|||||||
[Unit]
|
|
||||||
Description=Docker stacks image update
|
|
||||||
After=network-online.target docker.service
|
|
||||||
Wants=network-online.target
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
Type=oneshot
|
|
||||||
User=root
|
|
||||||
ExecStart=/root/stacks/_update/update.sh
|
|
||||||
Environment=HOME=/root
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
[Unit]
|
|
||||||
Description=Daily Docker stacks image update at 2:00 AM
|
|
||||||
|
|
||||||
[Timer]
|
|
||||||
OnCalendar=*-*-* 02:00:00 Europe/Amsterdam
|
|
||||||
Persistent=true
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=timers.target
|
|
||||||
@@ -1,62 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
|
||||||
STACKS_DIR="$(dirname "$SCRIPT_DIR")"
|
|
||||||
LOG_FILE="$SCRIPT_DIR/update.log"
|
|
||||||
|
|
||||||
set -a; source "$SCRIPT_DIR/.env"; set +a
|
|
||||||
|
|
||||||
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }
|
|
||||||
notify() {
|
|
||||||
local color="$1" text="$2"
|
|
||||||
curl -sf -o /dev/null -X POST "$WEBHOOK_URL" \
|
|
||||||
-H 'Content-Type: application/json' \
|
|
||||||
-d "{\"message\":\"$text\",\"color\":\"$color\"}"
|
|
||||||
}
|
|
||||||
|
|
||||||
: > "$LOG_FILE"
|
|
||||||
log "Starting image update check"
|
|
||||||
|
|
||||||
updated=(); failed=(); checked=0
|
|
||||||
|
|
||||||
for stack_dir in "$STACKS_DIR"/*/; do
|
|
||||||
stack=$(basename "$stack_dir")
|
|
||||||
[[ "$stack" == _* ]] && continue
|
|
||||||
[[ ! -f "$stack_dir/compose.yml" ]] && continue
|
|
||||||
|
|
||||||
log "Checking $stack"
|
|
||||||
cd "$stack_dir"
|
|
||||||
checked=$((checked + 1))
|
|
||||||
|
|
||||||
before=$(docker compose images -q 2>/dev/null | sort)
|
|
||||||
docker compose pull 2>&1 | tee -a "$LOG_FILE"
|
|
||||||
after=$(docker compose images -q 2>/dev/null | sort)
|
|
||||||
|
|
||||||
if [ "$before" != "$after" ]; then
|
|
||||||
log " Updates found — recreating containers"
|
|
||||||
if docker compose up -d 2>&1 | tee -a "$LOG_FILE"; then
|
|
||||||
updated+=("$stack")
|
|
||||||
else
|
|
||||||
log " FAILED to recreate $stack"
|
|
||||||
failed+=("$stack")
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
log " Up to date"
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
log "Pruning dangling images"
|
|
||||||
docker image prune -f 2>&1 | tee -a "$LOG_FILE"
|
|
||||||
|
|
||||||
date_str=$(date '+%Y-%m-%d %H:%M')
|
|
||||||
|
|
||||||
if [ ${#failed[@]} -gt 0 ]; then
|
|
||||||
notify "#cc0000" "❌ **Update failed** — $date_str\nUpdated: ${updated[*]:-none}\nFailed: ${failed[*]}"
|
|
||||||
elif [ ${#updated[@]} -gt 0 ]; then
|
|
||||||
notify "#36a64f" "✅ **Updates applied** — $date_str\nStacks updated (${#updated[@]}/$checked): ${updated[*]}"
|
|
||||||
else
|
|
||||||
notify "#36a64f" "✅ **All up to date** — $date_str ($checked stacks checked)"
|
|
||||||
fi
|
|
||||||
|
|
||||||
log "Update check complete"
|
|
||||||
@@ -3,5 +3,3 @@ NETWORK_NAME=falcon_network
|
|||||||
RUNNER_TOKEN=change_me
|
RUNNER_TOKEN=change_me
|
||||||
EMAIL_FROM=gitea@pivoine.art
|
EMAIL_FROM=gitea@pivoine.art
|
||||||
EMAIL_USER=Gitea
|
EMAIL_USER=Gitea
|
||||||
UMAMI_SRC=https://umami.example.com/script.js
|
|
||||||
UMAMI_ID=
|
|
||||||
|
|||||||
@@ -36,8 +36,6 @@ services:
|
|||||||
GITEA__ui__THEMES: gitea-auto,gitea-light,gitea-dark,arc-green,edge-auto,edge-dark,edge-light,everforest-auto,everforest-dark,everforest-light,gruvbox-auto,gruvbox-dark,gruvbox-light,gruvbox-material-auto,gruvbox-material-dark,gruvbox-material-light,nord,palenight,soft-era,sonokai,sonokai-andromeda,sonokai-atlantis,sonokai-espresso,sonokai-maia,sonokai-shusia
|
GITEA__ui__THEMES: gitea-auto,gitea-light,gitea-dark,arc-green,edge-auto,edge-dark,edge-light,everforest-auto,everforest-dark,everforest-light,gruvbox-auto,gruvbox-dark,gruvbox-light,gruvbox-material-auto,gruvbox-material-dark,gruvbox-material-light,nord,palenight,soft-era,sonokai,sonokai-andromeda,sonokai-atlantis,sonokai-espresso,sonokai-maia,sonokai-shusia
|
||||||
GITEA__ui__DEFAULT_THEME: nord
|
GITEA__ui__DEFAULT_THEME: nord
|
||||||
GITEA__ui__ENABLE_FEED: "true"
|
GITEA__ui__ENABLE_FEED: "true"
|
||||||
UMAMI_SRC: ${UMAMI_SRC}
|
|
||||||
UMAMI_ID: ${UMAMI_ID}
|
|
||||||
ports:
|
ports:
|
||||||
- "2222:2222"
|
- "2222:2222"
|
||||||
volumes:
|
volumes:
|
||||||
|
|||||||
@@ -1,3 +1,3 @@
|
|||||||
{{if and (eq .RunMode "prod") (not .IsSigned)}}
|
{{if and .RunModeIsProd (not .IsSigned)}}
|
||||||
<script defer src="{{env "UMAMI_SRC"}}" data-website-id="{{env "UMAMI_ID"}}"></script>
|
<script defer src="https://umami.pivoine.art/script.js" data-website-id="eee7e810-116b-408a-a46d-f51dcec217c2"></script>
|
||||||
{{end}}
|
{{end}}
|
||||||
|
|||||||
Reference in New Issue
Block a user