fix(ai): litellm config

feat(ai): bge over litellm
2025-11-30 23:03:32 +01:00 · 2025-11-30 22:32:49 +01:00 · 2025-11-30 20:12:07 +01:00 · 2025-11-28 09:58:44 +01:00 · 2025-11-28 09:53:14 +01:00 · 2025-11-28 09:48:25 +01:00
17 changed files with 1061 additions and 267 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -25,7 +25,7 @@ Root `compose.yaml` uses Docker Compose's `include` directive to orchestrate mul
 - **kit**: Unified toolkit with Vert file converter and miniPaint image editor (path-routed)
 - **jelly**: Jellyfin media server with hardware transcoding
 - **drop**: PairDrop peer-to-peer file sharing
- **ai**: AI infrastructure with Open WebUI, Crawl4AI, and pgvector (PostgreSQL)
+- **ai**: AI infrastructure with Open WebUI, ComfyUI proxy, Crawl4AI, and pgvector (PostgreSQL)
 - **asciinema**: Terminal recording and sharing platform (PostgreSQL)
 - **restic**: Backrest backup system with restic backend
 - **netdata**: Real-time infrastructure monitoring
@@ -451,11 +451,13 @@ AI infrastructure with Open WebUI, Crawl4AI, and dedicated PostgreSQL with pgvec
  - User signup enabled
  - Data persisted in `ai_webui_data` volume

- **crawl4ai**: Crawl4AI web scraping service (internal API, no public access)
-  - Optimized web scraper for LLM content preparation
-  - Internal API on port 11235 (not exposed via Traefik)
-  - Designed for integration with Open WebUI and n8n workflows
-  - Data persisted in `ai_crawl4ai_data` volume
+- **comfyui**: ComfyUI reverse proxy exposed at `comfy.ai.pivoine.art:80`
+  - Nginx-based proxy to ComfyUI running on RunPod GPU server
+  - Node-based UI for Flux.1 Schnell image generation workflows
+  - Proxies to RunPod via Tailscale VPN (100.121.199.88:8188)
+  - Protected by Authelia SSO authentication
+  - WebSocket support for real-time updates
+  - Stateless architecture (no data persistence on VPS)

 **Configuration**:
 - **Claude Integration**: Uses Anthropic API with OpenAI-compatible endpoint
@@ -476,11 +478,71 @@ AI infrastructure with Open WebUI, Crawl4AI, and dedicated PostgreSQL with pgvec
 4. Use web search feature for current information
 5. Integrate with n8n workflows for automation

+**Flux Image Generation** (`functions/flux_image_gen.py`):
+Open WebUI function for generating images via Flux.1 Schnell on RunPod GPU:
+- Manifold function adds "Flux.1 Schnell (4-5s)" model to Open WebUI
+- Routes requests through LiteLLM → Orchestrator → RunPod Flux
+- Generates 1024x1024 images in 4-5 seconds
+- Returns images as base64-encoded markdown
+- Configuration via Valves (API base, timeout, default size)
+- **Automatically loaded via Docker volume mount** (`./functions:/app/backend/data/functions:ro`)
+
+**Deployment**:
+- Function file tracked in `ai/functions/` directory
+- Automatically available after `pnpm arty up -d ai_webui`
+- No manual import required - infrastructure as code
+
+See `ai/FLUX_SETUP.md` for detailed setup instructions and troubleshooting.
+
+**ComfyUI Image Generation**:
+ComfyUI provides a professional node-based interface for creating Flux image generation workflows:
+
+**Architecture**:
+```
+User → Traefik (VPS) → Authelia SSO → ComfyUI Proxy (nginx) → Tailscale → ComfyUI (RunPod:8188) → Flux Model (GPU)
+```
+
+**Access**:
+1. Navigate to https://comfy.ai.pivoine.art
+2. Authenticate via Authelia SSO
+3. Create node-based workflows in ComfyUI interface
+4. Use Flux.1 Schnell model from HuggingFace cache at `/workspace/ComfyUI/models/huggingface_cache`
+
+**RunPod Setup** (via Ansible):
+ComfyUI is installed on RunPod using the Ansible playbook at `/home/valknar/Projects/runpod/playbook.yml`:
+- Clone ComfyUI from https://github.com/comfyanonymous/ComfyUI
+- Install dependencies from `models/comfyui/requirements.txt`
+- Create model directory structure (checkpoints, unet, vae, loras, clip, controlnet)
+- Symlink Flux model from HuggingFace cache
+- Start service via `models/comfyui/start.sh` on port 8188
+
+**To deploy ComfyUI on RunPod**:
+```bash
+# Run Ansible playbook with comfyui tag
+ssh -p 16186 root@213.173.110.150
+cd /workspace/ai
+ansible-playbook playbook.yml --tags comfyui --skip-tags always
+
+# Start ComfyUI service
+bash models/comfyui/start.sh &
+```
+
+**Proxy Configuration**:
+The VPS runs an nginx proxy (`ai/comfyui-nginx.conf`) that:
+- Listens on port 80 inside container
+- Forwards to RunPod via Tailscale (100.121.199.88:8188)
+- Supports WebSocket upgrades for real-time updates
+- Handles large file uploads (100M limit)
+- Uses extended timeouts for long-running generations (300s)
+
+**Note**: ComfyUI runs directly on RunPod GPU server, not in a container. All data is stored on RunPod's `/workspace` volume.
+
 **Integration Points**:
 - **n8n**: Workflow automation with AI tasks (scraping, RAG ingestion, webhooks)
 - **Mattermost**: Can send AI-generated notifications via webhooks
 - **Crawl4AI**: Internal API for advanced web scraping
 - **Claude API**: Primary LLM provider via Anthropic
+- **Flux via RunPod**: Image generation through orchestrator (GPU server) or ComfyUI

 **Future Enhancements**:
 - GPU server integration (IONOS A10 planned)
@@ -659,7 +721,7 @@ Backrest backup system with restic backend:
    - Retention: 7 daily, 4 weekly, 3 monthly

 16. **ai-backup** (3 AM daily)
-    - Paths: `/volumes/ai_postgres_data`, `/volumes/ai_webui_data`, `/volumes/ai_crawl4ai_data`
+    - Paths: `/volumes/ai_postgres_data`, `/volumes/ai_webui_data`
    - Retention: 7 daily, 4 weekly, 6 monthly, 2 yearly

 17. **asciinema-backup** (11 AM daily)
@@ -670,8 +732,7 @@ Backrest backup system with restic backend:
 All Docker volumes are mounted read-only to `/volumes/` with prefixed names (e.g., `backup_core_postgres_data`) to avoid naming conflicts with other compose stacks.

 **Configuration Management**:
- `config.json` template in repository defines all backup plans
- On first run, copy config into volume: `docker cp restic/config.json restic_app:/config/config.json`
+- `core/backrest/config.json` in repository defines all backup plans (bind-mounted to container)
 - Config version must be `4` for Backrest 1.10.1 compatibility
 - Backrest manages auth automatically (username: `valknar`, password set via web UI on first access)

@@ -709,7 +770,7 @@ Each service uses named volumes prefixed with project name:
 - `vault_data`: Vaultwarden password vault (SQLite database)
 - `joplin_data`: Joplin note-taking data
 - `jelly_config`: Jellyfin media server configuration
- `ai_postgres_data`, `ai_webui_data`, `ai_crawl4ai_data`: AI stack databases and application data
+- `ai_postgres_data`, `ai_webui_data`: AI stack databases and application data
 - `netdata_config`: Netdata monitoring configuration
 - `restic_data`, `restic_config`, `restic_cache`, `restic_tmp`: Backrest backup system
 - `proxy_letsencrypt_data`: SSL certificates
--- a/README.md
+++ b/README.md
@@ -406,11 +406,10 @@ THE FALCON (falcon_network)
 │  ├─ vaultwarden        [vault.pivoine.art] → Password Manager
 │  └─ tandoor            [tandoor.pivoine.art] → Recipe Manager
 │
-├─ 🤖 AI STACK (5 services)
+├─ 🤖 AI STACK (4 services)
 │  ├─ ai_postgres        [Internal] → pgvector Database
 │  ├─ webui              [ai.pivoine.art] → Open WebUI (Claude)
 │  ├─ litellm            [llm.ai.pivoine.art] → API Proxy
-│  ├─ crawl4ai           [Internal:11235] → Web Scraper
 │  └─ facefusion         [facefusion.ai.pivoine.art] → Face AI
 │
 ├─ 🛡️ NET STACK (4 services)
@@ -435,7 +434,7 @@ THE FALCON (falcon_network)
   ├─ Core: postgres_data, redis_data, backrest_*
   ├─ Sexy: directus_uploads, directus_bundle
   ├─ Util: pairdrop_*, joplin_data, linkwarden_*, mattermost_*, vaultwarden_data, tandoor_*
-   ├─ AI: ai_postgres_data, ai_webui_data, ai_crawl4ai_data, facefusion_*
+   ├─ AI: ai_postgres_data, ai_webui_data, facefusion_*
   ├─ Net: letsencrypt_data, netdata_*
   ├─ Media: jelly_config, jelly_cache, filestash_data
   └─ Dev: gitea_*, coolify_data, n8n_data, asciinema_data
--- a/ai/compose.yaml
+++ b/ai/compose.yaml
@@ -15,7 +15,7 @@ services:
      - ai_postgres_data:/var/lib/postgresql/data
      - ./postgres/init:/docker-entrypoint-initdb.d
    healthcheck:
-      test: ['CMD-SHELL', 'pg_isready -U ${AI_DB_USER}']
+      test: ["CMD-SHELL", "pg_isready -U ${AI_DB_USER}"]
      interval: 30s
      timeout: 10s
      retries: 3
@@ -38,6 +38,10 @@ services:
      OPENAI_API_BASE_URLS: http://litellm:4000
      OPENAI_API_KEYS: ${AI_LITELLM_API_KEY}

+      # Disable Ollama (we only use LiteLLM)
+      ENABLE_OLLAMA_API: false
+      OLLAMA_BASE_URLS: ""
+
      # WebUI configuration
      WEBUI_NAME: ${AI_WEBUI_NAME:-Pivoine AI}
      WEBUI_URL: https://${AI_TRAEFIK_HOST}
@@ -62,100 +66,87 @@ services:

    volumes:
      - ai_webui_data:/app/backend/data
+      - ./functions:/app/backend/data/functions:ro
    depends_on:
      - ai_postgres
      - litellm
    networks:
      - compose_network
    labels:
-      - 'traefik.enable=${AI_TRAEFIK_ENABLED}'
+      - "traefik.enable=${AI_TRAEFIK_ENABLED}"
      # HTTP to HTTPS redirect
-      - 'traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-redirect-web-secure.redirectscheme.scheme=https'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web.middlewares=${AI_COMPOSE_PROJECT_NAME}-redirect-web-secure'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web.rule=Host(`${AI_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web.entrypoints=web'
+      - "traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-redirect-web-secure.redirectscheme.scheme=https"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web.middlewares=${AI_COMPOSE_PROJECT_NAME}-redirect-web-secure"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web.rule=Host(`${AI_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web.entrypoints=web"
      # HTTPS router
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web-secure.rule=Host(`${AI_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web-secure.tls.certresolver=resolver'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web-secure.entrypoints=web-secure'
-      - 'traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-web-secure-compress.compress=true'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web-secure.middlewares=${AI_COMPOSE_PROJECT_NAME}-web-secure-compress,security-headers@file'
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web-secure.rule=Host(`${AI_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web-secure.tls.certresolver=resolver"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web-secure.entrypoints=web-secure"
+      - "traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-web-secure-compress.compress=true"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web-secure.middlewares=${AI_COMPOSE_PROJECT_NAME}-web-secure-compress,security-headers@file"
      # Service
-      - 'traefik.http.services.${AI_COMPOSE_PROJECT_NAME}-web-secure.loadbalancer.server.port=8080'
-      - 'traefik.docker.network=${NETWORK_NAME}'
+      - "traefik.http.services.${AI_COMPOSE_PROJECT_NAME}-web-secure.loadbalancer.server.port=8080"
+      - "traefik.docker.network=${NETWORK_NAME}"
      # Watchtower
-      - 'com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}'
+      - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}"

  # LiteLLM - Proxy to convert Anthropic API to OpenAI-compatible format
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: ${AI_COMPOSE_PROJECT_NAME}_litellm
    restart: unless-stopped
+    dns:
+      - 100.100.100.100
+      - 8.8.8.8
    environment:
      TZ: ${TIMEZONE:-Europe/Berlin}
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
      LITELLM_MASTER_KEY: ${AI_LITELLM_API_KEY}
      DATABASE_URL: postgresql://${AI_DB_USER}:${AI_DB_PASSWORD}@ai_postgres:5432/litellm
-      LITELLM_DROP_PARAMS: 'true'
-      NO_DOCS: 'true'
-      NO_REDOC: 'true'
+      GPU_VLLM_LLAMA_URL: ${GPU_VLLM_LLAMA_URL}
+      GPU_VLLM_BGE_URL: ${GPU_VLLM_BGE_URL}
+      # LITELLM_DROP_PARAMS: 'true'  # DISABLED: Was breaking streaming
+      NO_DOCS: "true"
+      NO_REDOC: "true"
+      # Performance optimizations
+      LITELLM_LOG: "DEBUG" # Enable detailed logging for debugging streaming issues
+      LITELLM_MODE: "PRODUCTION" # Production mode for better performance
    volumes:
      - ./litellm-config.yaml:/app/litellm-config.yaml:ro
    command:
      [
-        '--config',
-        '/app/litellm-config.yaml',
-        '--host',
-        '0.0.0.0',
-        '--port',
-        '4000',
-        '--detailed_debug',
-        '--drop_params'
+        "--config",
+        "/app/litellm-config.yaml",
+        "--host",
+        "0.0.0.0",
+        "--port",
+        "4000",
      ]
    depends_on:
      - ai_postgres
-    networks:
-      - compose_network
    healthcheck:
      disable: true
-    labels:
-      - 'traefik.enable=${AI_TRAEFIK_ENABLED}'
-      # HTTP to HTTPS redirect
-      - 'traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-litellm-redirect-web-secure.redirectscheme.scheme=https'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web.middlewares=${AI_COMPOSE_PROJECT_NAME}-litellm-redirect-web-secure'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web.rule=Host(`${AI_LITELLM_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web.entrypoints=web'
-      # HTTPS router
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure.rule=Host(`${AI_LITELLM_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure.tls.certresolver=resolver'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure.entrypoints=web-secure'
-      - 'traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure-compress.compress=true'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure.middlewares=${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure-compress,security-headers@file'
-      # Service
-      - 'traefik.http.services.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure.loadbalancer.server.port=4000'
-      - 'traefik.docker.network=${NETWORK_NAME}'
-      # Watchtower
-      - 'com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}'
-
-  # Crawl4AI - Web scraping for LLMs (internal API, no public access)
-  crawl4ai:
-    image: ${AI_CRAWL4AI_IMAGE:-unclecode/crawl4ai:latest}
-    container_name: ${AI_COMPOSE_PROJECT_NAME}_crawl4ai
-    restart: unless-stopped
-    environment:
-      TZ: ${TIMEZONE:-Europe/Berlin}
-      # API configuration
-      PORT: ${AI_CRAWL4AI_PORT:-11235}
-    volumes:
-      - ai_crawl4ai_data:/app/.crawl4ai
    networks:
      - compose_network
    labels:
-      # No Traefik exposure - internal only
-      - 'traefik.enable=false'
+      - "traefik.enable=${AI_TRAEFIK_ENABLED}"
+      # HTTP to HTTPS redirect
+      - "traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-litellm-redirect-web-secure.redirectscheme.scheme=https"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web.middlewares=${AI_COMPOSE_PROJECT_NAME}-litellm-redirect-web-secure"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web.rule=Host(`${AI_LITELLM_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web.entrypoints=web"
+      # HTTPS router
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure.rule=Host(`${AI_LITELLM_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure.tls.certresolver=resolver"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure.entrypoints=web-secure"
+      - "traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure-compress.compress=true"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure.middlewares=${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure-compress,security-headers@file"
+      # Service
+      - "traefik.http.services.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure.loadbalancer.server.port=4000"
+      - "traefik.docker.network=${NETWORK_NAME}"
      # Watchtower
-      - 'com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}'
-
+      - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}"
  # Facefusion - AI face swapping and enhancement
  facefusion:
    build:
@@ -165,7 +156,7 @@ services:
    container_name: ${AI_COMPOSE_PROJECT_NAME}_facefusion
    restart: unless-stopped
    tty: true
-    command: ['python', '-u', 'facefusion.py', 'run']
+    command: ["python", "-u", "facefusion.py", "run"]
    environment:
      TZ: ${TIMEZONE:-Europe/Berlin}
      GRADIO_SERVER_NAME: "0.0.0.0"
@@ -175,32 +166,175 @@ services:
    networks:
      - compose_network
    labels:
-      - 'traefik.enable=${AI_FACEFUSION_TRAEFIK_ENABLED}'
-      # HTTP Basic Auth middleware
-      - 'traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-facefusion-auth.basicauth.users=${AUTH_USERS}'
+      - "traefik.enable=${AI_FACEFUSION_TRAEFIK_ENABLED}"
      # HTTP to HTTPS redirect
-      - 'traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-facefusion-redirect-web-secure.redirectscheme.scheme=https'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web.middlewares=${AI_COMPOSE_PROJECT_NAME}-facefusion-redirect-web-secure'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web.rule=Host(`${AI_FACEFUSION_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web.entrypoints=web'
-      # HTTPS router with auth
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure.rule=Host(`${AI_FACEFUSION_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure.tls.certresolver=resolver'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure.entrypoints=web-secure'
-      - 'traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure-compress.compress=true'
-      - 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure.middlewares=${AI_COMPOSE_PROJECT_NAME}-facefusion-auth,${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure-compress,security-headers@file'
+      - "traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-facefusion-redirect-web-secure.redirectscheme.scheme=https"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web.middlewares=${AI_COMPOSE_PROJECT_NAME}-facefusion-redirect-web-secure"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web.rule=Host(`${AI_FACEFUSION_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web.entrypoints=web"
+      # HTTPS router with Authelia
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure.rule=Host(`${AI_FACEFUSION_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure.tls.certresolver=resolver"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure.entrypoints=web-secure"
+      - "traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure-compress.compress=true"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure.middlewares=${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure-compress,net-authelia,security-headers@file"
      # Service
-      - 'traefik.http.services.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure.loadbalancer.server.port=7860'
-      - 'traefik.docker.network=${NETWORK_NAME}'
+      - "traefik.http.services.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure.loadbalancer.server.port=7860"
+      - "traefik.docker.network=${NETWORK_NAME}"
      # Watchtower - disabled for custom local image
-      - 'com.centurylinklabs.watchtower.enable=false'
+      - "com.centurylinklabs.watchtower.enable=false"
+
+  # ComfyUI - Node-based UI for Flux image generation (proxies to RunPod GPU)
+  comfyui:
+    image: nginx:alpine
+    container_name: ${AI_COMPOSE_PROJECT_NAME}_comfyui
+    restart: unless-stopped
+    dns:
+      - 100.100.100.100
+      - 8.8.8.8
+    environment:
+      TZ: ${TIMEZONE:-Europe/Berlin}
+      GPU_SERVICE_HOST: ${GPU_TAILSCALE_HOST:-runpod-ai-orchestrator}
+      GPU_SERVICE_PORT: ${COMFYUI_BACKEND_PORT:-8188}
+    volumes:
+      - ./nginx.conf.template:/etc/nginx/nginx.conf.template:ro
+    command: /bin/sh -c "envsubst '$${GPU_SERVICE_HOST},$${GPU_SERVICE_PORT}' < /etc/nginx/nginx.conf.template > /etc/nginx/nginx.conf && exec nginx -g 'daemon off;'"
+    networks:
+      - compose_network
+    labels:
+      - "traefik.enable=${AI_COMFYUI_TRAEFIK_ENABLED:-true}"
+      # HTTP to HTTPS redirect
+      - "traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-comfyui-redirect-web-secure.redirectscheme.scheme=https"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-comfyui-web.middlewares=${AI_COMPOSE_PROJECT_NAME}-comfyui-redirect-web-secure"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-comfyui-web.rule=Host(`${AI_COMFYUI_TRAEFIK_HOST:-comfy.ai.pivoine.art}`)"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-comfyui-web.entrypoints=web"
+      # HTTPS router with Authelia SSO
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-comfyui-web-secure.rule=Host(`${AI_COMFYUI_TRAEFIK_HOST:-comfy.ai.pivoine.art}`)"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-comfyui-web-secure.tls.certresolver=resolver"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-comfyui-web-secure.entrypoints=web-secure"
+      - "traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-comfyui-web-secure-compress.compress=true"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-comfyui-web-secure.middlewares=${AI_COMPOSE_PROJECT_NAME}-comfyui-web-secure-compress,net-authelia,security-headers@file"
+      # Service
+      - "traefik.http.services.${AI_COMPOSE_PROJECT_NAME}-comfyui-web-secure.loadbalancer.server.port=80"
+      - "traefik.docker.network=${NETWORK_NAME}"
+      # Watchtower
+      - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}"
+
+  audiocraft:
+    image: nginx:alpine
+    container_name: ${AI_COMPOSE_PROJECT_NAME}_audiocraft
+    restart: unless-stopped
+    dns:
+      - 100.100.100.100
+      - 8.8.8.8
+    environment:
+      TZ: ${TIMEZONE:-Europe/Berlin}
+      GPU_SERVICE_HOST: ${GPU_TAILSCALE_HOST:-runpod-ai-orchestrator}
+      GPU_SERVICE_PORT: ${AUDIOCRAFT_BACKEND_PORT:-7860}
+    volumes:
+      - ./nginx.conf.template:/etc/nginx/nginx.conf.template:ro
+    command: /bin/sh -c "envsubst '$${GPU_SERVICE_HOST},$${GPU_SERVICE_PORT}' < /etc/nginx/nginx.conf.template > /etc/nginx/nginx.conf && exec nginx -g 'daemon off;'"
+    networks:
+      - compose_network
+    labels:
+      - "traefik.enable=${AI_AUDIOCRAFT_TRAEFIK_ENABLED:-true}"
+      # HTTP to HTTPS redirect
+      - "traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-audiocraft-redirect-web-secure.redirectscheme.scheme=https"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-audiocraft-web.middlewares=${AI_COMPOSE_PROJECT_NAME}-audiocraft-redirect-web-secure"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-audiocraft-web.rule=Host(`${AI_AUDIOCRAFT_TRAEFIK_HOST:-audiocraft.ai.pivoine.art}`)"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-audiocraft-web.entrypoints=web"
+      # HTTPS router with Authelia SSO
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-audiocraft-web-secure.rule=Host(`${AI_AUDIOCRAFT_TRAEFIK_HOST:-audiocraft.ai.pivoine.art}`)"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-audiocraft-web-secure.tls.certresolver=resolver"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-audiocraft-web-secure.entrypoints=web-secure"
+      - "traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-audiocraft-web-secure-compress.compress=true"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-audiocraft-web-secure.middlewares=${AI_COMPOSE_PROJECT_NAME}-audiocraft-web-secure-compress,net-authelia,security-headers@file"
+      # Service
+      - "traefik.http.services.${AI_COMPOSE_PROJECT_NAME}-audiocraft-web-secure.loadbalancer.server.port=80"
+      - "traefik.docker.network=${NETWORK_NAME}"
+      # Watchtower
+      - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}"
+
+  upscale:
+    image: nginx:alpine
+    container_name: ${AI_COMPOSE_PROJECT_NAME}_upscale
+    restart: unless-stopped
+    dns:
+      - 100.100.100.100
+      - 8.8.8.8
+    environment:
+      TZ: ${TIMEZONE:-Europe/Berlin}
+      GPU_SERVICE_HOST: ${GPU_TAILSCALE_HOST:-runpod-ai-orchestrator}
+      GPU_SERVICE_PORT: ${UPSCALE_BACKEND_PORT:-8080}
+    volumes:
+      - ./nginx.conf.template:/etc/nginx/nginx.conf.template:ro
+    command: /bin/sh -c "envsubst '$${GPU_SERVICE_HOST},$${GPU_SERVICE_PORT}' < /etc/nginx/nginx.conf.template > /etc/nginx/nginx.conf && exec nginx -g 'daemon off;'"
+    networks:
+      - compose_network
+    labels:
+      - "traefik.enable=${AI_UPSCALE_TRAEFIK_ENABLED:-true}"
+      # HTTP to HTTPS redirect
+      - "traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-upscale-redirect-web-secure.redirectscheme.scheme=https"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-upscale-web.middlewares=${AI_COMPOSE_PROJECT_NAME}-upscale-redirect-web-secure"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-upscale-web.rule=Host(`${AI_UPSCALE_TRAEFIK_HOST:-upscale.ai.pivoine.art}`)"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-upscale-web.entrypoints=web"
+      # HTTPS router with Authelia SSO
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-upscale-web-secure.rule=Host(`${AI_UPSCALE_TRAEFIK_HOST:-upscale.ai.pivoine.art}`)"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-upscale-web-secure.tls.certresolver=resolver"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-upscale-web-secure.entrypoints=web-secure"
+      - "traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-upscale-web-secure-compress.compress=true"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-upscale-web-secure.middlewares=${AI_COMPOSE_PROJECT_NAME}-upscale-web-secure-compress,net-authelia,security-headers@file"
+      # Service
+      - "traefik.http.services.${AI_COMPOSE_PROJECT_NAME}-upscale-web-secure.loadbalancer.server.port=80"
+      - "traefik.docker.network=${NETWORK_NAME}"
+      # Watchtower
+      - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}"
+
+  # Supervisor UI - Modern web interface for RunPod process management
+  supervisor:
+    image: dev.pivoine.art/valknar/supervisor-ui:latest
+    container_name: ${AI_COMPOSE_PROJECT_NAME}_supervisor_ui
+    restart: unless-stopped
+    dns:
+      - 100.100.100.100
+      - 8.8.8.8
+    environment:
+      TZ: ${TIMEZONE:-Europe/Berlin}
+      NODE_ENV: production
+      # Connect to RunPod Supervisor via Tailscale (host Tailscale provides DNS)
+      SUPERVISOR_HOST: ${GPU_TAILSCALE_HOST:-runpod-ai-orchestrator}
+      SUPERVISOR_PORT: ${SUPERVISOR_BACKEND_PORT:-9001}
+      # No auth needed - Supervisor has auth disabled (protected by Authelia)
+    networks:
+      - compose_network
+    labels:
+      - "traefik.enable=${AI_SUPERVISOR_TRAEFIK_ENABLED:-true}"
+      # HTTP to HTTPS redirect
+      - "traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-supervisor-redirect-web-secure.redirectscheme.scheme=https"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-supervisor-web.middlewares=${AI_COMPOSE_PROJECT_NAME}-supervisor-redirect-web-secure"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-supervisor-web.rule=Host(`${AI_SUPERVISOR_TRAEFIK_HOST:-supervisor.ai.pivoine.art}`)"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-supervisor-web.entrypoints=web"
+      # HTTPS router with Authelia SSO
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-supervisor-web-secure.rule=Host(`${AI_SUPERVISOR_TRAEFIK_HOST:-supervisor.ai.pivoine.art}`)"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-supervisor-web-secure.tls.certresolver=resolver"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-supervisor-web-secure.entrypoints=web-secure"
+      - "traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-supervisor-web-secure-compress.compress=true"
+      - "traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-supervisor-web-secure.middlewares=${AI_COMPOSE_PROJECT_NAME}-supervisor-web-secure-compress,net-authelia,security-headers@file"
+      # Service (port 3000 for Next.js app)
+      - "traefik.http.services.${AI_COMPOSE_PROJECT_NAME}-supervisor-web-secure.loadbalancer.server.port=3000"
+      - "traefik.docker.network=${NETWORK_NAME}"
+      # Watchtower
+      - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}"

 volumes:
  ai_postgres_data:
    name: ${AI_COMPOSE_PROJECT_NAME}_postgres_data
  ai_webui_data:
    name: ${AI_COMPOSE_PROJECT_NAME}_webui_data
-  ai_crawl4ai_data:
-    name: ${AI_COMPOSE_PROJECT_NAME}_crawl4ai_data
  ai_facefusion_data:
    name: ${AI_COMPOSE_PROJECT_NAME}_facefusion_data
+
+networks:
+  compose_network:
+    name: ${NETWORK_NAME}
+    external: true
--- a/ai/litellm-config.yaml
+++ b/ai/litellm-config.yaml
@@ -8,8 +8,6 @@ model_list:
    litellm_params:
      model: anthropic/claude-sonnet-4-5-20250929
      api_key: os.environ/ANTHROPIC_API_KEY
-      drop_params: true
-      additional_drop_params: ["prompt_cache_key"]

  - model_name: claude-3-5-sonnet
    litellm_params:
@@ -26,24 +24,63 @@ model_list:
      model: anthropic/claude-3-haiku-20240307
      api_key: os.environ/ANTHROPIC_API_KEY

+  # ===========================================================================
+  # SELF-HOSTED MODELS - DIRECT vLLM SERVERS (GPU Server via Tailscale VPN)
+  # ===========================================================================
+  # Direct connections to dedicated vLLM servers (no orchestrator)
+
+  # Text Generation - Llama 3.1 8B (Port 8001)
+  - model_name: llama-3.1-8b
+    litellm_params:
+      model: hosted_vllm/meta-llama/Llama-3.1-8B-Instruct # hosted_vllm/openai/ prefix for proper streaming
+      api_base: os.environ/GPU_VLLM_LLAMA_URL # Direct to vLLM Llama server
+      api_key: "EMPTY" # vLLM doesn't validate API keys
+      rpm: 1000
+      tpm: 100000
+      timeout: 600 # 10 minutes for generation
+      stream_timeout: 600
+      supports_system_messages: true # Llama supports system messages
+      stream: true # Enable streaming by default
+
+  # Embeddings - BGE Large (Port 8002)
+  - model_name: bge-large-en
+    litellm_params:
+      model: openai/BAAI/bge-large-en-v1.5
+      api_base: os.environ/GPU_VLLM_BGE_URL
+      api_key: "EMPTY"
+      rpm: 1000
+      tpm: 500000
+
 litellm_settings:
-  drop_params: true
-  set_verbose: true
-  # Disable prompt caching features
-  cache: false
+  drop_params: false # DISABLED: Was breaking streaming
+  set_verbose: true # Enable verbose logging for debugging streaming issues
+  # Enable caching now that streaming is fixed
+  cache: true
+  cache_params:
+    type: redis
+    host: core_redis
+    port: 6379
+    ttl: 3600 # Cache for 1 hour
  # Force strip specific parameters globally
  allowed_fails: 0
  # Modify params before sending to provider
-  modify_params: true
-  # Drop prompt_cache_key globally for all models
-  additional_drop_params: ["prompt_cache_key"]
+  modify_params: false # DISABLED: Was breaking streaming
+  # Enable success and failure logging but minimize overhead
+  success_callback: [] # Disable all success callbacks to reduce DB writes
+  failure_callback: [] # Disable all failure callbacks

 router_settings:
  allowed_fails: 0

 # Drop unsupported parameters
 default_litellm_params:
-  drop_params: true
+  drop_params: false # DISABLED: Was breaking streaming

 general_settings:
  disable_responses_id_security: true
+  # Disable spend tracking to reduce database overhead
+  disable_spend_logs: true
+  # Disable tag tracking
+  disable_tag_tracking: true
+  # Disable daily spend updates
+  disable_daily_spend_logs: true
--- a/ai/nginx.conf.template
+++ b/ai/nginx.conf.template
@@ -0,0 +1,60 @@
+events {
+    worker_connections 1024;
+}
+
+http {
+    # MIME types
+    include /etc/nginx/mime.types;
+    default_type application/octet-stream;
+
+    # DNS resolver for Tailscale MagicDNS
+    resolver 100.100.100.100 8.8.8.8 valid=30s;
+    resolver_timeout 5s;
+
+    # Proxy settings
+    proxy_http_version 1.1;
+    proxy_buffering off;
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    proxy_set_header X-Forwarded-Proto $scheme;
+
+    # WebSocket support
+    proxy_set_header Upgrade $http_upgrade;
+    proxy_set_header Connection "upgrade";
+
+    # Timeouts for long-running audio/image generation
+    proxy_connect_timeout 600;
+    proxy_send_timeout 600;
+    proxy_read_timeout 600;
+    send_timeout 600;
+
+    server {
+        listen 80;
+        server_name _;
+
+        # Increase client body size for image uploads
+        client_max_body_size 100M;
+
+        location / {
+            # Proxy to service on RunPod via Tailscale
+            # Use variable to force runtime DNS resolution (not startup)
+            set $backend http://${GPU_SERVICE_HOST}:${GPU_SERVICE_PORT};
+            proxy_pass $backend;
+
+            # WebSocket upgrade
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+
+            # Proxy headers
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+
+            # Disable buffering for real-time updates
+            proxy_buffering off;
+        }
+    }
+}
--- a/arty.yml
+++ b/arty.yml
@@ -78,7 +78,7 @@ envs:
    UTIL_JOPLIN_DB_NAME: joplin
    # PairDrop
    UTIL_DROP_TRAEFIK_HOST: drop.pivoine.art
-    # Media Stack (Jellyfin, Filestash)
+    # Media Stack (Jellyfin, Filestash, Pinchflat)
    MEDIA_TRAEFIK_ENABLED: true
    MEDIA_COMPOSE_PROJECT_NAME: media
    MEDIA_JELLYFIN_IMAGE: jellyfin/jellyfin:latest
@@ -86,6 +86,8 @@ envs:
    MEDIA_FILESTASH_IMAGE: machines/filestash:latest
    MEDIA_FILESTASH_TRAEFIK_HOST: filestash.media.pivoine.art
    MEDIA_FILESTASH_CANARY: true
+    MEDIA_PINCHFLAT_IMAGE: ghcr.io/kieraneglin/pinchflat:latest
+    MEDIA_PINCHFLAT_TRAEFIK_HOST: pinchflat.media.pivoine.art
    # Dev (Gitea + Coolify)
    DEV_TRAEFIK_ENABLED: true
    DEV_COMPOSE_PROJECT_NAME: dev
@@ -133,7 +135,6 @@ envs:
    AI_COMPOSE_PROJECT_NAME: ai
    AI_POSTGRES_IMAGE: pgvector/pgvector:pg16
    AI_WEBUI_IMAGE: ghcr.io/open-webui/open-webui:main
-    AI_CRAWL4AI_IMAGE: unclecode/crawl4ai:latest
    AI_FACEFUSION_IMAGE: facefusion/facefusion:3.5.0-cpu
    AI_FACEFUSION_TRAEFIK_ENABLED: true
    AI_FACEFUSION_TRAEFIK_HOST: facefusion.ai.pivoine.art
@@ -261,3 +262,57 @@ scripts:
    docker restart sexy_api &&
    echo "✓ Directus API restarted"
  net/create: docker network create "$NETWORK_NAME"
+  # Setup iptables NAT for Docker containers to reach Tailscale network
+  # Requires Tailscale installed on host: curl -fsSL https://tailscale.com/install.sh | sh
+  tailscale/setup: |
+    echo "Setting up iptables for Docker-to-Tailscale routing..."
+
+    # Enable IP forwarding
+    sudo sysctl -w net.ipv4.ip_forward=1
+    grep -q "net.ipv4.ip_forward=1" /etc/sysctl.conf || echo "net.ipv4.ip_forward=1" | sudo tee -a /etc/sysctl.conf
+
+    # Get Docker network CIDR
+    DOCKER_CIDR=$(docker network inspect ${NETWORK_NAME} --format '{{range .IPAM.Config}}{{.Subnet}}{{end}}' 2>/dev/null || echo "172.18.0.0/16")
+    echo "Docker network CIDR: $DOCKER_CIDR"
+
+    # Add NAT rule (check if already exists)
+    if ! sudo iptables -t nat -C POSTROUTING -s "$DOCKER_CIDR" -o tailscale0 -j MASQUERADE 2>/dev/null; then
+      sudo iptables -t nat -A POSTROUTING -s "$DOCKER_CIDR" -o tailscale0 -j MASQUERADE
+      echo "✓ iptables NAT rule added"
+    else
+      echo "✓ iptables NAT rule already exists"
+    fi
+
+    # Persist rules
+    sudo netfilter-persistent save 2>/dev/null || echo "Install iptables-persistent to persist rules: sudo apt install iptables-persistent"
+
+    echo "✓ Tailscale routing configured"
+
+  # Install and configure Tailscale on host with persistent state
+  tailscale/install: |
+    echo "Installing Tailscale..."
+
+    # Install Tailscale if not present
+    if ! command -v tailscale &> /dev/null; then
+      curl -fsSL https://tailscale.com/install.sh | sh
+    else
+      echo "✓ Tailscale already installed"
+    fi
+
+    # Create state directory for persistence
+    TAILSCALE_STATE="/var/lib/tailscale"
+    sudo mkdir -p "$TAILSCALE_STATE"
+
+    # Start and enable tailscaled service
+    sudo systemctl enable --now tailscaled
+
+    # Connect to Tailscale network
+    echo "Connecting to Tailscale..."
+    sudo tailscale up --authkey="$TAILSCALE_AUTHKEY" --hostname=vps
+
+    # Show status
+    echo ""
+    tailscale status
+    echo ""
+    echo "✓ Tailscale installed and connected"
+    echo "  Run 'arty tailscale/setup' to configure iptables routing for Docker"
--- a/core/backrest/config.json
+++ b/core/backrest/config.json
@@ -0,0 +1,368 @@
+{
+  "modno": 1,
+  "version": 4,
+  "instance": "falcon",
+  "repos": [
+    {
+      "id": "hidrive-backup",
+      "uri": "/repos",
+      "guid": "df03886ea215b0a3ff9730190d906d7034032bf0f1906ed4ad00f2c4f1748215",
+      "password": "falcon-backup-2025",
+      "prunePolicy": {
+        "schedule": {
+          "cron": "0 2 * * 0"
+        }
+      },
+      "checkPolicy": {
+        "schedule": {
+          "cron": "0 3 * * 0"
+        }
+      },
+      "autoUnlock": true
+    }
+  ],
+  "plans": [
+    {
+      "id": "ai-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/ai_postgres_data",
+        "/volumes/ai_webui_data"
+      ],
+      "schedule": {
+        "cron": "0 3 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 6,
+          "yearly": 2
+        }
+      }
+    },
+    {
+      "id": "asciinema-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/asciinema_data"
+      ],
+      "schedule": {
+        "cron": "0 11 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 6,
+          "yearly": 2
+        }
+      }
+    },
+    {
+      "id": "coolify-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/dev_coolify_data"
+      ],
+      "schedule": {
+        "cron": "0 0 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 6,
+          "yearly": 2
+        }
+      }
+    },
+    {
+      "id": "directus-bundle-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/directus_bundle"
+      ],
+      "schedule": {
+        "cron": "0 4 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 3
+        }
+      }
+    },
+    {
+      "id": "directus-uploads-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/directus_uploads"
+      ],
+      "schedule": {
+        "cron": "0 4 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 6,
+          "yearly": 2
+        }
+      }
+    },
+    {
+      "id": "filestash-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/filestash_data"
+      ],
+      "schedule": {
+        "cron": "0 7 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 3
+        }
+      }
+    },
+    {
+      "id": "gitea-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/dev_gitea_config",
+        "/volumes/dev_gitea_data",
+        "/volumes/dev_gitea_runner_data"
+      ],
+      "schedule": {
+        "cron": "0 11 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 6,
+          "yearly": 2
+        }
+      }
+    },
+    {
+      "id": "jellyfin-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/jelly_config"
+      ],
+      "schedule": {
+        "cron": "0 9 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 6,
+          "yearly": 2
+        }
+      }
+    },
+    {
+      "id": "joplin-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/joplin_data"
+      ],
+      "schedule": {
+        "cron": "0 2 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 6,
+          "yearly": 2
+        }
+      }
+    },
+    {
+      "id": "letsencrypt-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/letsencrypt_data"
+      ],
+      "schedule": {
+        "cron": "0 8 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 12,
+          "yearly": 3
+        }
+      }
+    },
+    {
+      "id": "linkwarden-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/linkwarden_data",
+        "/volumes/linkwarden_meili_data"
+      ],
+      "schedule": {
+        "cron": "0 7 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 6
+        }
+      }
+    },
+    {
+      "id": "mattermost-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/mattermost_config",
+        "/volumes/mattermost_data",
+        "/volumes/mattermost_plugins"
+      ],
+      "schedule": {
+        "cron": "0 5 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 6,
+          "yearly": 2
+        }
+      }
+    },
+    {
+      "id": "n8n-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/n8n_data"
+      ],
+      "schedule": {
+        "cron": "0 6 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 6
+        }
+      }
+    },
+    {
+      "id": "netdata-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/netdata_config"
+      ],
+      "schedule": {
+        "cron": "0 10 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 3
+        }
+      }
+    },
+    {
+      "id": "postgres-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/core_postgres_data"
+      ],
+      "schedule": {
+        "cron": "0 2 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 6,
+          "yearly": 2
+        }
+      }
+    },
+    {
+      "id": "redis-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/core_redis_data"
+      ],
+      "schedule": {
+        "cron": "0 3 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 3
+        }
+      }
+    },
+    {
+      "id": "scrapy-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/scrapy_code",
+        "/volumes/scrapyd_data"
+      ],
+      "schedule": {
+        "cron": "0 6 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 3
+        }
+      }
+    },
+    {
+      "id": "tandoor-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/tandoor_mediafiles",
+        "/volumes/tandoor_staticfiles"
+      ],
+      "schedule": {
+        "cron": "0 5 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 6
+        }
+      }
+    },
+    {
+      "id": "vaultwarden-backup",
+      "repo": "hidrive-backup",
+      "paths": [
+        "/volumes/vaultwarden_data"
+      ],
+      "schedule": {
+        "cron": "0 8 * * *"
+      },
+      "retention": {
+        "policyTimeBucketed": {
+          "daily": 7,
+          "weekly": 4,
+          "monthly": 12,
+          "yearly": 3
+        }
+      }
+    }
+  ]
+}
--- a/core/compose.yaml
+++ b/core/compose.yaml
@@ -56,7 +56,7 @@ services:
    volumes:
      # Backrest application data
      - backrest_data:/data
-      - backrest_config:/config
+      - ./backrest/config.json:/config/config.json
      - backrest_cache:/cache
      - backrest_tmp:/tmp

@@ -84,7 +84,6 @@ services:
      - backup_netdata_config:/volumes/netdata_config:ro
      - backup_ai_postgres_data:/volumes/ai_postgres_data:ro
      - backup_ai_webui_data:/volumes/ai_webui_data:ro
-      - backup_ai_crawl4ai_data:/volumes/ai_crawl4ai_data:ro
      - backup_asciinema_data:/volumes/asciinema_data:ro
      - backup_dev_gitea_data:/volumes/dev_gitea_data:ro
      - backup_dev_gitea_config:/volumes/dev_gitea_config:ro
@@ -124,8 +123,6 @@ volumes:
    name: ${CORE_COMPOSE_PROJECT_NAME}_redis_data
  backrest_data:
    name: ${CORE_COMPOSE_PROJECT_NAME}_backrest_data
-  backrest_config:
-    name: ${CORE_COMPOSE_PROJECT_NAME}_backrest_config
  backrest_cache:
    name: ${CORE_COMPOSE_PROJECT_NAME}_backrest_cache
  backrest_tmp:
@@ -192,9 +189,6 @@ volumes:
  backup_ai_webui_data:
    name: ai_webui_data
    external: true
-  backup_ai_crawl4ai_data:
-    name: ai_crawl4ai_data
-    external: true
  backup_asciinema_data:
    name: dev_asciinema_data
    external: true
--- a/dev/compose.yaml
+++ b/dev/compose.yaml
@@ -40,6 +40,8 @@ services:
      GITEA__mailer__FROM: ${EMAIL_FROM}
      GITEA__service__DISABLE_REGISTRATION: false
      GITEA__service__REQUIRE_SIGNIN_VIEW: false
+      GITEA__service__ENABLE_NOTIFY_MAIL: true
+      GITEA__service__DEFAULT_EMAIL_NOTIFICATIONS: enabled
      GITEA__packages__ENABLED: true
      GITEA__actions__ENABLED: true
      GITEA__ui__THEMES: gitea-auto,gitea-light,gitea-dark,arc-green,edge-auto,edge-dark,edge-light,everforest-auto,everforest-dark,everforest-light,gruvbox-auto,gruvbox-dark,gruvbox-light,gruvbox-material-auto,gruvbox-material-dark,gruvbox-material-light,nord,palenight,soft-era,sonokai,sonokai-andromeda,sonokai-atlantis,sonokai-espresso,sonokai-maia,sonokai-shusia
@@ -86,6 +88,9 @@ services:
      DOCKER_HOST: unix:///var/run/docker.sock
    networks:
      - compose_network
+    labels:
+      # Watchtower
+      - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}"

  # Coolify - Self-hosted deployment platform
  coolify:
@@ -93,8 +98,8 @@ services:
    container_name: ${DEV_COMPOSE_PROJECT_NAME}_coolify
    restart: unless-stopped
    depends_on:
-      coolify_soketi:
-        condition: service_started
+      coolify_realtime:
+        condition: service_healthy
    volumes:
      - coolify_data:/data/coolify
      - /var/run/docker.sock:/var/run/docker.sock
@@ -117,7 +122,7 @@ services:
      - DB_PASSWORD=${DB_PASSWORD}
      - REDIS_HOST=${CORE_REDIS_HOST}
      - REDIS_PORT=${CORE_REDIS_PORT}
-      - PUSHER_HOST=coolify-realtime.${DEV_COOLIFY_TRAEFIK_HOST}
+      - PUSHER_HOST=realtime.${DEV_COOLIFY_TRAEFIK_HOST}
      - PUSHER_PORT=443
      - PUSHER_APP_ID=${DEV_COOLIFY_PUSHER_APP_ID}
      - PUSHER_APP_KEY=${DEV_COOLIFY_PUSHER_APP_KEY}
@@ -128,50 +133,68 @@ services:
      - compose_network
    labels:
      - "traefik.enable=${DEV_TRAEFIK_ENABLED}"
-      # HTTP to HTTPS redirect
+      # Main web interface - HTTP to HTTPS redirect
      - "traefik.http.middlewares.${DEV_COMPOSE_PROJECT_NAME}-coolify-redirect-web-secure.redirectscheme.scheme=https"
      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-coolify-web.middlewares=${DEV_COMPOSE_PROJECT_NAME}-coolify-redirect-web-secure"
      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-coolify-web.rule=Host(`${DEV_COOLIFY_TRAEFIK_HOST}`)"
      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-coolify-web.entrypoints=web"
-      # HTTPS router
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-coolify-web.service=${DEV_COMPOSE_PROJECT_NAME}-coolify"
+      # Main web interface - HTTPS router
      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-coolify-web-secure.rule=Host(`${DEV_COOLIFY_TRAEFIK_HOST}`)"
      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-coolify-web-secure.tls.certresolver=resolver"
      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-coolify-web-secure.entrypoints=web-secure"
      - "traefik.http.middlewares.${DEV_COMPOSE_PROJECT_NAME}-coolify-web-secure-compress.compress=true"
      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-coolify-web-secure.middlewares=${DEV_COMPOSE_PROJECT_NAME}-coolify-web-secure-compress,security-headers@file"
-      # Service
-      - "traefik.http.services.${DEV_COMPOSE_PROJECT_NAME}-coolify-web-secure.loadbalancer.server.port=8080"
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-coolify-web-secure.service=${DEV_COMPOSE_PROJECT_NAME}-coolify"
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-coolify-web-secure.priority=1"
+      - "traefik.http.services.${DEV_COMPOSE_PROJECT_NAME}-coolify.loadbalancer.server.port=8080"
+      # Network
      - "traefik.docker.network=${NETWORK_NAME}"
      # Watchtower
      - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}"

-  # Coolify Soketi (WebSocket server)
-  coolify_soketi:
-    image: quay.io/soketi/soketi:1.0-16-alpine
-    container_name: ${DEV_COMPOSE_PROJECT_NAME}_coolify_soketi
+  # Coolify Realtime (WebSocket server for realtime AND terminal)
+  coolify_realtime:
+    image: ${DEV_COOLIFY_REALTIME_IMAGE:-ghcr.io/coollabsio/coolify-realtime:1.0.10}
+    container_name: ${DEV_COMPOSE_PROJECT_NAME}_coolify_realtime
    restart: unless-stopped
+    volumes:
+      - /data/coolify/ssh:/var/www/html/storage/app/ssh
    environment:
+      - APP_NAME=Coolify
+      - SOKETI_DEBUG=${SOKETI_DEBUG:-false}
      - SOKETI_DEFAULT_APP_ID=${DEV_COOLIFY_PUSHER_APP_ID}
      - SOKETI_DEFAULT_APP_KEY=${DEV_COOLIFY_PUSHER_APP_KEY}
      - SOKETI_DEFAULT_APP_SECRET=${DEV_COOLIFY_PUSHER_APP_SECRET}
    healthcheck:
-      test: ["CMD", "wget", "-qO-", "http://127.0.0.1:6001/ready"]
+      test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:6001/ready && wget -qO- http://127.0.0.1:6002/ready"]
      interval: 5s
-      timeout: 5s
+      timeout: 2s
      retries: 10
    networks:
      - compose_network
    labels:
      - "traefik.enable=${DEV_TRAEFIK_ENABLED}"
-      # HTTP router
-      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-soketi-web.rule=Host(`coolify-realtime.${DEV_COOLIFY_TRAEFIK_HOST}`)"
-      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-soketi-web.entrypoints=web"
-      # HTTPS router
-      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-soketi-web-secure.rule=Host(`coolify-realtime.${DEV_COOLIFY_TRAEFIK_HOST}`)"
-      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-soketi-web-secure.tls.certresolver=resolver"
-      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-soketi-web-secure.entrypoints=web-secure"
-      # Service
-      - "traefik.http.services.${DEV_COMPOSE_PROJECT_NAME}-soketi-web-secure.loadbalancer.server.port=6001"
+      # Realtime (port 6001) - HTTP router
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-realtime-web.rule=Host(`realtime.${DEV_COOLIFY_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-realtime-web.entrypoints=web"
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-realtime-web.service=${DEV_COMPOSE_PROJECT_NAME}-realtime"
+      # Realtime (port 6001) - HTTPS router
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-realtime-web-secure.rule=Host(`realtime.${DEV_COOLIFY_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-realtime-web-secure.tls.certresolver=resolver"
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-realtime-web-secure.entrypoints=web-secure"
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-realtime-web-secure.service=${DEV_COMPOSE_PROJECT_NAME}-realtime"
+      # Realtime service
+      - "traefik.http.services.${DEV_COMPOSE_PROJECT_NAME}-realtime.loadbalancer.server.port=6001"
+      # Terminal WebSocket (port 6002) - /terminal/ws path on main domain (PRIORITY 100)
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-terminal-ws.rule=Host(`${DEV_COOLIFY_TRAEFIK_HOST}`) && PathPrefix(`/terminal/ws`)"
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-terminal-ws.tls.certresolver=resolver"
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-terminal-ws.entrypoints=web-secure"
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-terminal-ws.service=${DEV_COMPOSE_PROJECT_NAME}-terminal"
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-terminal-ws.priority=100"
+      # Terminal service
+      - "traefik.http.services.${DEV_COMPOSE_PROJECT_NAME}-terminal.loadbalancer.server.port=6002"
+      # Network
      - "traefik.docker.network=${NETWORK_NAME}"
      # Watchtower
      - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}"
@@ -269,12 +292,11 @@ services:
      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-asciinema-admin-web.rule=Host(`admin.${DEV_ASCIINEMA_TRAEFIK_HOST}`)"
      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-asciinema-admin-web.entrypoints=web"
      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-asciinema-admin-web.service=${DEV_COMPOSE_PROJECT_NAME}-asciinema-admin"
-      # Admin interface - HTTPS router with Basic Auth
-      - "traefik.http.middlewares.${DEV_COMPOSE_PROJECT_NAME}-asciinema-auth.basicauth.users=${AUTH_USERS}"
+      # Admin interface - HTTPS router with Authelia
      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-asciinema-admin-web-secure.rule=Host(`admin.${DEV_ASCIINEMA_TRAEFIK_HOST}`)"
      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-asciinema-admin-web-secure.tls.certresolver=resolver"
      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-asciinema-admin-web-secure.entrypoints=web-secure"
-      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-asciinema-admin-web-secure.middlewares=${DEV_COMPOSE_PROJECT_NAME}-asciinema-auth,${DEV_COMPOSE_PROJECT_NAME}-asciinema-compress,security-headers@file"
+      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-asciinema-admin-web-secure.middlewares=${DEV_COMPOSE_PROJECT_NAME}-asciinema-compress,net-authelia,security-headers@file"
      - "traefik.http.routers.${DEV_COMPOSE_PROJECT_NAME}-asciinema-admin-web-secure.service=${DEV_COMPOSE_PROJECT_NAME}-asciinema-admin"
      - "traefik.http.services.${DEV_COMPOSE_PROJECT_NAME}-asciinema-admin.loadbalancer.server.port=4002"
      # Network
--- a/media/compose.yaml
+++ b/media/compose.yaml
@@ -63,6 +63,38 @@ services:
      - 'traefik.docker.network=${NETWORK_NAME}'
      - 'com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}'

+  # Pinchflat - YouTube download manager
+  pinchflat:
+    image: ${MEDIA_PINCHFLAT_IMAGE:-ghcr.io/kieraneglin/pinchflat:latest}
+    container_name: ${MEDIA_COMPOSE_PROJECT_NAME}_pinchflat
+    restart: unless-stopped
+    volumes:
+      - pinchflat_config:/config
+      - /mnt/hidrive/users/valknar/Downloads/pinchflat:/downloads
+    environment:
+      TZ: ${TIMEZONE:-Europe/Berlin}
+      JOURNAL_MODE: delete
+    networks:
+      - compose_network
+    labels:
+      - 'traefik.enable=${MEDIA_TRAEFIK_ENABLED}'
+      # HTTP to HTTPS redirect
+      - 'traefik.http.middlewares.${MEDIA_COMPOSE_PROJECT_NAME}-pinchflat-redirect-web-secure.redirectscheme.scheme=https'
+      - 'traefik.http.routers.${MEDIA_COMPOSE_PROJECT_NAME}-pinchflat-web.middlewares=${MEDIA_COMPOSE_PROJECT_NAME}-pinchflat-redirect-web-secure'
+      - 'traefik.http.routers.${MEDIA_COMPOSE_PROJECT_NAME}-pinchflat-web.rule=Host(`${MEDIA_PINCHFLAT_TRAEFIK_HOST}`)'
+      - 'traefik.http.routers.${MEDIA_COMPOSE_PROJECT_NAME}-pinchflat-web.entrypoints=web'
+      # HTTPS router with Authelia SSO protection
+      - 'traefik.http.routers.${MEDIA_COMPOSE_PROJECT_NAME}-pinchflat-web-secure.rule=Host(`${MEDIA_PINCHFLAT_TRAEFIK_HOST}`)'
+      - 'traefik.http.routers.${MEDIA_COMPOSE_PROJECT_NAME}-pinchflat-web-secure.tls.certresolver=resolver'
+      - 'traefik.http.routers.${MEDIA_COMPOSE_PROJECT_NAME}-pinchflat-web-secure.entrypoints=web-secure'
+      - 'traefik.http.middlewares.${MEDIA_COMPOSE_PROJECT_NAME}-pinchflat-web-secure-compress.compress=true'
+      - 'traefik.http.routers.${MEDIA_COMPOSE_PROJECT_NAME}-pinchflat-web-secure.middlewares=${MEDIA_COMPOSE_PROJECT_NAME}-pinchflat-web-secure-compress,net-authelia,security-headers@file'
+      # Service
+      - 'traefik.http.services.${MEDIA_COMPOSE_PROJECT_NAME}-pinchflat-web-secure.loadbalancer.server.port=8945'
+      - 'traefik.docker.network=${NETWORK_NAME}'
+      # Watchtower
+      - 'com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}'
+
 volumes:
  jellyfin_config:
    name: ${MEDIA_COMPOSE_PROJECT_NAME}_jellyfin_config
@@ -70,6 +102,8 @@ volumes:
    name: ${MEDIA_COMPOSE_PROJECT_NAME}_jellyfin_cache
  filestash_data:
    name: ${MEDIA_COMPOSE_PROJECT_NAME}_filestash_data
+  pinchflat_config:
+    name: ${MEDIA_COMPOSE_PROJECT_NAME}_pinchflat_config

 networks:
  compose_network:
--- a/net/authelia/.gitignore
+++ b/net/authelia/.gitignore
@@ -0,0 +1 @@
+net/authelia/users_database.yml
--- a/net/authelia/configuration.yml
+++ b/net/authelia/configuration.yml
@@ -6,17 +6,15 @@
 theme: auto

 server:
-  host: 0.0.0.0
-  port: 9091
-  path: ""
-  asset_path: /config/assets/
-  headers:
-    csp_template: ""
+  address: "tcp://:9091"

 log:
  level: info
  format: text

+# identity_validation jwt_secret set via environment variable:
+# AUTHELIA_IDENTITY_VALIDATION_RESET_PASSWORD_JWT_SECRET
+
 totp:
  issuer: pivoine.art
  period: 30
@@ -42,6 +40,7 @@ authentication_backend:
  refresh_interval: 5m
  file:
    path: /etc/authelia/users_database.yml
+    watch: true
    password:
      algorithm: argon2
      argon2:
@@ -71,38 +70,44 @@ access_control:
        - "mailpit.pivoine.art"
        - "scrapy.pivoine.art"
        - "restic.pivoine.art"
-        - "traefik.pivoine.art"
-      policy: two_factor
-
-    # Development services
-    - domain:
-        - "dev.pivoine.art"
-        - "n8n.pivoine.art"
-        - "asciinema.pivoine.art"
-        - "coolify.pivoine.art"
-      policy: two_factor
+        - "proxy.pivoine.art"
+        - "admin.asciinema.dev.pivoine.art"
+        - "facefusion.ai.pivoine.art"
+        - "pinchflat.media.pivoine.art"
+        - "comfy.ai.pivoine.art"
+        - "supervisor.ai.pivoine.art"
+        - "audiocraft.ai.pivoine.art"
+        - "upscale.ai.pivoine.art"
+      policy: one_factor

+# session secret set via environment variable: AUTHELIA_SESSION_SECRET
 session:
-  name: authelia_session
-  domain: pivoine.art
-  same_site: lax
-  expiration: 1h
-  inactivity: 5m
-  remember_me_duration: 1M
+  name: "authelia_session"
+  same_site: "lax"
+  expiration: "1h"
+  inactivity: "15m"
+  remember_me: "1M"
+  cookies:
+    - domain: "pivoine.art"
+      authelia_url: "https://auth.pivoine.art"
+      same_site: "lax"
+      expiration: "1h"
+      inactivity: "5m"
+      remember_me: "1M"

 regulation:
  max_retries: 3
  find_time: 2m
  ban_time: 5m

+# storage encryption_key and postgres password set via environment variables:
+# AUTHELIA_STORAGE_ENCRYPTION_KEY, AUTHELIA_STORAGE_POSTGRES_PASSWORD
 storage:
-  encryption_key: ${AUTHELIA_STORAGE_ENCRYPTION_KEY}
  postgres:
    host: postgres
    port: 5432
    database: authelia
    username: valknar
-    password: ${DB_PASSWORD}
    schema: public

 notifier:
--- a/net/authelia/users_database.template.yml
+++ b/net/authelia/users_database.template.yml
@@ -0,0 +1,29 @@
+---
+###############################################################
+#                   Users Database Template                    #
+###############################################################
+
+# This is a template file - copy to users_database.yml and edit
+# The actual users_database.yml is not tracked in git for security
+
+# Generate password hashes using:
+# docker run --rm authelia/authelia:latest authelia crypto hash generate argon2 --password 'yourpassword'
+
+# List of users
+users:
+  # Example user - replace with actual users
+  valknar:
+    displayname: "Valknar"
+    password: "$argon2id$v=19$m=65536,t=3,p=4$REPLACE_WITH_ACTUAL_HASH"
+    email: valknar@pivoine.art
+    groups:
+      - admins
+      - dev
+
+  # Add more users as needed:
+  # username:
+  #   displayname: "Full Name"
+  #   password: "$argon2id$v=19$m=65536,t=3,p=4$HASH_HERE"
+  #   email: user@pivoine.art
+  #   groups:
+  #     - users
--- a/net/authelia/users_database.yml
+++ b/net/authelia/users_database.yml
@@ -1,16 +0,0 @@
---
-###############################################################
-#                         Users Database                       #
-###############################################################
-
-# This file can be used if you do not have an LDAP set up.
-
-# List of users
-users:
-  valknar:
-    displayname: "Valknar"
-    password: "$argon2id$v=19$m=65536,t=3,p=4$c2FsdHNhbHRzYWx0$4oCb4oCh4oCd4oCi4oCl4oCm"  # CHANGE THIS - use: docker run --rm authelia/authelia:latest authelia crypto hash generate argon2 --password 'yourpassword'
-    email: valknar@pivoine.art
-    groups:
-      - admins
-      - dev
--- a/net/compose.yaml
+++ b/net/compose.yaml
@@ -6,49 +6,49 @@ services:
    restart: unless-stopped
    command:
      # API & Dashboard
-      - '--api.dashboard=true'
-      - '--api.insecure=false'
+      - "--api.dashboard=true"
+      - "--api.insecure=false"

      # Ping endpoint for healthcheck
-      - '--ping=true'
+      - "--ping=true"

      # Experimental plugins
-      - '--experimental.plugins.sablier.modulename=github.com/acouvreur/sablier'
-      - '--experimental.plugins.sablier.version=v1.8.0'
+      - "--experimental.plugins.sablier.modulename=github.com/acouvreur/sablier"
+      - "--experimental.plugins.sablier.version=v1.8.0"

      # Logging
-      - '--log.level=${NET_PROXY_LOG_LEVEL:-INFO}'
-      - '--accesslog=true'
+      - "--log.level=${NET_PROXY_LOG_LEVEL:-INFO}"
+      - "--accesslog=true"

      # Global
-      - '--global.sendAnonymousUsage=false'
-      - '--global.checkNewVersion=true'
+      - "--global.sendAnonymousUsage=false"
+      - "--global.checkNewVersion=true"

      # Docker Provider
-      - '--providers.docker=true'
-      - '--providers.docker.exposedbydefault=false'
-      - '--providers.docker.network=${NETWORK_NAME}'
+      - "--providers.docker=true"
+      - "--providers.docker.exposedbydefault=false"
+      - "--providers.docker.network=${NETWORK_NAME}"

      # File Provider for dynamic configuration
-      - '--providers.file.directory=/etc/traefik/dynamic'
-      - '--providers.file.watch=true'
+      - "--providers.file.directory=/etc/traefik/dynamic"
+      - "--providers.file.watch=true"

      # Entrypoints
-      - '--entrypoints.web.address=:${NET_PROXY_PORT_HTTP:-80}'
-      - '--entrypoints.web-secure.address=:${NET_PROXY_PORT_HTTPS:-443}'
+      - "--entrypoints.web.address=:${NET_PROXY_PORT_HTTP:-80}"
+      - "--entrypoints.web-secure.address=:${NET_PROXY_PORT_HTTPS:-443}"

      # Global HTTP to HTTPS redirect
-      - '--entrypoints.web.http.redirections.entryPoint.to=web-secure'
-      - '--entrypoints.web.http.redirections.entryPoint.scheme=https'
-      - '--entrypoints.web.http.redirections.entryPoint.permanent=true'
+      - "--entrypoints.web.http.redirections.entryPoint.to=web-secure"
+      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
+      - "--entrypoints.web.http.redirections.entryPoint.permanent=true"

      # Security Headers (applied globally)
-      - '--entrypoints.web-secure.http.middlewares=security-headers@file'
+      - "--entrypoints.web-secure.http.middlewares=security-headers@file"

      # Let's Encrypt
-      - '--certificatesresolvers.resolver.acme.tlschallenge=true'
-      - '--certificatesresolvers.resolver.acme.email=${ADMIN_EMAIL}'
-      - '--certificatesresolvers.resolver.acme.storage=/letsencrypt/acme.json'
+      - "--certificatesresolvers.resolver.acme.tlschallenge=true"
+      - "--certificatesresolvers.resolver.acme.email=${ADMIN_EMAIL}"
+      - "--certificatesresolvers.resolver.acme.storage=/letsencrypt/acme.json"

    healthcheck:
      test: ["CMD", "traefik", "healthcheck", "--ping"]
@@ -74,21 +74,20 @@ services:
      - ./dynamic:/etc/traefik/dynamic:ro

    labels:
-      - 'traefik.enable=true'
+      - "traefik.enable=true"
      # HTTP to HTTPS redirect
-      - 'traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-traefik-redirect-web-secure.redirectscheme.scheme=https'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web.middlewares=${NET_COMPOSE_PROJECT_NAME}-traefik-redirect-web-secure'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web.rule=Host(`${NET_PROXY_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web.entrypoints=web'
+      - "traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-traefik-redirect-web-secure.redirectscheme.scheme=https"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web.middlewares=${NET_COMPOSE_PROJECT_NAME}-traefik-redirect-web-secure"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web.rule=Host(`${NET_PROXY_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web.entrypoints=web"
      # HTTPS router with auth
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web-secure.rule=Host(`${NET_PROXY_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web-secure.tls.certresolver=resolver'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web-secure.entrypoints=web-secure'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web-secure.service=api@internal'
-      - 'traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-traefik-auth.basicauth.users=${AUTH_USERS}'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web-secure.middlewares=${NET_COMPOSE_PROJECT_NAME}-traefik-auth'
-      - 'traefik.http.services.${NET_COMPOSE_PROJECT_NAME}-traefik-web-secure.loadbalancer.server.port=8080'
-      - 'traefik.docker.network=${NETWORK_NAME}'
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web-secure.rule=Host(`${NET_PROXY_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web-secure.tls.certresolver=resolver"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web-secure.entrypoints=web-secure"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web-secure.service=api@internal"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-traefik-web-secure.middlewares=${NET_COMPOSE_PROJECT_NAME}-authelia,security-headers@file"
+      - "traefik.http.services.${NET_COMPOSE_PROJECT_NAME}-traefik-web-secure.loadbalancer.server.port=8080"
+      - "traefik.docker.network=${NETWORK_NAME}"

  # Netdata - Real-time monitoring
  netdata:
@@ -129,24 +128,23 @@ services:
    networks:
      - compose_network
    labels:
-      - 'traefik.enable=${NET_TRAEFIK_ENABLED}'
+      - "traefik.enable=${NET_TRAEFIK_ENABLED}"
      # HTTP to HTTPS redirect
-      - 'traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-netdata-redirect-web-secure.redirectscheme.scheme=https'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-netdata-web.middlewares=${NET_COMPOSE_PROJECT_NAME}-netdata-redirect-web-secure'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-netdata-web.rule=Host(`${NET_NETDATA_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-netdata-web.entrypoints=web'
+      - "traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-netdata-redirect-web-secure.redirectscheme.scheme=https"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-netdata-web.middlewares=${NET_COMPOSE_PROJECT_NAME}-netdata-redirect-web-secure"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-netdata-web.rule=Host(`${NET_NETDATA_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-netdata-web.entrypoints=web"
      # HTTPS router
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-netdata-web-secure.rule=Host(`${NET_NETDATA_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-netdata-web-secure.tls.certresolver=resolver'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-netdata-web-secure.entrypoints=web-secure'
-      - 'traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-netdata-compress.compress=true'
-      - 'traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-netdata-auth.basicauth.users=${AUTH_USERS}'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-netdata-web-secure.middlewares=${NET_COMPOSE_PROJECT_NAME}-netdata-auth,${NET_COMPOSE_PROJECT_NAME}-netdata-compress,security-headers@file'
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-netdata-web-secure.rule=Host(`${NET_NETDATA_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-netdata-web-secure.tls.certresolver=resolver"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-netdata-web-secure.entrypoints=web-secure"
+      - "traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-netdata-compress.compress=true"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-netdata-web-secure.middlewares=${NET_COMPOSE_PROJECT_NAME}-netdata-compress,${NET_COMPOSE_PROJECT_NAME}-authelia,security-headers@file"
      # Service
-      - 'traefik.http.services.${NET_COMPOSE_PROJECT_NAME}-netdata.loadbalancer.server.port=19999'
-      - 'traefik.docker.network=${NETWORK_NAME}'
+      - "traefik.http.services.${NET_COMPOSE_PROJECT_NAME}-netdata.loadbalancer.server.port=19999"
+      - "traefik.docker.network=${NETWORK_NAME}"
      # Watchtower
-      - 'com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}'
+      - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}"

  # Watchtower - Automatic container updates
  watchtower:
@@ -156,6 +154,8 @@ services:
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
+      # Docker API version negotiation
+      DOCKER_API_VERSION: "1.44"
      # Check for updates every 5 minutes (300 seconds)
      WATCHTOWER_POLL_INTERVAL: ${WATCHTOWER_POLL_INTERVAL:-300}
      # Only update containers with the watchtower label
@@ -202,7 +202,8 @@ services:
      - compose_network

    healthcheck:
-      test: ["CMD-SHELL", "curl -f http://localhost:3000/api/heartbeat || exit 1"]
+      test:
+        ["CMD-SHELL", "curl -f http://localhost:3000/api/heartbeat || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5
@@ -210,18 +211,21 @@ services:

    labels:
      # Traefik Configuration
-      - 'traefik.enable=${NET_TRAEFIK_ENABLED}'
+      - "traefik.enable=${NET_TRAEFIK_ENABLED}"

      # HTTP to HTTPS redirect
-      - 'traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-umami-redirect-web-secure.redirectscheme.scheme=https'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-umami-web.middlewares=${NET_COMPOSE_PROJECT_NAME}-umami-redirect-web-secure'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-umami-web.rule=Host(`${NET_TRACK_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-umami-web.entrypoints=web'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-umami-web-secure.rule=Host(`${NET_TRACK_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-umami-web-secure.tls.certresolver=resolver'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-umami-web-secure.entrypoints=web-secure'
-      - 'traefik.http.services.${NET_COMPOSE_PROJECT_NAME}-umami-web-secure.loadbalancer.server.port=3000'
-      - 'traefik.docker.network=${NETWORK_NAME}'
+      - "traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-umami-redirect-web-secure.redirectscheme.scheme=https"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-umami-web.middlewares=${NET_COMPOSE_PROJECT_NAME}-umami-redirect-web-secure"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-umami-web.rule=Host(`${NET_TRACK_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-umami-web.entrypoints=web"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-umami-web-secure.rule=Host(`${NET_TRACK_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-umami-web-secure.tls.certresolver=resolver"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-umami-web-secure.entrypoints=web-secure"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-umami-web-secure.middlewares=security-headers@file"
+      - "traefik.http.services.${NET_COMPOSE_PROJECT_NAME}-umami-web-secure.loadbalancer.server.port=3000"
+      - "traefik.docker.network=${NETWORK_NAME}"
+      # Watchtower
+      - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}"

  # Mailpit - SMTP server with web UI
  mailpit:
@@ -247,60 +251,62 @@ services:
    networks:
      - compose_network
    labels:
-      - 'traefik.enable=${NET_TRAEFIK_ENABLED}'
+      - "traefik.enable=${NET_TRAEFIK_ENABLED}"
      # HTTP to HTTPS redirect
-      - 'traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-mailpit-redirect-web-secure.redirectscheme.scheme=https'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-mailpit-web.middlewares=${NET_COMPOSE_PROJECT_NAME}-mailpit-redirect-web-secure'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-mailpit-web.rule=Host(`${NET_MAILPIT_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-mailpit-web.entrypoints=web'
+      - "traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-mailpit-redirect-web-secure.redirectscheme.scheme=https"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-mailpit-web.middlewares=${NET_COMPOSE_PROJECT_NAME}-mailpit-redirect-web-secure"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-mailpit-web.rule=Host(`${NET_MAILPIT_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-mailpit-web.entrypoints=web"
      # HTTPS router with auth
-      - 'traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-mailpit-auth.basicauth.users=${AUTH_USERS}'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-mailpit-web-secure.rule=Host(`${NET_MAILPIT_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-mailpit-web-secure.tls.certresolver=resolver'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-mailpit-web-secure.entrypoints=web-secure'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-mailpit-web-secure.middlewares=${NET_COMPOSE_PROJECT_NAME}-mailpit-auth,security-headers@file'
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-mailpit-web-secure.rule=Host(`${NET_MAILPIT_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-mailpit-web-secure.tls.certresolver=resolver"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-mailpit-web-secure.entrypoints=web-secure"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-mailpit-web-secure.middlewares=${NET_COMPOSE_PROJECT_NAME}-authelia,security-headers@file"
      # Service
-      - 'traefik.http.services.${NET_COMPOSE_PROJECT_NAME}-mailpit-web-secure.loadbalancer.server.port=8025'
-      - 'traefik.docker.network=${NETWORK_NAME}'
+      - "traefik.http.services.${NET_COMPOSE_PROJECT_NAME}-mailpit-web-secure.loadbalancer.server.port=8025"
+      - "traefik.docker.network=${NETWORK_NAME}"
      # Watchtower
-      - 'com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}'
+      - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}"

  # Authelia - SSO and authentication portal
  authelia:
    image: ${NET_AUTHELIA_IMAGE:-authelia/authelia:latest}
    container_name: ${NET_COMPOSE_PROJECT_NAME}_authelia
    restart: unless-stopped
+    command: --config /etc/authelia/configuration.yml
    environment:
      TZ: ${TIMEZONE:-Europe/Berlin}
-      AUTHELIA_JWT_SECRET: ${AUTHELIA_JWT_SECRET}
+      AUTHELIA_IDENTITY_VALIDATION_RESET_PASSWORD_JWT_SECRET: ${AUTHELIA_JWT_SECRET}
      AUTHELIA_SESSION_SECRET: ${AUTHELIA_SESSION_SECRET}
      AUTHELIA_STORAGE_ENCRYPTION_KEY: ${AUTHELIA_STORAGE_ENCRYPTION_KEY}
+      AUTHELIA_STORAGE_POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - authelia_config:/config
      - ./authelia:/etc/authelia:ro
    networks:
      - compose_network
    labels:
-      - 'traefik.enable=${NET_TRAEFIK_ENABLED}'
+      - "traefik.enable=${NET_TRAEFIK_ENABLED}"
      # HTTP to HTTPS redirect
-      - 'traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-authelia-redirect-web-secure.redirectscheme.scheme=https'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-authelia-web.middlewares=${NET_COMPOSE_PROJECT_NAME}-authelia-redirect-web-secure'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-authelia-web.rule=Host(`${NET_AUTHELIA_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-authelia-web.entrypoints=web'
+      - "traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-authelia-redirect-web-secure.redirectscheme.scheme=https"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-authelia-web.middlewares=${NET_COMPOSE_PROJECT_NAME}-authelia-redirect-web-secure"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-authelia-web.rule=Host(`${NET_AUTHELIA_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-authelia-web.entrypoints=web"
      # HTTPS router
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-authelia-web-secure.rule=Host(`${NET_AUTHELIA_TRAEFIK_HOST}`)'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-authelia-web-secure.tls.certresolver=resolver'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-authelia-web-secure.entrypoints=web-secure'
-      - 'traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-authelia-web-secure.middlewares=security-headers@file'
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-authelia-web-secure.rule=Host(`${NET_AUTHELIA_TRAEFIK_HOST}`)"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-authelia-web-secure.tls.certresolver=resolver"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-authelia-web-secure.entrypoints=web-secure"
+      - "traefik.http.routers.${NET_COMPOSE_PROJECT_NAME}-authelia-web-secure.middlewares=security-headers@file"
      # Service
-      - 'traefik.http.services.${NET_COMPOSE_PROJECT_NAME}-authelia-web-secure.loadbalancer.server.port=9091'
-      - 'traefik.docker.network=${NETWORK_NAME}'
+      - "traefik.http.services.${NET_COMPOSE_PROJECT_NAME}-authelia-web-secure.loadbalancer.server.port=9091"
+      - "traefik.docker.network=${NETWORK_NAME}"
      # ForwardAuth middleware for other services
-      - 'traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-authelia.forwardAuth.address=http://authelia:9091/api/verify?rd=https://${NET_AUTHELIA_TRAEFIK_HOST}'
-      - 'traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-authelia.forwardAuth.trustForwardHeader=true'
-      - 'traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-authelia.forwardAuth.authResponseHeaders=Remote-User,Remote-Groups,Remote-Name,Remote-Email'
+      - "traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-authelia.forwardAuth.address=http://net_authelia:9091/api/authz/forward-auth"
+      - "traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-authelia.forwardAuth.trustForwardHeader=true"
+      - "traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-authelia.forwardAuth.authResponseHeaders=Remote-User,Remote-Groups,Remote-Name,Remote-Email"
+      - "traefik.http.middlewares.${NET_COMPOSE_PROJECT_NAME}-authelia.forwardAuth.authResponseHeadersRegex=^Remote-"
      # Watchtower
-      - 'com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}'
+      - "com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}"

 volumes:
  letsencrypt_data:
--- a/sexy/compose.yaml
+++ b/sexy/compose.yaml
@@ -53,6 +53,8 @@ services:
      - 'traefik.http.routers.${SEXY_COMPOSE_PROJECT_NAME}-api-web-secure.middlewares=${SEXY_COMPOSE_PROJECT_NAME}-api-strip,${SEXY_COMPOSE_PROJECT_NAME}-api-web-secure-compress'
      - 'traefik.http.services.${SEXY_COMPOSE_PROJECT_NAME}-api-web-secure.loadbalancer.server.port=8055'
      - 'traefik.docker.network=${NETWORK_NAME}'
+      # Watchtower
+      - 'com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}'

  sexy_frontend:
    image: ${SEXY_FRONTEND_IMAGE}
--- a/util/compose.yaml
+++ b/util/compose.yaml
@@ -127,6 +127,9 @@ services:
      MEILI_NO_ANALYTICS: ${UTIL_LINKS_MEILI_NO_ANALYTICS:-true}
    volumes:
      - linkwarden_meili_data:/meili_data
+    labels:
+      # Watchtower
+      - 'com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}'

  # Mattermost - Team collaboration
  mattermost: