valknar/runpod
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: clean up runpod repository structure

Browse Source 
Removed facefusion and VPS-related files:
- compose.yaml, postgres/, litellm-config.yaml (VPS services)
- Dockerfile, entrypoint.sh, disable-nsfw-filter.patch (facefusion)

Removed outdated documentation:
- DOCKER_GPU_SETUP.md, README_GPU_SETUP.md, SETUP_GUIDE.md
- TAILSCALE_SETUP.md, WIREGUARD_SETUP.md (covered in DEPLOYMENT.md)
- GPU_EXPANSION_PLAN.md (historical planning doc)
- gpu-server-compose.yaml, litellm-config-gpu.yaml (old versions)
- deploy-gpu-stack.sh, simple_vllm_server.py (old scripts)

Organized documentation:
- Created docs/ directory
- Moved DEPLOYMENT.md, RUNPOD_TEMPLATE.md, GPU_DEPLOYMENT_LOG.md to docs/
- Updated all documentation links in README.md

Final structure:
- Clean root directory with only GPU-specific files
- Organized documentation in docs/
- Model services in dedicated directories (model-orchestrator/, vllm/, flux/, musicgen/)
- Automation scripts in scripts/
This commit is contained in:
Sebastian Krüger
2025-11-21 14:45:49 +01:00
parent 277f1c95bd
commit cafa0a1147
20 changed files with 8 additions and 4612 deletions
Download Patch File Download Diff File Expand all files Collapse all files

430
DOCKER_GPU_SETUP.md
View File

@@ -1,430 +0,0 @@
# Docker & NVIDIA Container Toolkit Setup
## Day 5: Docker Configuration on GPU Server
This guide sets up Docker with GPU support on your RunPod server.
---
## Step 1: Install Docker
### Quick Install (Recommended)
```bash
# SSH into GPU server
ssh gpu-pivoine
# Download and run Docker install script
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh
# Verify installation
docker --version
docker compose version
```
Expected output:
```
Docker version 24.0.7, build afdd53b
Docker Compose version v2.23.0
```
### Manual Install (Alternative)
```bash
# Add Docker's official GPG key
apt-get update
apt-get install -y ca-certificates curl gnupg
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
# Add repository
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
tee /etc/apt/sources.list.d/docker.list > /dev/null
# Install Docker
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# Start Docker
systemctl enable docker
systemctl start docker
```
---
## Step 2: Install NVIDIA Container Toolkit
This enables Docker containers to use the GPU.
```bash
# Add NVIDIA repository
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
# Install toolkit
apt-get update
apt-get install -y nvidia-container-toolkit
# Configure Docker to use NVIDIA runtime
nvidia-ctk runtime configure --runtime=docker
# Restart Docker
systemctl restart docker
```
---
## Step 3: Test GPU Access in Docker
### Test 1: Basic CUDA Container
```bash
docker run --rm --runtime=nvidia --gpus all \
nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi
```
Expected output: Same as `nvidia-smi` output showing your RTX 4090.
### Test 2: PyTorch Container
```bash
docker run --rm --runtime=nvidia --gpus all \
pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime \
python -c "import torch; print('CUDA:', torch.cuda.is_available(), 'Device:', torch.cuda.get_device_name(0))"
```
Expected output:
```
CUDA: True Device: NVIDIA GeForce RTX 4090
```
### Test 3: Multi-GPU Query (if you have multiple GPUs)
```bash
docker run --rm --runtime=nvidia --gpus all \
nvidia/cuda:12.1.0-base-ubuntu22.04 \
bash -c "echo 'GPU Count:' && nvidia-smi --list-gpus"
```
---
## Step 4: Configure Docker Compose with GPU Support
Docker Compose needs to know about NVIDIA runtime.
### Create daemon.json
```bash
cat > /etc/docker/daemon.json << 'EOF'
{
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
},
"default-runtime": "nvidia",
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3"
}
}
EOF
# Restart Docker
systemctl restart docker
```
---
## Step 5: Create GPU Project Structure
```bash
cd /workspace
# Create directory structure
mkdir -p gpu-stack/{vllm,comfyui,training,jupyter,monitoring}
cd gpu-stack
# Create .env file
cat > .env << 'EOF'
# GPU Stack Environment Variables
# Timezone
TIMEZONE=Europe/Berlin
# VPN Network
VPS_IP=10.8.0.1
GPU_IP=10.8.0.2
# Model Storage
MODELS_PATH=/workspace/models
# Hugging Face (optional, for private models)
HF_TOKEN=
# PostgreSQL (on VPS)
DB_HOST=10.8.0.1
DB_PORT=5432
DB_USER=valknar
DB_PASSWORD=ragnarok98
DB_NAME=openwebui
# Weights & Biases (optional, for training logging)
WANDB_API_KEY=
EOF
chmod 600 .env
```
---
## Step 6: Test Full Stack (Quick Smoke Test)
Let's deploy a minimal vLLM container to verify everything works:
```bash
cd /workspace/gpu-stack
# Create test compose file
cat > test-compose.yaml << 'EOF'
services:
test-vllm:
image: vllm/vllm-openai:latest
container_name: test_vllm
runtime: nvidia
environment:
NVIDIA_VISIBLE_DEVICES: all
command:
- --model
- facebook/opt-125m # Tiny model for testing
- --host
- 0.0.0.0
- --port
- 8000
ports:
- "8000:8000"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
EOF
# Start test
docker compose -f test-compose.yaml up -d
# Wait 30 seconds for model download
sleep 30
# Check logs
docker compose -f test-compose.yaml logs
# Test inference
curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "facebook/opt-125m",
"prompt": "Hello, my name is",
"max_tokens": 10
}'
```
Expected output (JSON response with generated text).
**Clean up test:**
```bash
docker compose -f test-compose.yaml down
```
---
## Step 7: Install Additional Tools
```bash
# Python tools
apt install -y python3-pip python3-venv
# Monitoring tools
apt install -y htop nvtop iotop
# Network tools
apt install -y iperf3 tcpdump
# Development tools
apt install -y build-essential
# Git LFS (for large model files)
apt install -y git-lfs
git lfs install
```
---
## Step 8: Configure Automatic Updates (Optional)
```bash
# Install unattended-upgrades
apt install -y unattended-upgrades
# Configure
dpkg-reconfigure -plow unattended-upgrades
# Enable automatic security updates
cat > /etc/apt/apt.conf.d/50unattended-upgrades << 'EOF'
Unattended-Upgrade::Allowed-Origins {
"${distro_id}:${distro_codename}-security";
};
Unattended-Upgrade::Automatic-Reboot "false";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
EOF
```
---
## Troubleshooting
### Docker can't access GPU
**Problem:** `docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]`
**Solution:**
```bash
# Verify NVIDIA runtime is configured
docker info | grep -i runtime
# Should show nvidia in runtimes list
# If not, reinstall nvidia-container-toolkit
# Check daemon.json
cat /etc/docker/daemon.json
# Restart Docker
systemctl restart docker
```
### Permission denied on docker commands
**Solution:**
```bash
# Add your user to docker group (if not root)
usermod -aG docker $USER
# Or always use sudo
sudo docker ...
```
### Out of disk space
**Check usage:**
```bash
df -h
du -sh /var/lib/docker
docker system df
```
**Clean up:**
```bash
# Remove unused images
docker image prune -a
# Remove unused volumes
docker volume prune
# Full cleanup
docker system prune -a --volumes
```
---
## Verification Checklist
Before deploying the full stack:
- [ ] Docker installed and running
- [ ] `docker --version` shows 24.x or newer
- [ ] `docker compose version` works
- [ ] NVIDIA Container Toolkit installed
- [ ] `docker run --gpus all nvidia/cuda:12.1.0-base nvidia-smi` works
- [ ] PyTorch container can see GPU
- [ ] Test vLLM deployment successful
- [ ] /workspace directory structure created
- [ ] .env file configured with VPN IPs
- [ ] Additional tools installed (nvtop, htop, etc.)
---
## Performance Monitoring Commands
**GPU Monitoring:**
```bash
# Real-time GPU stats
watch -n 1 nvidia-smi
# Or with nvtop (prettier)
nvtop
# GPU memory usage
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
```
**Docker Stats:**
```bash
# Container resource usage
docker stats
# Specific container
docker stats vllm --no-stream
```
**System Resources:**
```bash
# Overall system
htop
# I/O stats
iotop
# Network
iftop
```
---
## Next: Deploy Production Stack
Now you're ready to deploy the full GPU stack with vLLM, ComfyUI, and training tools.
**Proceed to:** Deploying the production docker-compose.yaml
**Save your progress:**
```bash
cat >> /workspace/SERVER_INFO.md << 'EOF'
## Docker Configuration
- Docker Version: [docker --version]
- NVIDIA Runtime: Enabled
- GPU Access in Containers: ✓
- Test vLLM Deployment: Successful
- Directory: /workspace/gpu-stack
## Tools Installed
- nvtop: GPU monitoring
- htop: System monitoring
- Docker Compose: v2.x
- Git LFS: Large file support
EOF
```

16
Dockerfile
View File

@@ -1,16 +0,0 @@
FROM facefusion/facefusion:3.5.0-cpu
# Patch content_analyser.py to disable NSFW filter
RUN sed -i '197s/.*/\treturn False # Patched: NSFW filter disabled/' /facefusion/facefusion/content_analyser.py && \
grep -q 'return False.*Patched' /facefusion/facefusion/content_analyser.py || (echo "ERROR: Patch failed!" && exit 1)
# Calculate new hash for patched content_analyser
RUN python3 -c "import inspect; import sys; sys.path.insert(0, '/facefusion'); from facefusion import content_analyser; from facefusion.hash_helper import create_hash; content = inspect.getsource(content_analyser).encode(); print('New hash:', create_hash(content))"
# Update hash check in core.py to accept patched version
RUN NEW_HASH=$(python3 -c "import inspect; import sys; sys.path.insert(0, '/facefusion'); from facefusion import content_analyser; from facefusion.hash_helper import create_hash; content = inspect.getsource(content_analyser).encode(); print(create_hash(content))") && \
sed -i "s/content_analyser_hash == 'b14e7b92'/content_analyser_hash == '$NEW_HASH'/" /facefusion/facefusion/core.py && \
echo "Updated hash check in core.py to: $NEW_HASH"
# Verify both patches were applied
RUN echo "NSFW filter patch successfully applied to image"

1306
GPU_EXPANSION_PLAN.md
View File

File diff suppressed because it is too large Load Diff

23
README.md
View File

@@ -35,7 +35,7 @@ For first-time setup on a new RunPod instance:
2. SSH to GPU server: `ssh gpu-server` 2. SSH to GPU server: `ssh gpu-server`
3. Run preparation script: `cd /workspace/ai && chmod +x scripts/prepare-template.sh && ./scripts/prepare-template.sh` 3. Run preparation script: `cd /workspace/ai && chmod +x scripts/prepare-template.sh && ./scripts/prepare-template.sh`
**See**: [DEPLOYMENT.md](DEPLOYMENT.md) for detailed deployment guide. **See**: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) for detailed deployment guide.
## Architecture ## Architecture
@@ -64,16 +64,9 @@ All requests route through the orchestrator, which automatically loads the appro
## Documentation ## Documentation
### Primary Docs - **[docs/DEPLOYMENT.md](docs/DEPLOYMENT.md)** - Complete deployment and usage guide
- **[DEPLOYMENT.md](DEPLOYMENT.md)** - Complete deployment and usage guide - **[docs/RUNPOD_TEMPLATE.md](docs/RUNPOD_TEMPLATE.md)** - Template creation and usage
- **[RUNPOD_TEMPLATE.md](RUNPOD_TEMPLATE.md)** - Template creation and usage - **[docs/GPU_DEPLOYMENT_LOG.md](docs/GPU_DEPLOYMENT_LOG.md)** - Deployment history and technical notes
- **[GPU_DEPLOYMENT_LOG.md](GPU_DEPLOYMENT_LOG.md)** - Deployment history and technical notes
### Setup Guides (Historical)
- `DOCKER_GPU_SETUP.md` - Docker configuration for GPU support
- `TAILSCALE_SETUP.md` - Tailscale VPN setup
- `WIREGUARD_SETUP.md` - WireGuard VPN (deprecated, use Tailscale)
- `SETUP_GUIDE.md` - General setup instructions
### Architecture Components ### Architecture Components
- `model-orchestrator/` - FastAPI orchestrator managing model lifecycle - `model-orchestrator/` - FastAPI orchestrator managing model lifecycle
@@ -96,7 +89,7 @@ All requests route through the orchestrator, which automatically loads the appro
3. Save pod as template in RunPod dashboard 3. Save pod as template in RunPod dashboard
4. Name: `multi-modal-ai-v1.0` 4. Name: `multi-modal-ai-v1.0`
**See**: [RUNPOD_TEMPLATE.md](RUNPOD_TEMPLATE.md) for step-by-step guide. **See**: [docs/RUNPOD_TEMPLATE.md](docs/RUNPOD_TEMPLATE.md) for step-by-step guide.
## Adding New Models ## Adding New Models
@@ -116,7 +109,7 @@ models:
Then add the Docker service to `docker-compose.gpu.yaml` and restart the orchestrator. Then add the Docker service to `docker-compose.gpu.yaml` and restart the orchestrator.
**See**: [DEPLOYMENT.md](DEPLOYMENT.md#adding-new-models) for complete instructions. **See**: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#adding-new-models) for complete instructions.
## Usage Examples ## Usage Examples
@@ -166,8 +159,8 @@ nvidia-smi
For issues: For issues:
1. Check orchestrator logs: `docker logs ai_orchestrator` 1. Check orchestrator logs: `docker logs ai_orchestrator`
2. Review [DEPLOYMENT.md](DEPLOYMENT.md#troubleshooting) 2. Review [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#troubleshooting)
3. Check [GPU_DEPLOYMENT_LOG.md](GPU_DEPLOYMENT_LOG.md) for deployment history 3. Check [docs/GPU_DEPLOYMENT_LOG.md](docs/GPU_DEPLOYMENT_LOG.md) for deployment history
## License ## License

444
README_GPU_SETUP.md
View File

@@ -1,444 +0,0 @@
# GPU-Enhanced AI Stack - Implementation Guide
Welcome to your GPU expansion setup! This directory contains everything you need to deploy a production-ready GPU server for LLM hosting, image generation, and model training.
## 📚 Documentation Files
### Planning & Architecture
- **`GPU_EXPANSION_PLAN.md`** - Complete 70-page plan with provider comparison, architecture, and roadmap
- **`README_GPU_SETUP.md`** - This file
### Step-by-Step Setup Guides
1. **`SETUP_GUIDE.md`** - Day 1-2: RunPod account & GPU server deployment
2. **`WIREGUARD_SETUP.md`** - Day 3-4: VPN connection between VPS and GPU server
3. **`DOCKER_GPU_SETUP.md`** - Day 5: Docker + NVIDIA Container Toolkit configuration
### Configuration Files
- **`gpu-server-compose.yaml`** - Production Docker Compose for GPU server
- **`litellm-config-gpu.yaml`** - Updated LiteLLM config with self-hosted models
- **`deploy-gpu-stack.sh`** - Automated deployment script
---
## 🚀 Quick Start (Week 1 Checklist)
### Day 1-2: RunPod & GPU Server ✓
- [ ] Create RunPod account at https://www.runpod.io/
- [ ] Add billing method ($50 initial credit recommended)
- [ ] Deploy RTX 4090 pod with PyTorch template
- [ ] Configure 500GB network volume
- [ ] Verify SSH access
- [ ] Test GPU with `nvidia-smi`
- [ ] **Guide:** `SETUP_GUIDE.md`
### Day 3-4: Network Configuration ✓
- [ ] Install Tailscale on VPS
- [ ] Install Tailscale on GPU server
- [ ] Authenticate both devices
- [ ] Test VPN connectivity
- [ ] Configure firewall rules
- [ ] Verify VPS can reach GPU server
- [ ] **Guide:** `TAILSCALE_SETUP.md`
### Day 5: Docker & GPU Setup ✓
- [ ] Install Docker on GPU server
- [ ] Install NVIDIA Container Toolkit
- [ ] Test GPU access in containers
- [ ] Create /workspace/gpu-stack directory
- [ ] Copy configuration files
- [ ] **Guide:** `DOCKER_GPU_SETUP.md`
### Day 6-7: Deploy Services ✓
- [ ] Copy `gpu-server-compose.yaml` to GPU server
- [ ] Edit `.env` with your settings
- [ ] Run `./deploy-gpu-stack.sh`
- [ ] Wait for vLLM to load model (~5 minutes)
- [ ] Test vLLM: `curl http://localhost:8000/v1/models`
- [ ] Access ComfyUI: `http://[tailscale-ip]:8188`
- [ ] **Script:** `deploy-gpu-stack.sh`
---
## 📦 Services Included
### vLLM (http://[tailscale-ip]:8000)
**Purpose:** High-performance LLM inference
**Default Model:** Llama 3.1 8B Instruct
**Performance:** 50-80 tokens/second on RTX 4090
**Use for:** General chat, Q&A, code generation, summarization
**Switch models:**
Edit `gpu-server-compose.yaml`, change `--model` parameter, restart:
```bash
docker compose restart vllm
```
### ComfyUI (http://[tailscale-ip]:8188)
**Purpose:** Advanced Stable Diffusion interface
**Features:** FLUX, SDXL, ControlNet, LoRA
**Use for:** Image generation, img2img, inpainting
**Download models:**
Access web UI → ComfyUI Manager → Install Models
### JupyterLab (http://[tailscale-ip]:8888)
**Purpose:** Interactive development environment
**Token:** `pivoine-ai-2025` (change in `.env`)
**Use for:** Research, experimentation, custom training scripts
### Axolotl (Training - on-demand)
**Purpose:** LLM fine-tuning framework
**Start:** `docker compose --profile training up -d axolotl`
**Use for:** LoRA training, full fine-tuning, RLHF
### Netdata (http://[tailscale-ip]:19999)
**Purpose:** System & GPU monitoring
**Features:** Real-time metrics, GPU utilization, memory usage
**Use for:** Performance monitoring, troubleshooting
---
## 🔧 Configuration
### Environment Variables (.env)
```bash
# VPN Network (Tailscale)
VPS_IP=100.x.x.x # Your VPS Tailscale IP (get with: tailscale ip -4)
GPU_IP=100.x.x.x # GPU server Tailscale IP (get with: tailscale ip -4)
# Model Storage
MODELS_PATH=/workspace/models
# Hugging Face Token (for gated models like Llama)
HF_TOKEN=hf_xxxxxxxxxxxxx
# Weights & Biases (for training logging)
WANDB_API_KEY=
# JupyterLab Access
JUPYTER_TOKEN=pivoine-ai-2025
# PostgreSQL (on VPS)
DB_HOST=100.x.x.x # Your VPS Tailscale IP
DB_PORT=5432
DB_USER=valknar
DB_PASSWORD=ragnarok98
DB_NAME=openwebui
```
### Updating LiteLLM on VPS
After GPU server is running, update your VPS LiteLLM config:
```bash
# On VPS
cd ~/Projects/docker-compose/ai
# Backup current config
cp litellm-config.yaml litellm-config.yaml.backup
# Copy new config with GPU models
cp litellm-config-gpu.yaml litellm-config.yaml
# Restart LiteLLM
arty restart litellm
```
Now Open WebUI will have access to both Claude (API) and Llama (self-hosted)!
---
## 💰 Cost Management
### Current Costs (24/7 Operation)
- **GPU Server:** RTX 4090 @ $0.50/hour = $360/month
- **Storage:** 500GB network volume = $50/month
- **Total:** **$410/month**
### Cost-Saving Options
**1. Pay-as-you-go (8 hours/day)**
- GPU: $0.50 × 8 × 30 = $120/month
- Storage: $50/month
- **Total: $170/month**
**2. Auto-stop idle pods**
RunPod can auto-stop after X minutes idle:
- Dashboard → Pod Settings → Auto-stop after 30 minutes
**3. Use smaller models**
- Mistral 7B instead of Llama 8B: Faster, cheaper GPU
- Quantized models: 4-bit = 1/4 the VRAM
**4. Batch image generation**
- Generate multiple images at once
- Use scheduled jobs (cron) during off-peak hours
### Cost Tracking
**Check GPU usage:**
```bash
# On RunPod dashboard
Billing → Usage History
# See hourly costs, total spent
```
**Check API vs GPU savings:**
```bash
# On VPS, check LiteLLM logs
docker logs ai_litellm | grep "model="
# Count requests to llama-3.1-8b vs claude-*
```
**Expected savings:**
- 80% of requests → self-hosted = $0 cost
- 20% of requests → Claude = API cost
- Break-even if currently spending >$500/month on APIs
---
## 🔍 Monitoring & Troubleshooting
### Check Service Status
```bash
# On GPU server
cd /workspace/gpu-stack
# View all services
docker compose ps
# Check specific service logs
docker compose logs -f vllm
docker compose logs -f comfyui
docker compose logs -f jupyter
# Check GPU usage
nvidia-smi
# or prettier:
nvtop
```
### Common Issues
**vLLM not loading model:**
```bash
# Check logs
docker compose logs vllm
# Common causes:
# - Model download in progress (wait 5-10 minutes)
# - Out of VRAM (try smaller model)
# - Missing HF_TOKEN (for gated models like Llama)
```
**ComfyUI slow/crashing:**
```bash
# Check GPU memory
nvidia-smi
# If VRAM full:
# - Close vLLM temporarily
# - Use smaller models
# - Reduce batch size in ComfyUI
```
**Can't access from VPS:**
```bash
# Test VPN
ping [tailscale-ip]
# If fails:
# - Check Tailscale status: tailscale status
# - Restart Tailscale: tailscale down && tailscale up
# - Check firewall: ufw status
```
**Docker can't see GPU:**
```bash
# Test GPU access
docker run --rm --runtime=nvidia --gpus all nvidia/cuda:12.1.0-base nvidia-smi
# If fails:
# - Check NVIDIA driver: nvidia-smi
# - Check nvidia-docker: nvidia-ctk --version
# - Restart Docker: systemctl restart docker
```
---
## 📊 Performance Benchmarks
### Expected Performance (RTX 4090)
**LLM Inference (vLLM):**
- Llama 3.1 8B: 50-80 tokens/second
- Qwen 2.5 14B: 30-50 tokens/second
- Batch size 32: ~1500 tokens/second
**Image Generation (ComfyUI):**
- SDXL (1024×1024): ~4-6 seconds
- FLUX (1024×1024): ~8-12 seconds
- SD 1.5 (512×512): ~1-2 seconds
**Training (Axolotl):**
- LoRA fine-tuning (8B model): ~3-5 hours for 3 epochs
- Full fine-tuning: Not recommended on 24GB VRAM
---
## 🔐 Security Best Practices
### Network Security
✅ All services behind Tailscale VPN (end-to-end encrypted)
✅ No public exposure (except RunPod's SSH)
✅ Firewall configured (no additional ports needed)
### Access Control
✅ JupyterLab password-protected
✅ ComfyUI accessible via VPN only
✅ vLLM internal API (no auth needed)
### SSH Security
```bash
# On GPU server, harden SSH
nano /etc/ssh/sshd_config
# Set:
PermitRootLogin prohibit-password
PasswordAuthentication no
PubkeyAuthentication yes
systemctl restart sshd
```
### Regular Updates
```bash
# Weekly updates
apt update && apt upgrade -y
# Update Docker images
docker compose pull
docker compose up -d
```
---
## 📈 Scaling Up
### When to Add More GPUs
**Current limitations (1× RTX 4090):**
- Can run ONE of these at a time:
- 8B LLM at full speed
- 14B LLM at moderate speed
- SDXL image generation
- Training job
**Add 2nd GPU if:**
- You want LLM + image gen simultaneously
- Training + inference at same time
- Multiple users with high demand
**Multi-GPU options:**
- 2× RTX 4090: Run vLLM + ComfyUI separately ($720/month)
- 1× A100 40GB: Larger models (70B with quantization) ($1,080/month)
- Mix: RTX 4090 (inference) + A100 (training) (~$1,300/month)
### Deploying Larger Models
**70B models (need 2× A100 or 4× RTX 4090):**
```yaml
# In gpu-server-compose.yaml
vllm:
command:
- --model
- meta-llama/Meta-Llama-3.1-70B-Instruct
- --tensor-parallel-size
- "2" # Split across 2 GPUs
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 2 # Use 2 GPUs
capabilities: [gpu]
```
---
## 🎯 Next Steps (Week 2+)
### Week 2: LLM Production Deployment
- [ ] Test Llama 3.1 8B performance
- [ ] Download additional models (Qwen, Mistral)
- [ ] Configure model routing in LiteLLM
- [ ] Set up usage monitoring
- [ ] Benchmark tokens/second for each model
### Week 3: Image Generation
- [ ] Download FLUX and SDXL models
- [ ] Install ComfyUI Manager
- [ ] Download ControlNet models
- [ ] Create sample workflows
- [ ] Test API integration with Open WebUI
### Week 4: Training Infrastructure
- [ ] Prepare a sample dataset
- [ ] Test LoRA fine-tuning with Axolotl
- [ ] Set up Weights & Biases logging
- [ ] Create training documentation
- [ ] Benchmark training speed
---
## 🆘 Getting Help
### Resources
- **RunPod Docs:** https://docs.runpod.io/
- **vLLM Docs:** https://docs.vllm.ai/
- **ComfyUI Wiki:** https://github.com/comfyanonymous/ComfyUI/wiki
- **Axolotl Docs:** https://github.com/OpenAccess-AI-Collective/axolotl
### Community
- **RunPod Discord:** https://discord.gg/runpod
- **vLLM Discord:** https://discord.gg/vllm
- **r/LocalLLaMA:** https://reddit.com/r/LocalLLaMA
### Support
If you encounter issues:
1. Check logs: `docker compose logs -f [service]`
2. Check GPU: `nvidia-smi`
3. Check VPN: `wg show`
4. Restart service: `docker compose restart [service]`
5. Full restart: `docker compose down && docker compose up -d`
---
## ✅ Success Criteria
You're ready to proceed when:
- [ ] GPU server responds to `ping [tailscale-ip]` from VPS
- [ ] vLLM returns models: `curl http://[tailscale-ip]:8000/v1/models`
- [ ] ComfyUI web interface loads: `http://[tailscale-ip]:8188`
- [ ] JupyterLab accessible with token
- [ ] Netdata shows GPU metrics
- [ ] Open WebUI shows both Claude and Llama models
**Total setup time:** 4-6 hours (if following guides sequentially)
---
## 🎉 You're All Set!
Your GPU-enhanced AI stack is ready. You now have:
- ✅ Self-hosted LLM inference (saves $$$)
- ✅ Advanced image generation (FLUX, SDXL)
- ✅ Model training capabilities (LoRA, fine-tuning)
- ✅ Secure VPN connection
- ✅ Full monitoring and logging
Enjoy building with your new AI infrastructure! 🚀

261
SETUP_GUIDE.md
View File

@@ -1,261 +0,0 @@
# GPU Server Setup Guide - Week 1
## Day 1-2: RunPod Account & GPU Server
### Step 1: Create RunPod Account
1. **Go to RunPod**: https://www.runpod.io/
2. **Sign up** with email or GitHub
3. **Add billing method**:
- Credit card required
- No charges until you deploy a pod
- Recommended: Add $50 initial credit
4. **Verify email** and complete account setup
### Step 2: Deploy Your First GPU Pod
#### 2.1 Navigate to Pods
1. Click **"Deploy"** in top menu
2. Select **"GPU Pods"**
#### 2.2 Choose GPU Type
**Recommended: RTX 4090**
- 24GB VRAM
- ~$0.50/hour
- Perfect for LLMs up to 14B params
- Great for SDXL/FLUX
**Filter options:**
- GPU Type: RTX 4090
- GPU Count: 1
- Sort by: Price (lowest first)
- Region: Europe (lower latency to Germany)
#### 2.3 Select Template
Choose: **"RunPod PyTorch"** template
- Includes: CUDA, PyTorch, Python
- Pre-configured for GPU workloads
- Docker pre-installed
**Alternative**: "Ubuntu 22.04 with CUDA 12.1" (more control)
#### 2.4 Configure Pod
**Container Settings:**
- **Container Disk**: 50GB (temporary, auto-included)
- **Expose Ports**:
- Add: 22 (SSH)
- Add: 8000 (vLLM)
- Add: 8188 (ComfyUI)
- Add: 8888 (JupyterLab)
**Volume Settings:**
- Click **"+ Network Volume"**
- **Name**: `gpu-models-storage`
- **Size**: 500GB
- **Region**: Same as pod
- **Cost**: ~$50/month
**Environment Variables:**
- Add later (not needed for initial setup)
#### 2.5 Deploy Pod
1. Review configuration
2. Click **"Deploy On-Demand"** (not Spot for reliability)
3. Wait 2-3 minutes for deployment
**Expected cost:**
- GPU: $0.50/hour = $360/month (24/7)
- Storage: $50/month
- **Total: $410/month**
### Step 3: Access Your GPU Server
#### 3.1 Get Connection Info
Once deployed, you'll see:
- **Pod ID**: e.g., `abc123def456`
- **SSH Command**: `ssh root@<pod-id>.runpod.io -p 12345`
- **Public IP**: May not be directly accessible (use SSH)
#### 3.2 SSH Access
RunPod automatically generates SSH keys for you:
```bash
# Copy the SSH command from RunPod dashboard
ssh root@abc123def456.runpod.io -p 12345
# First time: Accept fingerprint
# You should now be in the GPU server!
```
**Verify GPU:**
```bash
nvidia-smi
```
Expected output:
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 535.xx Driver Version: 535.xx CUDA Version: 12.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A |
| 30% 45C P0 50W / 450W | 0MiB / 24564MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
```
### Step 4: Initial Server Configuration
#### 4.1 Update System
```bash
# Update package lists
apt update
# Upgrade existing packages
apt upgrade -y
# Install essential tools
apt install -y \
vim \
htop \
tmux \
curl \
wget \
git \
net-tools \
iptables-persistent
```
#### 4.2 Set Timezone
```bash
timedatectl set-timezone Europe/Berlin
date # Verify
```
#### 4.3 Create Working Directory
```bash
# Create workspace
mkdir -p /workspace/{models,configs,data,scripts}
# Check network volume mount
ls -la /workspace
# Should show your 500GB volume
```
#### 4.4 Configure SSH (Optional but Recommended)
**Generate your own SSH key on your local machine:**
```bash
# On your local machine (not GPU server)
ssh-keygen -t ed25519 -C "gpu-server-pivoine" -f ~/.ssh/gpu_pivoine
# Copy public key to GPU server
ssh-copy-id -i ~/.ssh/gpu_pivoine.pub root@abc123def456.runpod.io -p 12345
```
**Add to your local ~/.ssh/config:**
```bash
Host gpu-pivoine
HostName abc123def456.runpod.io
Port 12345
User root
IdentityFile ~/.ssh/gpu_pivoine
```
Now you can connect with: `ssh gpu-pivoine`
### Step 5: Verify GPU Access
Run this test:
```bash
# Test CUDA
python3 -c "import torch; print('CUDA available:', torch.cuda.is_available()); print('GPU count:', torch.cuda.device_count())"
```
Expected output:
```
CUDA available: True
GPU count: 1
```
### Troubleshooting
**Problem: Can't connect via SSH**
- Check pod is running (not stopped)
- Verify port number in SSH command
- Try web terminal in RunPod dashboard
**Problem: GPU not detected**
- Run `nvidia-smi`
- Check RunPod selected correct GPU type
- Restart pod if needed
**Problem: Network volume not mounted**
- Check RunPod dashboard → Volume tab
- Verify volume is attached to pod
- Try: `df -h` to see mounts
### Next Steps
Once SSH access works and GPU is verified:
✅ Proceed to **Day 3-4: Network Configuration (Tailscale VPN)**
### Save Important Info
Create a file to track your setup:
```bash
# On GPU server
cat > /workspace/SERVER_INFO.md << 'EOF'
# GPU Server Information
## Connection
- SSH: ssh root@abc123def456.runpod.io -p 12345
- Pod ID: abc123def456
- Region: [YOUR_REGION]
## Hardware
- GPU: RTX 4090 24GB
- CPU: [Check with: lscpu]
- RAM: [Check with: free -h]
- Storage: 500GB network volume at /workspace
## Costs
- GPU: $0.50/hour
- Storage: $50/month
- Total: ~$410/month (24/7)
## Deployed: [DATE]
EOF
```
---
## Checkpoint ✓
Before moving to Day 3, verify:
- [ ] RunPod account created and billing added
- [ ] RTX 4090 pod deployed successfully
- [ ] 500GB network volume attached
- [ ] SSH access working
- [ ] `nvidia-smi` shows GPU
- [ ] `torch.cuda.is_available()` returns True
- [ ] Timezone set to Europe/Berlin
- [ ] Essential tools installed
**Ready for Tailscale setup? Let's go!**

417
TAILSCALE_SETUP.md
View File

@@ -1,417 +0,0 @@
# Tailscale VPN Setup - Better Alternative to WireGuard
## Why Tailscale?
RunPod doesn't support UDP ports, which blocks WireGuard. Tailscale solves this by:
- ✅ Works over HTTPS (TCP) - no UDP needed
- ✅ Zero configuration - automatic setup
- ✅ Free for personal use
- ✅ Built on WireGuard (same security)
- ✅ Automatic NAT traversal
- ✅ Peer-to-peer when possible (low latency)
---
## Step 1: Create Tailscale Account
1. Go to: https://tailscale.com/
2. Click **"Get Started"**
3. Sign up with **GitHub** or **Google** (easiest)
4. You'll be redirected to the Tailscale admin console
**No credit card required!** Free tier is perfect for our use case.
---
## Step 2: Install Tailscale on VPS
**SSH into your VPS:**
```bash
ssh root@vps
```
**Install Tailscale:**
```bash
# Download and run install script
curl -fsSL https://tailscale.com/install.sh | sh
# Start Tailscale
tailscale up
# You'll see a URL like:
# https://login.tailscale.com/a/xxxxxxxxxx
```
**Authenticate:**
1. Copy the URL and open in browser
2. Click **"Connect"** to authorize the device
3. Name it: `pivoine-vps`
**Check status:**
```bash
tailscale status
```
You should see your VPS listed with an IP like `100.x.x.x`
**Save your VPS Tailscale IP:**
```bash
tailscale ip -4
# Example output: 100.101.102.103
```
**Write this down - you'll need it!**
---
## Step 3: Install Tailscale on GPU Server
**SSH into your RunPod GPU server:**
```bash
ssh root@abc123def456-12345678.runpod.io -p 12345
```
**Install Tailscale:**
```bash
# Download and run install script
curl -fsSL https://tailscale.com/install.sh | sh
# Start Tailscale
tailscale up --advertise-tags=tag:gpu
# You'll see another URL
```
**Authenticate:**
1. Copy the URL and open in browser
2. Click **"Connect"**
3. Name it: `gpu-runpod`
**Check status:**
```bash
tailscale status
```
You should now see BOTH devices:
- `pivoine-vps` - 100.x.x.x
- `gpu-runpod` - 100.x.x.x
**Save your GPU server Tailscale IP:**
```bash
tailscale ip -4
# Example output: 100.104.105.106
```
---
## Step 4: Test Connectivity
**From VPS, ping GPU server:**
```bash
# SSH into VPS
ssh root@vps
# Ping GPU server (use its Tailscale IP)
ping 100.104.105.106 -c 4
```
Expected output:
```
PING 100.104.105.106 (100.104.105.106) 56(84) bytes of data.
64 bytes from 100.104.105.106: icmp_seq=1 ttl=64 time=15.3 ms
64 bytes from 100.104.105.106: icmp_seq=2 ttl=64 time=14.8 ms
...
```
**From GPU server, ping VPS:**
```bash
# SSH into GPU server
ssh root@abc123def456-12345678.runpod.io -p 12345
# Ping VPS (use its Tailscale IP)
ping 100.101.102.103 -c 4
```
**Both should work!** ✅
---
## Step 5: Update Configuration Files
Now update the IP addresses in your configs to use Tailscale IPs.
### On GPU Server (.env file)
**Edit your .env file:**
```bash
# On GPU server
cd /workspace/gpu-stack
nano .env
```
**Update these lines:**
```bash
# VPN Network (use your actual Tailscale IPs)
VPS_IP=100.101.102.103 # Your VPS Tailscale IP
GPU_IP=100.104.105.106 # Your GPU Tailscale IP
# PostgreSQL (on VPS)
DB_HOST=100.101.102.103 # Your VPS Tailscale IP
DB_PORT=5432
```
Save and exit (Ctrl+X, Y, Enter)
### On VPS (LiteLLM config)
**Edit your LiteLLM config:**
```bash
# On VPS
ssh root@vps
cd ~/Projects/docker-compose/ai
nano litellm-config-gpu.yaml
```
**Update the GPU server IP:**
```yaml
# Find this section and update IP:
- model_name: llama-3.1-8b
litellm_params:
model: openai/meta-llama/Meta-Llama-3.1-8B-Instruct
api_base: http://100.104.105.106:8000/v1 # Use GPU Tailscale IP
api_key: dummy
```
Save and exit.
---
## Step 6: Verify PostgreSQL Access
**From GPU server, test database connection:**
```bash
# Install PostgreSQL client
apt install -y postgresql-client
# Test connection (use your VPS Tailscale IP)
psql -h 100.101.102.103 -U valknar -d openwebui -c "SELECT 1;"
```
**If this fails, allow Tailscale network on VPS PostgreSQL:**
```bash
# On VPS
ssh root@vps
# Check if postgres allows Tailscale network
docker exec core_postgres cat /var/lib/postgresql/data/pg_hba.conf | grep 100
# If not present, add it:
docker exec -it core_postgres bash
# Inside container:
echo "host all all 100.0.0.0/8 scram-sha-256" >> /var/lib/postgresql/data/pg_hba.conf
# Restart postgres
exit
docker restart core_postgres
```
Try connecting again - should work now!
---
## Tailscale Management
### View Connected Devices
**Web dashboard:**
https://login.tailscale.com/admin/machines
You'll see all your devices with their Tailscale IPs.
**Command line:**
```bash
tailscale status
```
### Disconnect/Reconnect
```bash
# Stop Tailscale
tailscale down
# Start Tailscale
tailscale up
```
### Remove Device
From web dashboard:
1. Click on device
2. Click "..." menu
3. Select "Disable" or "Delete"
---
## Advantages Over WireGuard
✅ **Works anywhere** - No UDP ports needed
✅ **Auto-reconnect** - Survives network changes
✅ **Multiple devices** - Easy to add laptop, phone, etc.
✅ **NAT traversal** - Direct peer-to-peer when possible
✅ **Access Control** - Manage from web dashboard
✅ **Monitoring** - See connection status in real-time
---
## Security Notes
🔒 **Tailscale is secure:**
- End-to-end encrypted (WireGuard)
- Zero-trust architecture
- No Tailscale servers can see your traffic
- Only authenticated devices can connect
🔒 **Access control:**
- Only devices you authorize can join
- Revoke access anytime from dashboard
- Set ACLs for fine-grained control
---
## Network Reference (Updated)
**Old (WireGuard):**
- VPS: `10.8.0.1`
- GPU: `10.8.0.2`
**New (Tailscale):**
- VPS: `100.101.102.103` (example - use your actual IP)
- GPU: `100.104.105.106` (example - use your actual IP)
**All services now accessible via Tailscale:**
**From VPS to GPU:**
- vLLM: `http://100.104.105.106:8000`
- ComfyUI: `http://100.104.105.106:8188`
- JupyterLab: `http://100.104.105.106:8888`
- Netdata: `http://100.104.105.106:19999`
**From GPU to VPS:**
- PostgreSQL: `100.101.102.103:5432`
- Redis: `100.101.102.103:6379`
- LiteLLM: `http://100.101.102.103:4000`
---
## Troubleshooting
### Can't ping between devices
**Check Tailscale status:**
```bash
tailscale status
```
Both devices should show "active" or "online".
**Check connectivity:**
```bash
tailscale ping 100.104.105.106
```
**Restart Tailscale:**
```bash
tailscale down && tailscale up
```
### PostgreSQL connection refused
**Check if postgres is listening on all interfaces:**
```bash
# On VPS
docker exec core_postgres cat /var/lib/postgresql/data/postgresql.conf | grep listen_addresses
```
Should show: `listen_addresses = '*'`
**Check pg_hba.conf allows Tailscale network:**
```bash
docker exec core_postgres cat /var/lib/postgresql/data/pg_hba.conf | grep 100
```
Should have line:
```
host all all 100.0.0.0/8 scram-sha-256
```
### Device not showing in network
**Re-authenticate:**
```bash
tailscale logout
tailscale up
# Click the new URL to re-authenticate
```
---
## Verification Checklist
Before proceeding:
- [ ] Tailscale account created
- [ ] Tailscale installed on VPS
- [ ] Tailscale installed on GPU server
- [ ] Both devices visible in `tailscale status`
- [ ] VPS can ping GPU server (via Tailscale IP)
- [ ] GPU server can ping VPS (via Tailscale IP)
- [ ] PostgreSQL accessible from GPU server
- [ ] .env file updated with Tailscale IPs
- [ ] LiteLLM config updated with GPU Tailscale IP
---
## Next Steps
✅ **Network configured!** Proceed to Docker & GPU setup:
```bash
cat /home/valknar/Projects/docker-compose/ai/DOCKER_GPU_SETUP.md
```
**Your Tailscale IPs (save these!):**
- VPS: `__________________` (from `tailscale ip -4` on VPS)
- GPU: `__________________` (from `tailscale ip -4` on GPU server)
---
## Bonus: Add Your Local Machine
Want to access GPU server from your laptop?
```bash
# On your local machine
curl -fsSL https://tailscale.com/install.sh | sh
tailscale up
# Now you can SSH directly via Tailscale:
ssh root@100.104.105.106
# Or access ComfyUI in browser:
# http://100.104.105.106:8188
```
No more port forwarding needed! 🎉

393
WIREGUARD_SETUP.md
View File

@@ -1,393 +0,0 @@
# WireGuard VPN Setup - Connecting GPU Server to VPS
## Day 3-4: Network Configuration
This guide connects your RunPod GPU server to your VPS via WireGuard VPN, enabling secure, low-latency communication.
### Architecture
```
┌─────────────────────────────┐ ┌──────────────────────────────┐
│ VPS (pivoine.art) │ │ GPU Server (RunPod) │
│ 10.8.0.1 (WireGuard) │◄───────►│ 10.8.0.2 (WireGuard) │
├─────────────────────────────┤ ├──────────────────────────────┤
│ - LiteLLM Proxy │ │ - vLLM (10.8.0.2:8000) │
│ - Open WebUI │ │ - ComfyUI (10.8.0.2:8188) │
│ - PostgreSQL │ │ - Training │
└─────────────────────────────┘ └──────────────────────────────┘
```
### Prerequisites
- ✅ VPS with root access
- ✅ GPU server with root access
- ✅ Both servers have public IPs
---
## Method 1: Using Existing wg-easy (Recommended)
You already have `wg-easy` running on your VPS. Let's use it!
### Step 1: Access wg-easy Dashboard
**On your local machine:**
1. Open browser: https://vpn.pivoine.art (or whatever your wg-easy URL is)
2. Login with admin password
**Don't have wg-easy set up? Skip to Method 2.**
### Step 2: Create GPU Server Client
1. In wg-easy dashboard, click **"+ New Client"**
2. **Name**: `gpu-server-runpod`
3. Click **"Create"**
4. **Download** configuration file (or copy QR code data)
You'll get a file like: `gpu-server-runpod.conf`
### Step 3: Install WireGuard on GPU Server
**SSH into GPU server:**
```bash
ssh gpu-pivoine # or your SSH command
# Install WireGuard
apt update
apt install -y wireguard wireguard-tools
```
### Step 4: Configure WireGuard on GPU Server
**Upload the config file:**
```bash
# On your local machine, copy the config to GPU server
scp gpu-server-runpod.conf gpu-pivoine:/etc/wireguard/wg0.conf
# Or manually create it on GPU server:
nano /etc/wireguard/wg0.conf
# Paste the configuration from wg-easy
```
**Example config (yours will be different):**
```ini
[Interface]
PrivateKey = <PRIVATE_KEY_FROM_WG_EASY>
Address = 10.8.0.2/24
DNS = 10.8.0.1
[Peer]
PublicKey = <VPS_PUBLIC_KEY_FROM_WG_EASY>
PresharedKey = <PRESHARED_KEY>
AllowedIPs = 10.8.0.0/24
Endpoint = <VPS_PUBLIC_IP>:51820
PersistentKeepalive = 25
```
### Step 5: Start WireGuard
```bash
# Enable IP forwarding
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl -p
# Set permissions
chmod 600 /etc/wireguard/wg0.conf
# Start WireGuard
systemctl enable wg-quick@wg0
systemctl start wg-quick@wg0
# Check status
systemctl status wg-quick@wg0
wg show
```
Expected output:
```
interface: wg0
public key: <GPU_SERVER_PUBLIC_KEY>
private key: (hidden)
listening port: 51820
peer: <VPS_PUBLIC_KEY>
endpoint: <VPS_IP>:51820
allowed ips: 10.8.0.0/24
latest handshake: 1 second ago
transfer: 1.2 KiB received, 892 B sent
persistent keepalive: every 25 seconds
```
### Step 6: Test Connectivity
**From GPU server, ping VPS:**
```bash
ping 10.8.0.1 -c 4
```
Expected output:
```
PING 10.8.0.1 (10.8.0.1) 56(84) bytes of data.
64 bytes from 10.8.0.1: icmp_seq=1 ttl=64 time=25.3 ms
64 bytes from 10.8.0.1: icmp_seq=2 ttl=64 time=24.8 ms
...
```
**From VPS, ping GPU server:**
```bash
ssh root@vps
ping 10.8.0.2 -c 4
```
**Test PostgreSQL access from GPU server:**
```bash
# On GPU server
apt install -y postgresql-client
# Try connecting to VPS postgres
psql -h 10.8.0.1 -U valknar -d openwebui -c "SELECT 1;"
# Should work if postgres allows 10.8.0.0/24
```
---
## Method 2: Manual WireGuard Setup (If no wg-easy)
### Step 1: Install WireGuard on Both Servers
**On VPS:**
```bash
ssh root@vps
apt update
apt install -y wireguard wireguard-tools
```
**On GPU Server:**
```bash
ssh gpu-pivoine
apt update
apt install -y wireguard wireguard-tools
```
### Step 2: Generate Keys
**On VPS:**
```bash
cd /etc/wireguard
umask 077
wg genkey | tee vps-private.key | wg pubkey > vps-public.key
```
**On GPU Server:**
```bash
cd /etc/wireguard
umask 077
wg genkey | tee gpu-private.key | wg pubkey > gpu-public.key
```
### Step 3: Create Config on VPS
**On VPS (`/etc/wireguard/wg0.conf`):**
```bash
cat > /etc/wireguard/wg0.conf << 'EOF'
[Interface]
PrivateKey = <VPS_PRIVATE_KEY>
Address = 10.8.0.1/24
ListenPort = 51820
SaveConfig = false
# GPU Server Peer
[Peer]
PublicKey = <GPU_PUBLIC_KEY>
AllowedIPs = 10.8.0.2/32
PersistentKeepalive = 25
EOF
```
Replace `<VPS_PRIVATE_KEY>` with contents of `vps-private.key`
Replace `<GPU_PUBLIC_KEY>` with contents from GPU server's `gpu-public.key`
### Step 4: Create Config on GPU Server
**On GPU Server (`/etc/wireguard/wg0.conf`):**
```bash
cat > /etc/wireguard/wg0.conf << 'EOF'
[Interface]
PrivateKey = <GPU_PRIVATE_KEY>
Address = 10.8.0.2/24
[Peer]
PublicKey = <VPS_PUBLIC_KEY>
AllowedIPs = 10.8.0.0/24
Endpoint = <VPS_PUBLIC_IP>:51820
PersistentKeepalive = 25
EOF
```
Replace:
- `<GPU_PRIVATE_KEY>` with contents of `gpu-private.key`
- `<VPS_PUBLIC_KEY>` with contents from VPS's `vps-public.key`
- `<VPS_PUBLIC_IP>` with your VPS's public IP address
### Step 5: Start WireGuard on Both
**On VPS:**
```bash
# Enable IP forwarding
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl -p
# Start WireGuard
chmod 600 /etc/wireguard/wg0.conf
systemctl enable wg-quick@wg0
systemctl start wg-quick@wg0
```
**On GPU Server:**
```bash
# Enable IP forwarding
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl -p
# Start WireGuard
chmod 600 /etc/wireguard/wg0.conf
systemctl enable wg-quick@wg0
systemctl start wg-quick@wg0
```
### Step 6: Configure Firewall
**On VPS:**
```bash
# Allow WireGuard port
ufw allow 51820/udp
ufw reload
# Or with iptables
iptables -A INPUT -p udp --dport 51820 -j ACCEPT
iptables-save > /etc/iptables/rules.v4
```
**On GPU Server (RunPod):**
```bash
# Allow WireGuard
ufw allow 51820/udp
ufw reload
```
### Step 7: Test Connection
Same as Method 1 Step 6.
---
## Troubleshooting
### No handshake
**Check:**
```bash
wg show
```
If "latest handshake" shows "never":
1. Verify public keys are correct (easy to swap them!)
2. Check firewall allows UDP 51820
3. Verify endpoint IP is correct
4. Check `systemctl status wg-quick@wg0` for errors
### Can ping but can't access services
**On VPS, check PostgreSQL allows 10.8.0.0/24:**
```bash
# Edit postgresql.conf
nano /var/lib/postgresql/data/postgresql.conf
# Add or modify:
listen_addresses = '*'
# Edit pg_hba.conf
nano /var/lib/postgresql/data/pg_hba.conf
# Add:
host all all 10.8.0.0/24 scram-sha-256
# Restart
docker restart core_postgres
```
### WireGuard won't start
```bash
# Check logs
journalctl -u wg-quick@wg0 -n 50
# Common issues:
# - Wrong permissions: chmod 600 /etc/wireguard/wg0.conf
# - Invalid keys: regenerate with wg genkey
# - Port already in use: lsof -i :51820
```
---
## Verification Checklist
Before proceeding to Day 5:
- [ ] WireGuard installed on both VPS and GPU server
- [ ] VPN tunnel established (wg show shows handshake)
- [ ] GPU server can ping VPS (10.8.0.1)
- [ ] VPS can ping GPU server (10.8.0.2)
- [ ] Firewall allows WireGuard (UDP 51820)
- [ ] PostgreSQL accessible from GPU server
- [ ] WireGuard starts on boot (systemctl enable)
---
## Network Reference
**VPN IPs:**
- VPS: `10.8.0.1`
- GPU Server: `10.8.0.2`
**Service Access from GPU Server:**
- PostgreSQL: `postgresql://valknar:password@10.8.0.1:5432/dbname`
- Redis: `10.8.0.1:6379`
- LiteLLM: `http://10.8.0.1:4000`
- Mailpit: `10.8.0.1:1025`
**Service Access from VPS:**
- vLLM: `http://10.8.0.2:8000`
- ComfyUI: `http://10.8.0.2:8188`
- JupyterLab: `http://10.8.0.2:8888`
---
## Next: Docker & GPU Setup
Once VPN is working, proceed to **Day 5: Docker & NVIDIA Container Toolkit Setup**.
**Save connection info:**
```bash
# On GPU server
cat >> /workspace/SERVER_INFO.md << 'EOF'
## VPN Configuration
- VPN IP: 10.8.0.2
- VPS VPN IP: 10.8.0.1
- WireGuard Status: Active
- Latest Handshake: [Check with: wg show]
## Network Access
- Can reach VPS services: ✓
- VPS can reach GPU services: ✓
EOF
```

206
compose.yaml
View File

@@ -1,206 +0,0 @@
services:
# PostgreSQL with pgvector for AI/RAG workloads
ai_postgres:
image: ${AI_POSTGRES_IMAGE:-pgvector/pgvector:pg16}
container_name: ${AI_COMPOSE_PROJECT_NAME}_postgres
restart: unless-stopped
environment:
TZ: ${TIMEZONE:-Europe/Berlin}
POSTGRES_USER: ${AI_DB_USER}
POSTGRES_PASSWORD: ${AI_DB_PASSWORD}
POSTGRES_DB: ${AI_DB_NAME}
POSTGRES_HOST_AUTH_METHOD: scram-sha-256
POSTGRES_INITDB_ARGS: --auth-host=scram-sha-256
volumes:
- ai_postgres_data:/var/lib/postgresql/data
- ./postgres/init:/docker-entrypoint-initdb.d
healthcheck:
test: ['CMD-SHELL', 'pg_isready -U ${AI_DB_USER}']
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
networks:
- compose_network
# Open WebUI - ChatGPT-like interface for AI models
webui:
image: ${AI_WEBUI_IMAGE:-ghcr.io/open-webui/open-webui:main}
container_name: ${AI_COMPOSE_PROJECT_NAME}_webui
restart: unless-stopped
environment:
TZ: ${TIMEZONE:-Europe/Berlin}
# Database configuration
DATABASE_URL: postgresql://${AI_DB_USER}:${AI_DB_PASSWORD}@ai_postgres:5432/${AI_DB_NAME}
# OpenAI API configuration (pointing to LiteLLM proxy)
OPENAI_API_BASE_URLS: http://litellm:4000
OPENAI_API_KEYS: ${AI_LITELLM_API_KEY}
# WebUI configuration
WEBUI_NAME: ${AI_WEBUI_NAME:-Pivoine AI}
WEBUI_URL: https://${AI_TRAEFIK_HOST}
WEBUI_SECRET_KEY: ${AI_WEBUI_SECRET_KEY}
# Feature flags
ENABLE_SIGNUP: ${AI_ENABLE_SIGNUP:-true}
ENABLE_RAG_WEB_SEARCH: ${AI_ENABLE_RAG_WEB_SEARCH:-true}
ENABLE_RAG_WEB_LOADER_SSL_VERIFICATION: ${AI_ENABLE_RAG_SSL_VERIFY:-true}
# RAG configuration
RAG_EMBEDDING_ENGINE: ${AI_RAG_EMBEDDING_ENGINE:-openai}
RAG_EMBEDDING_MODEL: ${AI_RAG_EMBEDDING_MODEL:-text-embedding-3-small}
VECTOR_DB: ${AI_VECTOR_DB:-pgvector}
# Email configuration (Mailpit SMTP relay)
SMTP_HOST: net_mailpit
SMTP_PORT: 1025
SMTP_FROM_EMAIL: ${EMAIL_FROM}
SMTP_USE_TLS: false
SMTP_USE_SSL: false
volumes:
- ai_webui_data:/app/backend/data
depends_on:
- ai_postgres
- litellm
networks:
- compose_network
labels:
- 'traefik.enable=${AI_TRAEFIK_ENABLED}'
# HTTP to HTTPS redirect
- 'traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-redirect-web-secure.redirectscheme.scheme=https'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web.middlewares=${AI_COMPOSE_PROJECT_NAME}-redirect-web-secure'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web.rule=Host(`${AI_TRAEFIK_HOST}`)'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web.entrypoints=web'
# HTTPS router
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web-secure.rule=Host(`${AI_TRAEFIK_HOST}`)'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web-secure.tls.certresolver=resolver'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web-secure.entrypoints=web-secure'
- 'traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-web-secure-compress.compress=true'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-web-secure.middlewares=${AI_COMPOSE_PROJECT_NAME}-web-secure-compress,security-headers@file'
# Service
- 'traefik.http.services.${AI_COMPOSE_PROJECT_NAME}-web-secure.loadbalancer.server.port=8080'
- 'traefik.docker.network=${NETWORK_NAME}'
# Watchtower
- 'com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}'
# LiteLLM - Proxy to convert Anthropic API to OpenAI-compatible format
litellm:
image: ghcr.io/berriai/litellm:main-latest
container_name: ${AI_COMPOSE_PROJECT_NAME}_litellm
restart: unless-stopped
environment:
TZ: ${TIMEZONE:-Europe/Berlin}
ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
LITELLM_MASTER_KEY: ${AI_LITELLM_API_KEY}
DATABASE_URL: postgresql://${AI_DB_USER}:${AI_DB_PASSWORD}@ai_postgres:5432/litellm
LITELLM_DROP_PARAMS: 'true'
NO_DOCS: 'true'
NO_REDOC: 'true'
# Performance optimizations
LITELLM_LOG: 'ERROR' # Only log errors
LITELLM_MODE: 'PRODUCTION' # Production mode for better performance
volumes:
- ./litellm-config.yaml:/app/litellm-config.yaml:ro
command:
[
'--config',
'/app/litellm-config.yaml',
'--host',
'0.0.0.0',
'--port',
'4000',
'--drop_params'
]
depends_on:
- ai_postgres
networks:
- compose_network
healthcheck:
disable: true
labels:
- 'traefik.enable=${AI_TRAEFIK_ENABLED}'
# HTTP to HTTPS redirect
- 'traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-litellm-redirect-web-secure.redirectscheme.scheme=https'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web.middlewares=${AI_COMPOSE_PROJECT_NAME}-litellm-redirect-web-secure'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web.rule=Host(`${AI_LITELLM_TRAEFIK_HOST}`)'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web.entrypoints=web'
# HTTPS router
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure.rule=Host(`${AI_LITELLM_TRAEFIK_HOST}`)'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure.tls.certresolver=resolver'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure.entrypoints=web-secure'
- 'traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure-compress.compress=true'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure.middlewares=${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure-compress,security-headers@file'
# Service
- 'traefik.http.services.${AI_COMPOSE_PROJECT_NAME}-litellm-web-secure.loadbalancer.server.port=4000'
- 'traefik.docker.network=${NETWORK_NAME}'
# Watchtower
- 'com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}'
# Crawl4AI - Web scraping for LLMs (internal API, no public access)
crawl4ai:
image: ${AI_CRAWL4AI_IMAGE:-unclecode/crawl4ai:latest}
container_name: ${AI_COMPOSE_PROJECT_NAME}_crawl4ai
restart: unless-stopped
environment:
TZ: ${TIMEZONE:-Europe/Berlin}
# API configuration
PORT: ${AI_CRAWL4AI_PORT:-11235}
volumes:
- ai_crawl4ai_data:/app/.crawl4ai
networks:
- compose_network
labels:
# No Traefik exposure - internal only
- 'traefik.enable=false'
# Watchtower
- 'com.centurylinklabs.watchtower.enable=${WATCHTOWER_LABEL_ENABLE}'
# Facefusion - AI face swapping and enhancement
facefusion:
build:
context: .
dockerfile: Dockerfile
image: facefusion-patched:3.5.0-cpu
container_name: ${AI_COMPOSE_PROJECT_NAME}_facefusion
restart: unless-stopped
tty: true
command: ['python', '-u', 'facefusion.py', 'run']
environment:
TZ: ${TIMEZONE:-Europe/Berlin}
GRADIO_SERVER_NAME: "0.0.0.0"
GRADIO_SERVER_PORT: "7860"
volumes:
- ai_facefusion_data:/workspace
networks:
- compose_network
labels:
- 'traefik.enable=${AI_FACEFUSION_TRAEFIK_ENABLED}'
# HTTP to HTTPS redirect
- 'traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-facefusion-redirect-web-secure.redirectscheme.scheme=https'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web.middlewares=${AI_COMPOSE_PROJECT_NAME}-facefusion-redirect-web-secure'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web.rule=Host(`${AI_FACEFUSION_TRAEFIK_HOST}`)'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web.entrypoints=web'
# HTTPS router with Authelia
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure.rule=Host(`${AI_FACEFUSION_TRAEFIK_HOST}`)'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure.tls.certresolver=resolver'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure.entrypoints=web-secure'
- 'traefik.http.middlewares.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure-compress.compress=true'
- 'traefik.http.routers.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure.middlewares=${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure-compress,net-authelia,security-headers@file'
# Service
- 'traefik.http.services.${AI_COMPOSE_PROJECT_NAME}-facefusion-web-secure.loadbalancer.server.port=7860'
- 'traefik.docker.network=${NETWORK_NAME}'
# Watchtower - disabled for custom local image
- 'com.centurylinklabs.watchtower.enable=false'
volumes:
ai_postgres_data:
name: ${AI_COMPOSE_PROJECT_NAME}_postgres_data
ai_webui_data:
name: ${AI_COMPOSE_PROJECT_NAME}_webui_data
ai_crawl4ai_data:
name: ${AI_COMPOSE_PROJECT_NAME}_crawl4ai_data
ai_facefusion_data:
name: ${AI_COMPOSE_PROJECT_NAME}_facefusion_data

229
deploy-gpu-stack.sh
View File

@@ -1,229 +0,0 @@
#!/bin/bash
# GPU Stack Deployment Script
# Run this on the GPU server after SSH access is established
set -e # Exit on error
echo "=================================="
echo "GPU Stack Deployment Script"
echo "=================================="
echo ""
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Functions
print_success() {
echo -e "${GREEN}✓ $1${NC}"
}
print_error() {
echo -e "${RED}✗ $1${NC}"
}
print_info() {
echo -e "${YELLOW}→ $1${NC}"
}
# Check if running as root
if [[ $EUID -ne 0 ]]; then
print_error "This script must be run as root (use sudo)"
exit 1
fi
# Step 1: Check prerequisites
print_info "Checking prerequisites..."
if ! command -v docker &> /dev/null; then
print_error "Docker is not installed. Please run DOCKER_GPU_SETUP.md first."
exit 1
fi
print_success "Docker installed"
if ! command -v nvidia-smi &> /dev/null; then
print_error "nvidia-smi not found. Is this a GPU server?"
exit 1
fi
print_success "NVIDIA GPU detected"
if ! docker run --rm --runtime=nvidia --gpus all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi &> /dev/null; then
print_error "Docker cannot access GPU. Please configure NVIDIA Container Toolkit."
exit 1
fi
print_success "Docker GPU access working"
# Step 2: Create directory structure
print_info "Creating directory structure..."
mkdir -p /workspace/gpu-stack/{vllm,comfyui,training/{configs,data,output},notebooks,monitoring}
cd /workspace/gpu-stack
print_success "Directory structure created"
# Step 3: Create .env file
if [ ! -f .env ]; then
print_info "Creating .env file..."
cat > .env << 'EOF'
# GPU Stack Environment Variables
# Timezone
TIMEZONE=Europe/Berlin
# VPN Network
VPS_IP=10.8.0.1
GPU_IP=10.8.0.2
# Model Storage (network volume)
MODELS_PATH=/workspace/models
# Hugging Face Token (optional, for gated models like Llama)
# Get from: https://huggingface.co/settings/tokens
HF_TOKEN=
# Weights & Biases (optional, for training logging)
# Get from: https://wandb.ai/authorize
WANDB_API_KEY=
# JupyterLab Access Token
JUPYTER_TOKEN=pivoine-ai-2025
# PostgreSQL (on VPS)
DB_HOST=10.8.0.1
DB_PORT=5432
DB_USER=valknar
DB_PASSWORD=ragnarok98
DB_NAME=openwebui
EOF
chmod 600 .env
print_success ".env file created (please edit with your tokens)"
else
print_success ".env file already exists"
fi
# Step 4: Download docker-compose.yaml
print_info "Downloading docker-compose.yaml..."
# In production, this would be copied from the repo
# For now, assume it's already in the current directory
if [ ! -f docker-compose.yaml ]; then
print_error "docker-compose.yaml not found. Please copy gpu-server-compose.yaml to docker-compose.yaml"
exit 1
fi
print_success "docker-compose.yaml found"
# Step 5: Pre-download models (optional but recommended)
print_info "Do you want to pre-download models? (y/n)"
read -r response
if [[ "$response" =~ ^[Yy]$ ]]; then
print_info "Downloading Llama 3.1 8B Instruct (this will take a while)..."
mkdir -p /workspace/models
# Use huggingface-cli to download
pip install -q huggingface-hub
huggingface-cli download \
meta-llama/Meta-Llama-3.1-8B-Instruct \
--local-dir /workspace/models/Meta-Llama-3.1-8B-Instruct \
--local-dir-use-symlinks False || print_error "Model download failed (may need HF_TOKEN)"
print_success "Model downloaded to /workspace/models"
fi
# Step 6: Start services
print_info "Starting GPU stack services..."
docker compose up -d vllm comfyui jupyter netdata
print_success "Services starting (this may take a few minutes)..."
# Step 7: Wait for services
print_info "Waiting for services to be ready..."
sleep 10
# Check service health
print_info "Checking service status..."
if docker ps | grep -q gpu_vllm; then
print_success "vLLM container running"
else
print_error "vLLM container not running"
fi
if docker ps | grep -q gpu_comfyui; then
print_success "ComfyUI container running"
else
print_error "ComfyUI container not running"
fi
if docker ps | grep -q gpu_jupyter; then
print_success "JupyterLab container running"
else
print_error "JupyterLab container not running"
fi
if docker ps | grep -q gpu_netdata; then
print_success "Netdata container running"
else
print_error "Netdata container not running"
fi
# Step 8: Display access information
echo ""
echo "=================================="
echo "Deployment Complete!"
echo "=================================="
echo ""
echo "Services accessible via VPN (from VPS):"
echo " - vLLM API: http://10.8.0.2:8000"
echo " - ComfyUI: http://10.8.0.2:8188"
echo " - JupyterLab: http://10.8.0.2:8888 (token: pivoine-ai-2025)"
echo " - Netdata: http://10.8.0.2:19999"
echo ""
echo "Local access (from GPU server):"
echo " - vLLM API: http://localhost:8000"
echo " - ComfyUI: http://localhost:8188"
echo " - JupyterLab: http://localhost:8888"
echo " - Netdata: http://localhost:19999"
echo ""
echo "Useful commands:"
echo " - View logs: docker compose logs -f"
echo " - Check status: docker compose ps"
echo " - Stop all: docker compose down"
echo " - Restart service: docker compose restart vllm"
echo " - Start training: docker compose --profile training up -d axolotl"
echo ""
echo "Next steps:"
echo " 1. Wait for vLLM to load model (check logs: docker compose logs -f vllm)"
echo " 2. Test vLLM: curl http://localhost:8000/v1/models"
echo " 3. Configure LiteLLM on VPS to use http://10.8.0.2:8000"
echo " 4. Download ComfyUI models via web interface"
echo ""
# Step 9: Create helpful aliases
print_info "Creating helpful aliases..."
cat >> ~/.bashrc << 'EOF'
# GPU Stack Aliases
alias gpu-logs='cd /workspace/gpu-stack && docker compose logs -f'
alias gpu-ps='cd /workspace/gpu-stack && docker compose ps'
alias gpu-restart='cd /workspace/gpu-stack && docker compose restart'
alias gpu-down='cd /workspace/gpu-stack && docker compose down'
alias gpu-up='cd /workspace/gpu-stack && docker compose up -d'
alias gpu-stats='watch -n 1 nvidia-smi'
alias gpu-top='nvtop'
EOF
print_success "Aliases added to ~/.bashrc (reload with: source ~/.bashrc)"
echo ""
print_success "All done! 🚀"

12
disable-nsfw-filter.patch
View File

@@ -1,12 +0,0 @@
--- a/facefusion/content_analyser.py
+++ b/facefusion/content_analyser.py
@@ -194,7 +194,8 @@ def analyse_frame(vision_frame : VisionFrame) -> bool:
is_nsfw_2 = detect_with_nsfw_2(vision_frame)
is_nsfw_3 = detect_with_nsfw_3(vision_frame)
- return is_nsfw_1 and is_nsfw_2 or is_nsfw_1 and is_nsfw_3 or is_nsfw_2 and is_nsfw_3
+ # Patched to disable NSFW filter - always return False (content is safe)
+ return False
def detect_with_nsfw_1(vision_frame : VisionFrame) -> bool:

0
DEPLOYMENT.md → docs/DEPLOYMENT.md
View File

0
GPU_DEPLOYMENT_LOG.md → docs/GPU_DEPLOYMENT_LOG.md
View File

0
RUNPOD_TEMPLATE.md → docs/RUNPOD_TEMPLATE.md
View File

16
entrypoint.sh
View File

@@ -1,16 +0,0 @@
#!/bin/sh
echo "Patching Facefusion to disable NSFW filter..."
# Patch content_analyser.py line 197 to always return False (content is safe)
sed -i '197s/.*/\treturn False # Patched: NSFW filter disabled/' /facefusion/facefusion/content_analyser.py
# Verify the patch was applied
if grep -q 'return False.*Patched' /facefusion/facefusion/content_analyser.py; then
echo "NSFW filter successfully disabled"
else
echo "ERROR: Patch failed!"
exit 1
fi
echo "Starting Facefusion..."
cd /facefusion && exec python -u facefusion.py run

237
gpu-server-compose.yaml
View File

@@ -1,237 +0,0 @@
# GPU Server Docker Compose Configuration
# Deploy on RunPod GPU server (10.8.0.2)
# Services accessible from VPS (10.8.0.1) via WireGuard VPN
version: '3.8'
services:
# =============================================================================
# vLLM - High-performance LLM Inference Server
# =============================================================================
vllm:
image: vllm/vllm-openai:latest
container_name: gpu_vllm
restart: unless-stopped
runtime: nvidia
environment:
NVIDIA_VISIBLE_DEVICES: all
CUDA_VISIBLE_DEVICES: "0"
HF_TOKEN: ${HF_TOKEN:-}
volumes:
- ${MODELS_PATH:-/workspace/models}:/root/.cache/huggingface
command:
- --model
- meta-llama/Meta-Llama-3.1-8B-Instruct # Change model here
- --host
- 0.0.0.0
- --port
- 8000
- --tensor-parallel-size
- "1"
- --gpu-memory-utilization
- "0.85" # Leave 15% for other tasks
- --max-model-len
- "8192"
- --dtype
- auto
- --trust-remote-code
ports:
- "8000:8000"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 120s # Model loading takes time
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
labels:
- "service=vllm"
- "stack=gpu-ai"
# =============================================================================
# ComfyUI - Advanced Stable Diffusion Interface
# =============================================================================
comfyui:
image: ghcr.io/ai-dock/comfyui:latest
container_name: gpu_comfyui
restart: unless-stopped
runtime: nvidia
environment:
NVIDIA_VISIBLE_DEVICES: all
TZ: ${TIMEZONE:-Europe/Berlin}
# ComfyUI auto-installs custom nodes on first run
COMFYUI_FLAGS: "--listen 0.0.0.0 --port 8188"
volumes:
- comfyui_data:/data
- ${MODELS_PATH:-/workspace/models}/comfyui:/opt/ComfyUI/models
- comfyui_output:/opt/ComfyUI/output
- comfyui_input:/opt/ComfyUI/input
ports:
- "8188:8188"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8188/"]
interval: 30s
timeout: 10s
retries: 3
start_period: 60s
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
labels:
- "service=comfyui"
- "stack=gpu-ai"
# =============================================================================
# Axolotl - LLM Fine-tuning Framework
# =============================================================================
# Note: This service uses "profiles" - only starts when explicitly requested
# Start with: docker compose --profile training up -d axolotl
axolotl:
image: winglian/axolotl:main-py3.11-cu121-2.2.2
container_name: gpu_training
runtime: nvidia
volumes:
- ./training/configs:/workspace/configs
- ./training/data:/workspace/data
- ./training/output:/workspace/output
- ${MODELS_PATH:-/workspace/models}:/workspace/models
- training_cache:/root/.cache
environment:
NVIDIA_VISIBLE_DEVICES: all
WANDB_API_KEY: ${WANDB_API_KEY:-}
HF_TOKEN: ${HF_TOKEN:-}
working_dir: /workspace
# Default command - override when running specific training
command: sleep infinity
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
profiles:
- training
labels:
- "service=axolotl"
- "stack=gpu-ai"
# =============================================================================
# JupyterLab - Interactive Development Environment
# =============================================================================
jupyter:
image: pytorch/pytorch:2.3.0-cuda12.1-cudnn8-devel
container_name: gpu_jupyter
restart: unless-stopped
runtime: nvidia
volumes:
- ./notebooks:/workspace/notebooks
- ${MODELS_PATH:-/workspace/models}:/workspace/models
- jupyter_cache:/root/.cache
ports:
- "8888:8888"
environment:
NVIDIA_VISIBLE_DEVICES: all
JUPYTER_ENABLE_LAB: "yes"
JUPYTER_TOKEN: ${JUPYTER_TOKEN:-pivoine-ai-2025}
HF_TOKEN: ${HF_TOKEN:-}
command: |
bash -c "
pip install --quiet jupyterlab transformers datasets accelerate bitsandbytes peft trl sentencepiece protobuf &&
jupyter lab --ip=0.0.0.0 --port=8888 --allow-root --no-browser --NotebookApp.token='${JUPYTER_TOKEN:-pivoine-ai-2025}'
"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8888/"]
interval: 30s
timeout: 10s
retries: 3
start_period: 60s
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
labels:
- "service=jupyter"
- "stack=gpu-ai"
# =============================================================================
# Netdata - System & GPU Monitoring
# =============================================================================
netdata:
image: netdata/netdata:latest
container_name: gpu_netdata
restart: unless-stopped
runtime: nvidia
hostname: gpu-runpod
cap_add:
- SYS_PTRACE
- SYS_ADMIN
security_opt:
- apparmor:unconfined
environment:
NVIDIA_VISIBLE_DEVICES: all
TZ: ${TIMEZONE:-Europe/Berlin}
volumes:
- /sys:/host/sys:ro
- /proc:/host/proc:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- /etc/os-release:/host/etc/os-release:ro
- netdata_config:/etc/netdata
- netdata_cache:/var/cache/netdata
- netdata_lib:/var/lib/netdata
ports:
- "19999:19999"
labels:
- "service=netdata"
- "stack=gpu-ai"
# =============================================================================
# Volumes
# =============================================================================
volumes:
# ComfyUI data
comfyui_data:
driver: local
comfyui_output:
driver: local
comfyui_input:
driver: local
# Training data
training_cache:
driver: local
# Jupyter data
jupyter_cache:
driver: local
# Netdata data
netdata_config:
driver: local
netdata_cache:
driver: local
netdata_lib:
driver: local
# =============================================================================
# Networks
# =============================================================================
networks:
default:
driver: bridge
ipam:
config:
- subnet: 172.25.0.0/24

199
litellm-config-gpu.yaml
View File

@@ -1,199 +0,0 @@
# LiteLLM Configuration with GPU Server Integration
# This config includes both Anthropic Claude (API) and self-hosted models (vLLM on GPU server)
model_list:
# =============================================================================
# Anthropic Claude Models (API-based, for complex reasoning)
# =============================================================================
- model_name: claude-sonnet-4
litellm_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-sonnet-4.5
litellm_params:
model: anthropic/claude-sonnet-4-5-20250929
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-3-5-sonnet
litellm_params:
model: anthropic/claude-3-5-sonnet-20241022
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-3-opus
litellm_params:
model: anthropic/claude-3-opus-20240229
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-3-haiku
litellm_params:
model: anthropic/claude-3-haiku-20240307
api_key: os.environ/ANTHROPIC_API_KEY
# =============================================================================
# Self-Hosted Models (vLLM on GPU server via WireGuard VPN)
# =============================================================================
# Llama 3.1 8B Instruct - Fast, general-purpose, good for routine tasks
- model_name: llama-3.1-8b
litellm_params:
model: openai/meta-llama/Meta-Llama-3.1-8B-Instruct
api_base: http://10.8.0.2:8000/v1
api_key: dummy # vLLM doesn't require auth
rpm: 1000 # Rate limit: requests per minute
tpm: 100000 # Rate limit: tokens per minute
# Alternative models (uncomment and configure on GPU server as needed)
# Qwen 2.5 14B Instruct - Excellent multilingual, stronger reasoning
# - model_name: qwen-2.5-14b
# litellm_params:
# model: openai/Qwen/Qwen2.5-14B-Instruct
# api_base: http://10.8.0.2:8000/v1
# api_key: dummy
# rpm: 800
# tpm: 80000
# Mistral 7B Instruct - Very fast, lightweight
# - model_name: mistral-7b
# litellm_params:
# model: openai/mistralai/Mistral-7B-Instruct-v0.3
# api_base: http://10.8.0.2:8000/v1
# api_key: dummy
# rpm: 1200
# tpm: 120000
# DeepSeek Coder 6.7B - Code generation specialist
# - model_name: deepseek-coder-6.7b
# litellm_params:
# model: openai/deepseek-ai/deepseek-coder-6.7b-instruct
# api_base: http://10.8.0.2:8000/v1
# api_key: dummy
# rpm: 1000
# tpm: 100000
# =============================================================================
# Router Settings - Intelligent Model Selection
# =============================================================================
# Model aliases for easy switching in Open WebUI
model_name_map:
# Default model (self-hosted, fast)
gpt-3.5-turbo: llama-3.1-8b
# Power users can use Claude for complex tasks
gpt-4: claude-sonnet-4.5
gpt-4-turbo: claude-sonnet-4.5
# LiteLLM Settings
litellm_settings:
drop_params: true
set_verbose: false # Disable verbose logging for better performance
# Enable caching with Redis for better performance
cache: true
cache_params:
type: redis
host: redis
port: 6379
ttl: 3600 # Cache for 1 hour
# Force strip specific parameters globally
allowed_fails: 0
# Modify params before sending to provider
modify_params: true
# Enable success and failure logging but minimize overhead
success_callback: [] # Disable all success callbacks to reduce DB writes
failure_callback: [] # Disable all failure callbacks
# Router Settings
router_settings:
allowed_fails: 0
# Routing strategy: Try self-hosted first, fallback to Claude on failure
routing_strategy: simple-shuffle
# Cooldown for failed models
cooldown_time: 30 # seconds
# Drop unsupported parameters
default_litellm_params:
drop_params: true
# General Settings
general_settings:
disable_responses_id_security: true
# Disable spend tracking to reduce database overhead
disable_spend_logs: false # Keep enabled to track API vs GPU costs
# Disable tag tracking
disable_tag_tracking: true
# Disable daily spend updates
disable_daily_spend_logs: false # Keep enabled for cost analysis
# Master key for authentication (set via env var)
master_key: os.environ/LITELLM_MASTER_KEY
# Database for logging (optional but recommended for cost tracking)
database_url: os.environ/DATABASE_URL
# Enable OpenAPI docs
docs_url: /docs
# =============================================================================
# Usage Guidelines (for Open WebUI users)
# =============================================================================
#
# Model Selection Guide:
#
# Use llama-3.1-8b for:
# - General chat and Q&A
# - Simple code generation
# - Data extraction
# - Summarization
# - Translation
# - Most routine tasks
# Cost: ~$0/month (self-hosted)
# Speed: ~50-80 tokens/second
#
# Use qwen-2.5-14b for:
# - Complex reasoning
# - Multi-step problems
# - Advanced code generation
# - Multilingual tasks
# Cost: ~$0/month (self-hosted)
# Speed: ~30-50 tokens/second
#
# Use claude-sonnet-4.5 for:
# - Very complex reasoning
# - Long documents (200K context)
# - Production-critical code
# - When quality matters most
# Cost: ~$3/million input tokens, ~$15/million output tokens
# Speed: ~30-40 tokens/second
#
# Use claude-3-haiku for:
# - API fallback (if self-hosted down)
# - Very fast responses needed
# Cost: ~$0.25/million input tokens, ~$1.25/million output tokens
# Speed: ~60-80 tokens/second
#
# =============================================================================
# Health Check Configuration
health_check:
# Check vLLM health endpoint
enabled: true
interval: 30 # seconds
timeout: 5 # seconds
# Fallback Configuration
# If GPU server is down, automatically use Claude
fallback:
- ["llama-3.1-8b", "claude-3-haiku"]
- ["qwen-2.5-14b", "claude-sonnet-4.5"]

91
litellm-config.yaml
View File

@@ -1,91 +0,0 @@
model_list:
- model_name: claude-sonnet-4
litellm_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-sonnet-4.5
litellm_params:
model: anthropic/claude-sonnet-4-5-20250929
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-3-5-sonnet
litellm_params:
model: anthropic/claude-3-5-sonnet-20241022
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-3-opus
litellm_params:
model: anthropic/claude-3-opus-20240229
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-3-haiku
litellm_params:
model: anthropic/claude-3-haiku-20240307
api_key: os.environ/ANTHROPIC_API_KEY
# ===========================================================================
# SELF-HOSTED MODELS VIA ORCHESTRATOR (GPU Server via Tailscale VPN)
# ===========================================================================
# All requests route through orchestrator (port 9000) which manages model loading
# Text Generation
- model_name: qwen-2.5-7b
litellm_params:
model: openai/qwen-2.5-7b
api_base: http://100.100.108.13:9000/v1 # Orchestrator endpoint
api_key: dummy
rpm: 1000
tpm: 100000
# Image Generation
- model_name: flux-schnell
litellm_params:
model: openai/dall-e-3 # OpenAI-compatible mapping
api_base: http://100.100.108.13:9000/v1 # Orchestrator endpoint
api_key: dummy
rpm: 100
max_parallel_requests: 3
# Music Generation
- model_name: musicgen-medium
litellm_params:
model: openai/musicgen-medium
api_base: http://100.100.108.13:9000/v1 # Orchestrator endpoint
api_key: dummy
rpm: 50
max_parallel_requests: 1
litellm_settings:
drop_params: true
set_verbose: false # Disable verbose logging for better performance
# Enable caching with Redis for better performance
cache: true
cache_params:
type: redis
host: redis
port: 6379
ttl: 3600 # Cache for 1 hour
# Force strip specific parameters globally
allowed_fails: 0
# Modify params before sending to provider
modify_params: true
# Enable success and failure logging but minimize overhead
success_callback: [] # Disable all success callbacks to reduce DB writes
failure_callback: [] # Disable all failure callbacks
router_settings:
allowed_fails: 0
# Drop unsupported parameters
default_litellm_params:
drop_params: true
general_settings:
disable_responses_id_security: true
# Disable spend tracking to reduce database overhead
disable_spend_logs: true
# Disable tag tracking
disable_tag_tracking: true
# Disable daily spend updates
disable_daily_spend_logs: true

38
postgres/init/01-init-databases.sh
View File

@@ -1,38 +0,0 @@
#!/bin/bash
set -e
# PostgreSQL initialization script for AI stack
# This script runs on first database initialization
# Creates all databases required by AI services
echo "Starting AI stack database initialization..."
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
-- Create databases for AI services
-- Open WebUI database
SELECT 'CREATE DATABASE openwebui'
WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'openwebui')\gexec
-- LiteLLM proxy database
SELECT 'CREATE DATABASE litellm'
WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'litellm')\gexec
-- Grant privileges to all databases
GRANT ALL PRIVILEGES ON DATABASE openwebui TO $POSTGRES_USER;
GRANT ALL PRIVILEGES ON DATABASE litellm TO $POSTGRES_USER;
-- Log success
SELECT 'AI stack databases initialized:' AS status;
SELECT datname FROM pg_database
WHERE datname IN ('openwebui', 'litellm')
ORDER BY datname;
EOSQL
echo ""
echo "✓ PostgreSQL initialization completed"
echo "✓ All AI stack databases created successfully"
echo ""
echo "Databases available:"
echo " • openwebui - Open WebUI application database"
echo " • litellm - LiteLLM proxy database"
echo ""

302
simple_vllm_server.py
View File

@@ -1,302 +0,0 @@
#!/usr/bin/env python3
"""
Simple vLLM server using AsyncLLMEngine directly
Bypasses the multiprocessing issues we hit with the default vLLM API server
OpenAI-compatible endpoints: /v1/models and /v1/completions
"""
import asyncio
import json
import logging
import os
from typing import AsyncIterator, Dict, List, Optional
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse, StreamingResponse
from pydantic import BaseModel, Field
from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams
from vllm.utils import random_uuid
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
# FastAPI app
app = FastAPI(title="Simple vLLM Server", version="1.0.0")
# Global engine instance
engine: Optional[AsyncLLMEngine] = None
model_name: str = "Qwen/Qwen2.5-7B-Instruct"
# Request/Response models
class CompletionRequest(BaseModel):
"""OpenAI-compatible completion request"""
model: str = Field(default="qwen-2.5-7b")
prompt: str | List[str] = Field(..., description="Text prompt(s)")
max_tokens: int = Field(default=512, ge=1, le=4096)
temperature: float = Field(default=0.7, ge=0.0, le=2.0)
top_p: float = Field(default=1.0, ge=0.0, le=1.0)
n: int = Field(default=1, ge=1, le=10)
stream: bool = Field(default=False)
stop: Optional[str | List[str]] = None
presence_penalty: float = Field(default=0.0, ge=-2.0, le=2.0)
frequency_penalty: float = Field(default=0.0, ge=-2.0, le=2.0)
class ChatMessage(BaseModel):
"""Chat message format"""
role: str = Field(..., description="Role: system, user, or assistant")
content: str = Field(..., description="Message content")
class ChatCompletionRequest(BaseModel):
"""OpenAI-compatible chat completion request"""
model: str = Field(default="qwen-2.5-7b")
messages: List[ChatMessage] = Field(..., description="Chat messages")
max_tokens: int = Field(default=512, ge=1, le=4096)
temperature: float = Field(default=0.7, ge=0.0, le=2.0)
top_p: float = Field(default=1.0, ge=0.0, le=1.0)
n: int = Field(default=1, ge=1, le=10)
stream: bool = Field(default=False)
stop: Optional[str | List[str]] = None
@app.on_event("startup")
async def startup_event():
"""Initialize vLLM engine on startup"""
global engine, model_name
logger.info(f"Initializing vLLM AsyncLLMEngine with model: {model_name}")
# Configure engine
engine_args = AsyncEngineArgs(
model=model_name,
tensor_parallel_size=1, # Single GPU
gpu_memory_utilization=0.85, # Use 85% of GPU memory
max_model_len=4096, # Context length
dtype="auto", # Auto-detect dtype
download_dir="/workspace/huggingface_cache", # Large disk
trust_remote_code=True, # Some models require this
enforce_eager=False, # Use CUDA graphs for better performance
)
# Create async engine
engine = AsyncLLMEngine.from_engine_args(engine_args)
logger.info("vLLM AsyncLLMEngine initialized successfully")
@app.get("/")
async def root():
"""Health check endpoint"""
return {"status": "ok", "model": model_name}
@app.get("/health")
async def health():
"""Detailed health check"""
return {
"status": "healthy" if engine else "initializing",
"model": model_name,
"ready": engine is not None
}
@app.get("/v1/models")
async def list_models():
"""OpenAI-compatible models endpoint"""
return {
"object": "list",
"data": [
{
"id": "qwen-2.5-7b",
"object": "model",
"created": 1234567890,
"owned_by": "pivoine-gpu",
"permission": [],
"root": model_name,
"parent": None,
}
]
}
def messages_to_prompt(messages: List[ChatMessage]) -> str:
"""Convert chat messages to a single prompt string"""
# Qwen 2.5 chat template format
prompt_parts = []
for msg in messages:
role = msg.role
content = msg.content
if role == "system":
prompt_parts.append(f"<|im_start|>system\n{content}<|im_end|>")
elif role == "user":
prompt_parts.append(f"<|im_start|>user\n{content}<|im_end|>")
elif role == "assistant":
prompt_parts.append(f"<|im_start|>assistant\n{content}<|im_end|>")
# Add final assistant prompt
prompt_parts.append("<|im_start|>assistant\n")
return "\n".join(prompt_parts)
@app.post("/v1/completions")
async def create_completion(request: CompletionRequest):
"""OpenAI-compatible completion endpoint"""
if not engine:
return JSONResponse(
status_code=503,
content={"error": "Engine not initialized"}
)
# Handle both single prompt and batch prompts
prompts = [request.prompt] if isinstance(request.prompt, str) else request.prompt
# Configure sampling parameters
sampling_params = SamplingParams(
temperature=request.temperature,
top_p=request.top_p,
max_tokens=request.max_tokens,
n=request.n,
stop=request.stop if request.stop else [],
presence_penalty=request.presence_penalty,
frequency_penalty=request.frequency_penalty,
)
# Generate completions
results = []
for prompt in prompts:
request_id = random_uuid()
if request.stream:
# Streaming response
async def generate_stream():
async for output in engine.generate(prompt, sampling_params, request_id):
chunk = {
"id": request_id,
"object": "text_completion",
"created": 1234567890,
"model": request.model,
"choices": [
{
"text": output.outputs[0].text,
"index": 0,
"logprobs": None,
"finish_reason": output.outputs[0].finish_reason,
}
]
}
yield f"data: {json.dumps(chunk)}\n\n"
yield "data: [DONE]\n\n"
return StreamingResponse(generate_stream(), media_type="text/event-stream")
else:
# Non-streaming response
async for output in engine.generate(prompt, sampling_params, request_id):
final_output = output
results.append({
"text": final_output.outputs[0].text,
"index": len(results),
"logprobs": None,
"finish_reason": final_output.outputs[0].finish_reason,
})
return {
"id": random_uuid(),
"object": "text_completion",
"created": 1234567890,
"model": request.model,
"choices": results,
"usage": {
"prompt_tokens": 0, # vLLM doesn't expose this easily
"completion_tokens": 0,
"total_tokens": 0,
}
}
@app.post("/v1/chat/completions")
async def create_chat_completion(request: ChatCompletionRequest):
"""OpenAI-compatible chat completion endpoint"""
if not engine:
return JSONResponse(
status_code=503,
content={"error": "Engine not initialized"}
)
# Convert messages to prompt
prompt = messages_to_prompt(request.messages)
# Configure sampling parameters
sampling_params = SamplingParams(
temperature=request.temperature,
top_p=request.top_p,
max_tokens=request.max_tokens,
n=request.n,
stop=request.stop if request.stop else ["<|im_end|>"],
)
request_id = random_uuid()
if request.stream:
# Streaming response
async def generate_stream():
async for output in engine.generate(prompt, sampling_params, request_id):
chunk = {
"id": request_id,
"object": "chat.completion.chunk",
"created": 1234567890,
"model": request.model,
"choices": [
{
"index": 0,
"delta": {"content": output.outputs[0].text},
"finish_reason": output.outputs[0].finish_reason,
}
]
}
yield f"data: {json.dumps(chunk)}\n\n"
yield "data: [DONE]\n\n"
return StreamingResponse(generate_stream(), media_type="text/event-stream")
else:
# Non-streaming response
async for output in engine.generate(prompt, sampling_params, request_id):
final_output = output
return {
"id": request_id,
"object": "chat.completion",
"created": 1234567890,
"model": request.model,
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": final_output.outputs[0].text,
},
"finish_reason": final_output.outputs[0].finish_reason,
}
],
"usage": {
"prompt_tokens": 0,
"completion_tokens": 0,
"total_tokens": 0,
}
}
if __name__ == "__main__":
import uvicorn
# Get configuration from environment
host = os.getenv("VLLM_HOST", "0.0.0.0")
port = int(os.getenv("VLLM_PORT", "8000"))
logger.info(f"Starting vLLM server on {host}:{port}")
uvicorn.run(
app,
host=host,
port=port,
log_level="info",
access_log=True,
)