runpod/COMFYUI_MODELS.md

# ComfyUI Models & Nodes - Usage Guide

This document explains how to use the extended playbook.yml for installing ComfyUI models and custom nodes.

## What's Been Added

### Model Categories

**Image Generation Models:**
- FLUX.1 Schnell (23GB, essential) - Fast 4-step inference
- FLUX.1 Dev (23GB, optional) - Balanced quality/speed
- SDXL Base 1.0 (7GB, essential) - Industry standard
- SDXL Refiner 1.0 (6GB, optional) - Enhances base output
- SD 3.5 Large (18GB, optional) - Latest Stability AI

**Video Generation Models:**
- CogVideoX-5B (20GB, essential) - Professional text-to-video
- SVD (8GB, essential) - 14 frame image-to-video
- SVD-XT (8GB, optional) - 25 frame image-to-video

**Audio Generation Models:**
- MusicGen Small (3GB, optional) - Fast generation
- MusicGen Medium (11GB, essential) - Balanced quality
- MusicGen Large (22GB, optional) - Highest quality

**Support Models:**
- CLIP H (2GB, essential) - For SD 1.5 IP-Adapter
- CLIP G (7GB, essential) - For SDXL IP-Adapter
- SigLIP (2GB, essential) - For FLUX models

### Custom Nodes

**Essential Nodes (installed by default):**
- ComfyUI-Manager - Install/manage custom nodes and models
- ComfyUI-VideoHelperSuite - Video operations
- ComfyUI-AnimateDiff-Evolved - Video generation
- ComfyUI_IPAdapter_plus - Style transfer
- ComfyUI-Impact-Pack - Auto face enhancement
- comfyui-sound-lab - Audio generation

**Optional Nodes:**
- ComfyUI-CogVideoXWrapper - CogVideoX integration
- Comfyui-Inspire-Pack - Additional tools
- ComfyUI-Advanced-ControlNet - Advanced control
- ComfyUI-3D-Pack - 3D asset generation

## Usage Examples

### Quick Setup (Essential Only)

Install ComfyUI with only essential models and nodes:

```bash
ansible-playbook playbook.yml --tags comfyui-essential
```

This downloads:
- Image: FLUX Schnell (23GB) + SDXL Base (7GB)
- Video: CogVideoX-5B (20GB) + SVD (8GB)
- Audio: MusicGen Medium (11GB)
- Support: All 3 CLIP models (11GB)
- **Total: ~80GB**

### Selective Installation

#### Install ComfyUI Base Only

```bash
ansible-playbook playbook.yml --tags comfyui
```

#### Install All Image Models

```bash
ansible-playbook playbook.yml --tags comfyui-models-image
```

#### Install All Video Models

```bash
ansible-playbook playbook.yml --tags comfyui-models-video
```

#### Install All Audio Models

```bash
ansible-playbook playbook.yml --tags comfyui-models-audio
```

#### Install Support Models (CLIP, IP-Adapter)

```bash
ansible-playbook playbook.yml --tags comfyui-models-support
```

#### Install Custom Nodes

```bash
ansible-playbook playbook.yml --tags comfyui-nodes
```

### Combined Installation

#### Full ComfyUI Setup

```bash
ansible-playbook playbook.yml --tags comfyui,comfyui-models-all,comfyui-nodes
```

#### ComfyUI + Image + Video Only

```bash
ansible-playbook playbook.yml --tags comfyui,comfyui-models-image,comfyui-models-video,comfyui-nodes
```

## Adding New Models

### Add Image Model

Edit `playbook.yml` and append to `comfyui_image_models`:

```yaml
- name: "author/model-name"
  type: "checkpoint"
  category: "image"
  size_gb: 15
  vram_gb: 18
  format: "fp16"
  description: "Model description"
  essential: false  # Set to true for default installs
```

### Add Custom Node

Append to `comfyui_custom_nodes`:

```yaml
- name: "NodeName"
  repo: "https://github.com/author/repo.git"
  category: "image"  # or video, audio, control, etc.
  description: "What it does"
  essential: false
```

## VRAM Requirements Summary

For your **24GB GPU**, you can run:

### Simultaneously Loadable Combinations

**Option 1: Image Generation Focus**
- FLUX Schnell FP16 (23GB) → Full model, leaves 1GB for operations
- FLUX Schnell FP8 (12GB) + SDXL (12GB) → Both loaded
- SDXL (12GB) + Refiner (12GB) → Both loaded with room

**Option 2: Video Generation**
- CogVideoX-5B with optimizations (12GB) + SDXL (12GB)
- SVD-XT (20GB) → Good frame count

**Option 3: Multi-Modal**
- SDXL (12GB) + MusicGen Medium (8GB)
- CogVideoX-5B (12GB) + MusicGen Small (4GB)

## Storage Requirements

### Essential Models Only
- Image: 30GB
- Video: 28GB
- Audio: 11GB
- Support: 11GB
- **Total: ~80GB**

### All Models (including optional)
- Image: 54GB
- Video: 36GB
- Audio: 36GB
- Support: 11GB
- **Total: ~137GB**

## Directory Structure Created

```
/workspace/ComfyUI/
├── models/
│   ├── checkpoints/        # FLUX, SDXL, SD3 models
│   ├── unet/               # Diffusion U-Nets
│   ├── vae/                # VAE models
│   ├── loras/              # LoRA adapters
│   ├── clip/               # CLIP text encoders
│   ├── clip_vision/        # CLIP vision models
│   ├── controlnet/         # ControlNet models
│   ├── ipadapter/          # IP-Adapter models
│   ├── embeddings/         # Text embeddings
│   ├── upscale_models/     # Upscalers
│   ├── video_models/       # Video generation models
│   ├── animatediff_models/ # AnimateDiff models
│   ├── audio_models/       # Audio generation models
│   └── configs/            # Model configs
└── custom_nodes/           # Extension nodes
```

## Extending the Playbook

### Add a New Model Category

1. Add variable list in `vars` section:

```yaml
comfyui_new_category_models:
  - name: "model-repo-id"
    type: "new_type"
    category: "new_category"
    size_gb: 10
    description: "Description"
    essential: true
```

2. Create download task section:

```yaml
- name: Download New Category Models
  tags: [comfyui-models-newcat, comfyui-models-all]
  block:
    # Copy pattern from existing model download tasks
```

3. Update tags documentation at top of file

## Model Download Notes

- **Async Downloads**: Large models download asynchronously with 1-hour timeouts
- **Smart Caching**: Models are only downloaded if not already cached
- **HuggingFace Token**: Set `HF_TOKEN` environment variable for gated models
- **Error Handling**: Download failures are ignored to allow partial installations

## Performance Tips

### For 24GB GPU:

1. **Use FP8 Quantized FLUX** when you need multiple models loaded
2. **Use Full FP16 FLUX** when you want maximum quality
3. **Enable CogVideoX Optimizations** to fit in 12GB (edit model config)
4. **Use GGUF variants** for FLUX to save VRAM (manually download)

### Recommended First Install:

```bash
# Essential setup (~80GB, ~1 hour)
ansible-playbook playbook.yml --tags comfyui-essential
```

Then add specific models as needed:

```bash
# Add video models later
ansible-playbook playbook.yml --tags comfyui-models-video
```

## Troubleshooting

### Downloads Failing

Check HuggingFace token:
```bash
export HF_TOKEN="your_token_here"
ansible-playbook playbook.yml --tags comfyui-models-image
```

### Out of Space

Check available storage:
```bash
df -h /workspace
du -sh /workspace/huggingface_cache
```

Remove optional models:
```bash
# Keep only essential models in playbook vars
```

### Custom Node Dependencies

Some nodes require manual intervention:
```bash
cd /workspace/ComfyUI/custom_nodes/NodeName
pip3 install -r requirements.txt
```

## Next Steps

After installation:

1. **Start ComfyUI:**
   ```bash
   cd /workspace/ComfyUI
   python3 main.py --listen 0.0.0.0 --port 8188
   ```

2. **Access via Browser:**
   ```
   http://your-runpod-ip:8188
   ```

3. **Install Additional Nodes via Manager:**
   - ComfyUI Manager provides GUI for installing more nodes
   - Browse and install from the manager interface

4. **Download Model Variants:**
   - Use ComfyUI Manager to download LoRAs
   - Download ControlNet models as needed
   - Get additional checkpoints from Civitai

## Support

For issues or questions:
- Check Ansible logs for download status
- Verify HuggingFace token is valid
- Ensure sufficient storage space
- Check VRAM requirements for your GPU