runpod/COMFYUI_MODELS.md

ComfyUI Models & Nodes - Usage Guide

This document explains how to use the extended playbook.yml for installing ComfyUI models and custom nodes.

What's Been Added

Model Categories

Image Generation Models:

• FLUX.1 Schnell (23GB, essential) - Fast 4-step inference
• FLUX.1 Dev (23GB, optional) - Balanced quality/speed
• SDXL Base 1.0 (7GB, essential) - Industry standard
• SDXL Refiner 1.0 (6GB, optional) - Enhances base output
• SD 3.5 Large (18GB, optional) - Latest Stability AI

Video Generation Models:

• CogVideoX-5B (20GB, essential) - Professional text-to-video
• SVD (8GB, essential) - 14 frame image-to-video
• SVD-XT (8GB, optional) - 25 frame image-to-video

Audio Generation Models:

• MusicGen Small (3GB, optional) - Fast generation
• MusicGen Medium (11GB, essential) - Balanced quality
• MusicGen Large (22GB, optional) - Highest quality

Support Models:

• CLIP H (2GB, essential) - For SD 1.5 IP-Adapter
• CLIP G (7GB, essential) - For SDXL IP-Adapter
• SigLIP (2GB, essential) - For FLUX models

Custom Nodes

Essential Nodes (installed by default):

• ComfyUI-Manager - Install/manage custom nodes and models
• ComfyUI-VideoHelperSuite - Video operations
• ComfyUI-AnimateDiff-Evolved - Video generation
• ComfyUI_IPAdapter_plus - Style transfer
• ComfyUI-Impact-Pack - Auto face enhancement
• comfyui-sound-lab - Audio generation

Optional Nodes:

• ComfyUI-CogVideoXWrapper - CogVideoX integration
• Comfyui-Inspire-Pack - Additional tools
• ComfyUI-Advanced-ControlNet - Advanced control
• ComfyUI-3D-Pack - 3D asset generation

Usage Examples

Quick Setup (Essential Only)

Install ComfyUI with only essential models and nodes:

ansible-playbook playbook.yml --tags comfyui-essential

This downloads:

• Image: FLUX Schnell (23GB) + SDXL Base (7GB)
• Video: CogVideoX-5B (20GB) + SVD (8GB)
• Audio: MusicGen Medium (11GB)
• Support: All 3 CLIP models (11GB)
• Total: ~80GB

Selective Installation

Install ComfyUI Base Only

ansible-playbook playbook.yml --tags comfyui

Install All Image Models

ansible-playbook playbook.yml --tags comfyui-models-image

Install All Video Models

ansible-playbook playbook.yml --tags comfyui-models-video

Install All Audio Models

ansible-playbook playbook.yml --tags comfyui-models-audio

Install Support Models (CLIP, IP-Adapter)

ansible-playbook playbook.yml --tags comfyui-models-support

Install Custom Nodes

ansible-playbook playbook.yml --tags comfyui-nodes

Combined Installation

Full ComfyUI Setup

ansible-playbook playbook.yml --tags comfyui,comfyui-models-all,comfyui-nodes

ComfyUI + Image + Video Only

ansible-playbook playbook.yml --tags comfyui,comfyui-models-image,comfyui-models-video,comfyui-nodes

Adding New Models

Add Image Model

Edit playbook.yml and append to comfyui_image_models:

- name: "author/model-name"
  type: "checkpoint"
  category: "image"
  size_gb: 15
  vram_gb: 18
  format: "fp16"
  description: "Model description"
  essential: false  # Set to true for default installs

Add Custom Node

Append to comfyui_custom_nodes:

- name: "NodeName"
  repo: "https://github.com/author/repo.git"
  category: "image"  # or video, audio, control, etc.
  description: "What it does"
  essential: false

VRAM Requirements Summary

For your 24GB GPU, you can run:

Simultaneously Loadable Combinations

Option 1: Image Generation Focus

• FLUX Schnell FP16 (23GB) → Full model, leaves 1GB for operations
• FLUX Schnell FP8 (12GB) + SDXL (12GB) → Both loaded
• SDXL (12GB) + Refiner (12GB) → Both loaded with room

Option 2: Video Generation

• CogVideoX-5B with optimizations (12GB) + SDXL (12GB)
• SVD-XT (20GB) → Good frame count

Option 3: Multi-Modal

• SDXL (12GB) + MusicGen Medium (8GB)
• CogVideoX-5B (12GB) + MusicGen Small (4GB)

Storage Requirements

Essential Models Only

• Image: 30GB
• Video: 28GB
• Audio: 11GB
• Support: 11GB
• Total: ~80GB

All Models (including optional)

• Image: 54GB
• Video: 36GB
• Audio: 36GB
• Support: 11GB
• Total: ~137GB

Directory Structure Created

/workspace/ComfyUI/
├── models/
│   ├── checkpoints/        # FLUX, SDXL, SD3 models
│   ├── unet/               # Diffusion U-Nets
│   ├── vae/                # VAE models
│   ├── loras/              # LoRA adapters
│   ├── clip/               # CLIP text encoders
│   ├── clip_vision/        # CLIP vision models
│   ├── controlnet/         # ControlNet models
│   ├── ipadapter/          # IP-Adapter models
│   ├── embeddings/         # Text embeddings
│   ├── upscale_models/     # Upscalers
│   ├── video_models/       # Video generation models
│   ├── animatediff_models/ # AnimateDiff models
│   ├── audio_models/       # Audio generation models
│   └── configs/            # Model configs
└── custom_nodes/           # Extension nodes

Extending the Playbook

Add a New Model Category

1. Add variable list in vars section:

comfyui_new_category_models:
  - name: "model-repo-id"
    type: "new_type"
    category: "new_category"
    size_gb: 10
    description: "Description"
    essential: true

2. Create download task section:

- name: Download New Category Models
  tags: [comfyui-models-newcat, comfyui-models-all]
  block:
    # Copy pattern from existing model download tasks

3. Update tags documentation at top of file

Model Download Notes

• Async Downloads: Large models download asynchronously with 1-hour timeouts
• Smart Caching: Models are only downloaded if not already cached
• HuggingFace Token: Set HF_TOKEN environment variable for gated models
• Error Handling: Download failures are ignored to allow partial installations

Performance Tips

For 24GB GPU:

1. Use FP8 Quantized FLUX when you need multiple models loaded
2. Use Full FP16 FLUX when you want maximum quality
3. Enable CogVideoX Optimizations to fit in 12GB (edit model config)
4. Use GGUF variants for FLUX to save VRAM (manually download)

Recommended First Install:

# Essential setup (~80GB, ~1 hour)
ansible-playbook playbook.yml --tags comfyui-essential

Then add specific models as needed:

# Add video models later
ansible-playbook playbook.yml --tags comfyui-models-video

Troubleshooting

Downloads Failing

Check HuggingFace token:

export HF_TOKEN="your_token_here"
ansible-playbook playbook.yml --tags comfyui-models-image

Out of Space

Check available storage:

df -h /workspace
du -sh /workspace/huggingface_cache

Remove optional models:

# Keep only essential models in playbook vars

Custom Node Dependencies

Some nodes require manual intervention:

cd /workspace/ComfyUI/custom_nodes/NodeName
pip3 install -r requirements.txt

Next Steps

After installation:

1. Start ComfyUI:

   cd /workspace/ComfyUI
   python3 main.py --listen 0.0.0.0 --port 8188
   

2. Access via Browser:

   http://your-runpod-ip:8188
   

3. Install Additional Nodes via Manager:

   • ComfyUI Manager provides GUI for installing more nodes
   • Browse and install from the manager interface

4. Download Model Variants:

   • Use ComfyUI Manager to download LoRAs
   • Download ControlNet models as needed
   • Get additional checkpoints from Civitai

Support

For issues or questions:

• Check Ansible logs for download status
• Verify HuggingFace token is valid
• Ensure sufficient storage space
• Check VRAM requirements for your GPU