runpod/models/comfyui/workflows/README.md

# ComfyUI Production Workflows

Comprehensive collection of production-ready ComfyUI workflows for RunPod AI Model Orchestrator.

## Overview

This directory contains 20 sophisticated, battle-tested workflows designed for production use with the RunPod orchestrator. Each workflow is optimized for 24GB VRAM and includes API compatibility, error handling, and quality gates.

## Directory Structure

```
workflows/
├── text-to-image/       # Text-to-image generation workflows
├── image-to-image/      # Image-to-image transformation workflows
├── image-to-video/      # Image-to-video animation workflows
├── text-to-music/       # Text-to-music generation workflows
├── upscaling/           # Image upscaling and enhancement workflows
├── advanced/            # Advanced multi-model workflows
├── templates/           # Reusable workflow templates
├── README.md            # This file
└── WORKFLOW_STANDARDS.md # Workflow development standards
```

## Workflows by Category

### Text-to-Image (4 workflows)

| Workflow | Model | Speed | Quality | Use Case |
|----------|-------|-------|---------|----------|
| `flux-schnell-t2i-production-v1.json` | FLUX.1-schnell | Fast (4 steps) | Good | Rapid prototyping, iteration |
| `flux-dev-t2i-production-v1.json` | FLUX.1-dev | Medium (20-50 steps) | Excellent | High-quality final images |
| `sdxl-refiner-t2i-production-v1.json` | SDXL + Refiner | Medium (30+20 steps) | Excellent | Detailed, refined outputs |
| `sd35-large-t2i-production-v1.json` | SD3.5-large | Medium (28 steps) | Excellent | Latest Stable Diffusion |

### Image-to-Image (3 workflows)

| Workflow | Technique | Use Case |
|----------|-----------|----------|
| `ipadapter-style-i2i-production-v1.json` | IP-Adapter | Style transfer, composition |
| `ipadapter-face-i2i-production-v1.json` | IP-Adapter + Face | Portrait generation, face swap |
| `ipadapter-composition-i2i-production-v1.json` | IP-Adapter Multi | Complex scene composition |

### Image-to-Video (3 workflows)

| Workflow | Model | Length | Use Case |
|----------|-------|--------|----------|
| `cogvideox-i2v-production-v1.json` | CogVideoX-5b | 6s @ 8fps | AI-driven video generation |
| `svd-i2v-production-v1.json` | SVD | 14 frames | Quick animations |
| `svd-xt-i2v-production-v1.json` | SVD-XT | 25 frames | Extended animations |

### Text-to-Music (4 workflows)

| Workflow | Model | Duration | Use Case |
|----------|-------|----------|----------|
| `musicgen-small-t2m-production-v1.json` | MusicGen-small | 30s | Fast generation, low VRAM |
| `musicgen-medium-t2m-production-v1.json` | MusicGen-medium | 30s | Balanced quality/speed |
| `musicgen-large-t2m-production-v1.json` | MusicGen-large | 30s | Highest quality |
| `musicgen-melody-t2m-production-v1.json` | MusicGen-melody | 30s | Melody conditioning |

### Upscaling (3 workflows)

| Workflow | Technique | Scale | Use Case |
|----------|-----------|-------|----------|
| `ultimate-sd-upscale-production-v1.json` | Ultimate SD | 2x-4x | Professional upscaling with detailing |
| `simple-upscale-production-v1.json` | Model-based | 2x-4x | Fast, straightforward upscaling |
| `face-upscale-production-v1.json` | Face-focused | 2x | Portrait enhancement |

### Advanced (3 workflows)

| Workflow | Technique | Use Case |
|----------|-----------|----------|
| `controlnet-fusion-production-v1.json` | Multi-ControlNet | Precise composition control |
| `animatediff-video-production-v1.json` | AnimateDiff | Text-to-video animation |
| `batch-pipeline-production-v1.json` | Batch processing | Multiple variations |

## Quick Start

### Using with ComfyUI Web Interface

1. Open ComfyUI at `http://localhost:8188`
2. Click "Load" button
3. Navigate to `/workspace/ai/models/comfyui/workflows/`
4. Select desired workflow category and file
5. Adjust parameters as needed
6. Click "Queue Prompt"

### Using with RunPod Orchestrator API

```bash
# Example: FLUX Schnell text-to-image
curl -X POST http://localhost:9000/api/comfyui/generate \
  -H "Content-Type: application/json" \
  -d '{
    "workflow": "text-to-image/flux-schnell-t2i-production-v1.json",
    "inputs": {
      "prompt": "A serene mountain landscape at sunset",
      "seed": 42,
      "steps": 4
    }
  }'

# Example: Image upscaling
curl -X POST http://localhost:9000/api/comfyui/generate \
  -H "Content-Type: application/json" \
  -d '{
    "workflow": "upscaling/ultimate-sd-upscale-production-v1.json",
    "inputs": {
      "image": "path/to/image.png",
      "scale": 2
    }
  }'
```

## Workflow Features

All production workflows include:

- **API Compatibility**: Input/output nodes for orchestrator integration
- **Error Handling**: Validation, fallback nodes, graceful degradation
- **Quality Gates**: Preview nodes, checkpoints, validation steps
- **VRAM Optimization**: Model unloading, efficient memory management
- **Documentation**: Embedded descriptions, parameter guides
- **Versioning**: Semantic versioning in filenames

## Model Requirements

### Required Models (Essential)

These models are required by most workflows and are auto-downloaded by Ansible:

- **FLUX.1-schnell**: Fast text-to-image (17GB)
- **FLUX.1-dev**: High-quality text-to-image (23GB)
- **SDXL Base + Refiner**: Stable Diffusion XL (13GB)
- **SD3.5-large**: Latest Stable Diffusion (16GB)
- **CLIP ViT-L/14**: Image-text understanding (1.7GB)

### Optional Models

- **CogVideoX-5b**: Text-to-video, image-to-video (9.7GB)
- **SVD/SVD-XT**: Image-to-video (10GB)
- **MusicGen variants**: Text-to-music (1.5-3.4GB)
- **IP-Adapter**: Image conditioning (varies)
- **ControlNet models**: Precise control (varies)

Check `/workspace/ai/COMFYUI_MODELS.md` for complete model list.

## VRAM Considerations

All workflows are designed for **24GB VRAM** with these optimizations:

- **Sequential Loading**: Only one heavy model loaded at a time
- **Model Unloading**: Explicit cleanup between stages
- **Attention Slicing**: Enabled for large models
- **VAE Tiling**: For high-resolution processing
- **Batch Size Limits**: Capped at VRAM-safe values

## Performance Tips

### For Speed
- Use FLUX Schnell (4 steps) or SDXL base (20 steps)
- Lower resolution: 512x512 or 768x768
- Disable refiners and upscalers
- Use `--lowvram` flag if needed

### For Quality
- Use FLUX Dev (50 steps) or SDXL + Refiner
- Higher resolution: 1024x1024 or higher
- Enable face enhancement (Impact-Pack)
- Use Ultimate SD Upscale for final output

### For VRAM Efficiency
- Enable model unloading between stages
- Use VAE tiling for >1024px images
- Process batches sequentially, not in parallel
- Monitor with `nvidia-smi` during generation

## Troubleshooting

### Out of Memory (OOM) Errors

```bash
# Check VRAM usage
nvidia-smi

# Solutions:
# 1. Lower resolution
# 2. Reduce batch size
# 3. Enable model unloading
# 4. Use tiled VAE
# 5. Restart ComfyUI to clear VRAM
supervisorctl restart comfyui
```

### Missing Models

```bash
# Check which models are linked
ls -lah /workspace/ComfyUI/models/diffusers/
ls -lah /workspace/ComfyUI/models/clip_vision/

# Re-run Ansible to download missing models
cd /workspace/ai
ansible-playbook playbook.yml --tags comfyui-models-all

# Re-link models
arty run models/link-comfyui
```

### Workflow Load Errors

```bash
# Check ComfyUI logs
supervisorctl tail -f comfyui

# Common issues:
# - Missing custom nodes: Check custom_nodes/ directory
# - Node version mismatch: Update ComfyUI and custom nodes
# - Corrupted workflow: Validate JSON syntax
```

## Development

### Creating New Workflows

See `WORKFLOW_STANDARDS.md` for detailed guidelines on creating production-ready workflows.

Quick checklist:
- [ ] Use semantic versioning in filename
- [ ] Add API input/output nodes
- [ ] Include preview and save nodes
- [ ] Add error handling and validation
- [ ] Optimize for 24GB VRAM
- [ ] Document all parameters
- [ ] Test with orchestrator API

### Testing Workflows

```bash
# Manual test via ComfyUI UI
# 1. Load workflow in ComfyUI
# 2. Set test parameters
# 3. Queue prompt
# 4. Verify output quality

# API test via orchestrator
curl -X POST http://localhost:9000/api/comfyui/generate \
  -H "Content-Type: application/json" \
  -d @test-payload.json

# Batch test multiple workflows
cd /workspace/ai/models/comfyui/workflows
for workflow in text-to-image/*.json; do
  echo "Testing $workflow..."
  # Add test logic here
done
```

## Contributing

When adding new workflows:

1. Follow naming convention: `{category}-{model}-{type}-production-v{version}.json`
2. Place in appropriate category directory
3. Update this README with workflow details
4. Add to `comfyui_models.yaml` if new models are required
5. Test with both UI and API
6. Document any special requirements or setup

## Resources

- **ComfyUI Documentation**: https://github.com/comfyanonymous/ComfyUI
- **Custom Nodes Manager**: Install via ComfyUI-Manager in UI
- **Model Registry**: `/workspace/ai/model-orchestrator/models.yaml`
- **Ansible Playbook**: `/workspace/ai/playbook.yml`
- **Orchestrator API**: http://localhost:9000/docs

## License

MIT License - Part of RunPod AI Model Orchestrator

## Support

For issues or questions:
1. Check ComfyUI logs: `supervisorctl tail -f comfyui`
2. Check orchestrator logs: `supervisorctl tail -f orchestrator`
3. Review `/workspace/ai/CLAUDE.md` for troubleshooting
4. Check GPU status: `nvidia-smi`