Complete rewrite of both model download scripts with:
- Beautiful colorful CLI output with progress indicators
- Pure bash/curl downloads (no Python dependencies for downloading)
- yq-based YAML parsing (consistent with arty.sh)
- Three commands: download, link, verify
- Filtering by --category and --repo-id (comma-separated)
- --dry-run mode for previewing operations
- Respects format field for file extensions (.safetensors, .pt, etc.)
- Uses type field for output subdirectories (checkpoints, embeddings, loras)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
HuggingFace cache flattens repository directory structure into snapshots.
When YAML specifies "vae/model.safetensors", the cache stores it as just
"model.safetensors" without preserving the vae/ subdirectory.
Changed grep pattern from "/$source_pattern$" to "/$filename_only$" to
match the actual flattened cache structure instead of the logical repo path.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The source_pattern can include subdirectories (e.g., "unet/model.safetensors"),
but find_model_files() filters by filename only. Now we:
1. Extract just the filename using basename
2. Find files matching that filename
3. Filter results to match the full source_pattern path
This allows proper matching of files in subdirectories.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The previous fix still called find_model_files() with empty filter,
which doesn't work. Now we call find_model_files() for EACH file mapping
individually, passing the source_pattern as the filter.
This eliminates the "No files matched filter (filter: )" warnings and
allows the script to properly find and link downloaded models.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Previously, both functions called find_model_files() with empty filters before
checking for explicit file mappings, causing premature exit when filters were empty.
Now both functions check for file_mappings FIRST, then call find_model_files()
only after confirming mappings exist. This fixes critical linking failures for
models with explicit file mappings but no filename filters.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Updated link_model() and verify_model_links() functions to extract target
directories directly from dest paths instead of using separate model_type parameter.
Key changes:
- link_model() now parses dest paths (e.g., "checkpoints/model.safetensors") to
determine target directories dynamically
- verify_model_links() updated to work with dest paths containing directory info
- Removed model_type parameter from both function signatures and all callers
- Support for models with files in multiple ComfyUI subdirectories
- Updated display output to remove model_type from status messages
This simplifies the script by eliminating redundant type information, as the
directory structure is now embedded in the dest paths within the YAML file.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Removed the DEBUG line that printed snapshot paths during model linking.
This cleans up the output to only show user-relevant information.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Changed CROSS to CROSS_MARK to match the variable defined in the
Unicode characters section (line 91).
This fixes the "CROSS: unbound variable" error when running the
cleanup function in dry-run mode.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fix the bash parameter expansion to properly replace all forward slashes
with double hyphens when converting HuggingFace repo IDs to cache paths.
Changed from: ${repo_id//\/--} (incorrect syntax)
Changed to: ${repo_id//\//--} (correct syntax)
This fixes the 'Cache directory not found' error when using --cleanup flag.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fix bash string substitution to replace all forward slashes with
double hyphens when constructing HuggingFace cache paths.
Changed from: ${repo_id/\//-} (replaces first / with single -)
Changed to: ${repo_id//\/--} (replaces all / with --)
This fixes the "Cache directory not found" warning when using the
--cleanup flag with repositories that have slashes in their names
(e.g., black-forest-labs/FLUX.1-schnell).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add three major features to artifact_huggingface_download.sh:
1. Category filtering (--category flag):
- Filter models by category (single or comma-separated multiple)
- Validates categories against YAML configuration
- Works with all commands (download, link, both, verify)
2. Cleanup mode (--cleanup flag):
- Removes unreferenced files from HuggingFace cache
- Only deletes files not referenced by symlinks
- Per-category cleanup (safe and isolated)
- Works with link and both commands only
3. Dry-run mode (--dry-run/-n flag):
- Preview operations without making changes
- Shows what would be downloaded, linked, or cleaned
- Includes file counts and size estimates
- Warning banner to indicate dry-run mode
Additional improvements:
- Updated help text with comprehensive examples
- Added validation for invalid flag combinations
- Enhanced user feedback with detailed preview information
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Updated find_model_files() to include .index.json files in the file
discovery logic. These files are essential for sharded models like
CogVideoX which split safetensors into multiple parts.
Changes:
- Modified JSON file filtering to allow files ending with .index.json
- Updated comment to document support for sharded model index files
Fixes automatic linking for:
- CogVideoX-5b (3 files including index.json)
- CogVideoX-5b-I2V (4 files including index.json)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
HuggingFace models use symlinks pointing to blobs in the cache. The stat
command was returning the size of the symlink itself (76 bytes) instead of
the actual file it points to.
Fixed by:
- Adding -L flag to stat commands to follow symlinks
- Testing which stat format works (Linux -c vs macOS -f) before executing
- Using proper spacing and quotes in stat commands
- Checking for both regular files and symlinks in the condition
This fixes disk usage showing as 1 KB instead of the actual ~50+ GB for models
like FLUX.
The stat command was producing multi-line output which broke the pipe-delimited
return format of verify_model_download(). Fixed by:
- Testing which stat format works (Linux vs macOS) before executing
- Using proper format flags to get single-line date output
- Defaulting to 'Unknown' instead of relying on fallback chain
This fixes the issue where all models showed as 'NOT DOWNLOADED' even when
present in the cache.
The verify command was showing all models as "NOT DOWNLOADED" because find_model_files()
was exiting silently without diagnostic output. This made debugging impossible.
Changes:
- Added detailed error messages to Python script in find_model_files()
- Reports which directories were checked and why they failed
- Shows actual vs expected paths for model/snapshots directories
- Includes DEBUG messages showing which snapshot is being used
- Warns when no files match the filter
- Modified verify_model_download() to capture and display stderr
- Changed from suppressing stderr (2>/dev/null) to capturing it (2>&1)
- Filters ERROR/WARN/DEBUG prefixes from file paths
- Logs diagnostic messages to stderr for visibility
This will help identify the actual cache structure mismatch causing verification failures.
🤖 Generated with Claude Code (https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Added new 'verify' command to artifact_huggingface_download.sh that performs comprehensive health checks on all downloaded models and their symlinks.
Features:
- Download status verification (existence, size, location, timestamps)
- Link status verification (valid, broken, or missing symlinks)
- Size mismatch detection (warns if actual differs >10% from expected)
- Per-model detailed logging with beautiful formatting
- Category-level and global statistics summaries
- Actionable fix suggestions for detected issues
- Disk space usage analysis
New Functions:
- get_model_disk_usage() - Calculate actual model file sizes
- format_bytes() - Human-readable size formatting
- verify_model_download() - Check model download status
- verify_model_links() - Verify symlink integrity
- verify_category() - Process category with verification
- display_verification_summary() - Show global results
Usage:
artifact_huggingface_download.sh verify -c models.yaml
Output includes:
✓ Downloaded/Missing model counts
✓ Properly linked/Broken link statistics
✓ File sizes and locations
✓ Last modified timestamps
⚠️ Size mismatch warnings
📊 Disk space usage per category
💡 Fix suggestions with exact commands
🤖 Generated with Claude Code (https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The build_reference_tree() function was prepending config_dir to all
'into' paths without expanding environment variables or checking if the
result was absolute. This caused paths like /workspace/ai//workspace/ComfyUI
instead of /workspace/ComfyUI.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fixed issue where absolute paths from environment variables (like $COMFYUI_ROOT=/workspace/ComfyUI) were being treated as relative paths and prepended with the config directory.
Now checks if the expanded path is absolute (starts with /) and uses it as-is, otherwise treats it as relative to the config file directory.
Before: /home/valknar/Projects/runpod//workspace/ComfyUI
After: /workspace/ComfyUI
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Adds support for environment variables like $COMFYUI_ROOT in the 'into' field of references when running 'arty deps'. Features:
- New expand_env_vars() function with nested variable expansion support
- Validates that no undefined variables remain after expansion
- Fails with clear error messages showing undefined variables
- Lists available environment variables from arty.yml in error output
- Expands variables from envs section before cloning repositories
Example usage in arty.yml:
```yaml
envs:
default:
COMFYUI_ROOT: /workspace/ComfyUI
references:
- url: https://github.com/ltdrdata/ComfyUI-Manager.git
into: $COMFYUI_ROOT/custom_nodes/ComfyUI-Manager
```
Expands to: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add artifact_civitai_download.sh with beautiful purple/magenta CLI
- Rename artifact_comfyui_download.sh to artifact_hugginface_download.sh
- Remove comfyui_models.example.yaml (moved to runpod repo)
Features:
- Dedicated downloader for CivitAI models
- Beautiful CLI with progress bars and retry logic
- Same architecture as HuggingFace downloader
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Automatically detects RunPod environment (checks for /workspace directory)
- Uses RunPod paths by default: /workspace/huggingface_cache and /workspace/ComfyUI/models
- Falls back to local paths (~/.cache/huggingface and ~/ComfyUI/models) on non-RunPod systems
- Updated help text to show both RunPod and local defaults
- Updated example YAML with RunPod paths
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Removed automatic checkpoint linking logic that was incorrectly
duplicating diffusers models to checkpoints directory. Models should
only be linked to their explicitly configured directory type.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Added parse_file_mappings() function to extract file mappings from YAML
- Modified link_model() to accept and use explicit source→dest mappings
- Falls back to automatic prefixing if no mappings specified
- Updated process_category() to parse and pass file mappings
- This allows workflows to reference models by their expected names
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Extract model name from repo_id (e.g., 'musicgen-medium' from 'facebook/musicgen-medium')
and prefix all linked files with it for better organization and clarity.
Examples:
- pytorch_model.bin -> musicgen-medium-pytorch_model.bin
- model.safetensors -> musicgen-small-model.safetensors
HuggingFace cache stores files as symlinks to blobs, so stat without -L
was returning the symlink size (76 bytes) instead of actual file size (6.5GB).
This caused the >500MB check to fail and skip linking checkpoint files.
- Apply same sys.argv pattern to find_model_files function
- Ensures all Python heredocs avoid bash interpolation issues
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Initialize HF_TOKEN="${HF_TOKEN:-}" before conditional checks
- Fixes "unbound variable" error with set -euo pipefail
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Try PROJECT_ROOT/ai/.env first
- Fallback to PROJECT_ROOT/.env
- Fallback to /workspace/ai/.env
- Fixes HF_TOKEN loading on RunPod
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Check for models in cache_dir/models--xxx--yyy/ first
- Fallback to cache_dir/hub/models--xxx--yyy/ for compatibility
- Fixes issue where script couldn't find downloaded models
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Adds 'arty notes' command to display project notes from arty.yml with beautiful markdown rendering. Features include:
- Reads 'notes' field from arty.yml
- Supports full markdown syntax (headers, bold, italic, code blocks, links, lists)
- Smart rendering fallbacks: glow → mdcat → python3 → plain text
- Beautiful terminal output with colors and formatting
- ANSI escape codes for syntax highlighting
- Horizontal separators for code blocks
Also renamed arty.sh to artifact_git_download.sh for better clarity.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Adds subcommand structure (download, link, both) to the ComfyUI downloader script, allowing users to download models from HuggingFace and/or create symlinks to ComfyUI model directories. Features include:
- New subcommands: download (download only), link (symlink only), both (default)
- ComfyUI directory configuration (--comfyui-dir, default: ~/ComfyUI/models)
- Smart symlink creation to appropriate ComfyUI subdirectories (checkpoints, vae, loras, controlnet, upscale_models, etc.)
- YAML configuration extended with 'type' and 'filename' fields for precise model organization
- Automatic ComfyUI subdirectory creation
- Graceful handling of existing symlinks and files
- HF_TOKEN validation only when needed (download/both commands)
- Example configuration file (comfyui_models.example.yaml) demonstrating proper setup
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Adds artifact_comfyui_download.sh for downloading AI models from HuggingFace with configuration-driven approach, beautiful CLI output, and flexible options. Features include optional YAML configuration, customizable cache directory (defaults to ~/.cache/huggingface), HF_TOKEN support, and robust error handling.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Fix race condition in load_env_vars() caused by process substitution
- Replace unreliable `< <(yq eval ...)` with stable variable capture
- Ensure environment variables are consistently loaded on every execution
- Add yq binary (mikefarah/yq v4.48.1) for reliable YAML parsing
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
