valknar/docker-compose
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: revert to openai prefix, remove /v1 suffix from api_base

Browse Source 
- Changed back from hosted_vllm/qwen-2.5-7b to openai/qwen-2.5-7b
- Removed /v1 suffix from api_base (LiteLLM adds it automatically)
- Added supports_system_messages: false for vLLM compatibility
This commit is contained in:
Sebastian Krüger
2025-11-21 17:55:10 +01:00
parent 699c8537b0
commit 42a68bc0b5
1 changed files with 3 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
ai/litellm-config.yaml
View File

@@ -32,13 +32,14 @@ model_list:
# Text Generation # Text Generation
- model_name: qwen-2.5-7b - model_name: qwen-2.5-7b
litellm_params: litellm_params:
model: hosted_vllm/qwen-2.5-7b model: openai/qwen-2.5-7b
api_base: http://100.121.199.88:9000/v1 # RunPod GPU via Tailscale api_base: http://100.121.199.88:9000 # RunPod GPU via Tailscale (no /v1 suffix)
api_key: dummy api_key: dummy
rpm: 1000 rpm: 1000
tpm: 100000 tpm: 100000
timeout: 600 # 10 minutes for generation timeout: 600 # 10 minutes for generation
stream_timeout: 600 stream_timeout: 600
supports_system_messages: false # vLLM handles system messages differently
# Image Generation # Image Generation
- model_name: flux-schnell - model_name: flux-schnell