fix: disable drop_params to preserve streaming metadata in LiteLLM
- Set drop_params: false in litellm_settings - Set modify_params: false in litellm_settings - Set drop_params: false in default_litellm_params - Commented out LITELLM_DROP_PARAMS env var - Removed --drop_params command flag These settings were stripping critical streaming parameters causing vLLM streaming responses to collapse into empty deltas
This commit is contained in:
@@ -100,7 +100,7 @@ services:
|
||||
ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
|
||||
LITELLM_MASTER_KEY: ${AI_LITELLM_API_KEY}
|
||||
DATABASE_URL: postgresql://${AI_DB_USER}:${AI_DB_PASSWORD}@ai_postgres:5432/litellm
|
||||
LITELLM_DROP_PARAMS: 'true'
|
||||
# LITELLM_DROP_PARAMS: 'true' # DISABLED: Was breaking streaming
|
||||
NO_DOCS: 'true'
|
||||
NO_REDOC: 'true'
|
||||
# Performance optimizations
|
||||
@@ -115,8 +115,7 @@ services:
|
||||
'--host',
|
||||
'0.0.0.0',
|
||||
'--port',
|
||||
'4000',
|
||||
'--drop_params'
|
||||
'4000'
|
||||
]
|
||||
depends_on:
|
||||
- ai_postgres
|
||||
|
||||
Reference in New Issue
Block a user