Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.80.10.rc.1

pip install litellm==1.80.10

Key Highlights​

• Agent (A2A) Gateway with Cost Tracking - Track agent costs per query, per token pricing, and view agent usage in the dashboard
• 2 New Agent Providers - LangGraph Agents and Azure AI Foundry Agents for agentic workflows
• New Provider: SAP Gen AI Hub - Full support for SAP Generative AI Hub with chat completions
• New Bedrock Writer Models - Add Palmyra-X4 and Palmyra-X5 models on Bedrock
• OpenAI GPT-5.2 Models - Full support for GPT-5.2, GPT-5.2-pro, and Azure GPT-5.2 models with reasoning support
• 227 New Fireworks AI Models - Comprehensive model coverage for Fireworks AI platform
• MCP Support on /chat/completions - Use MCP servers directly via chat completions endpoint
• Performance Improvements - Reduced memory leaks by 50%

Agent Gateway - 4 New Agent Providers​

This release adds support for agents from the following providers:

• LangGraph Agents - Deploy and manage LangGraph-based agents
• Azure AI Foundry Agents - Enterprise agent deployments on Azure
• Bedrock AgentCore - AWS Bedrock agent integration
• A2A Agents - Agent-to-Agent protocol support

AI Gateway admins can now add agents from any of these providers, and developers can invoke them through a unified interface using the A2A protocol.

For all agent requests running through the AI Gateway, LiteLLM automatically tracks request/response logs, cost, and token usage.

Agent (A2A) Usage UI​

Users can now filter usage statistics by agents, providing the same granular filtering capabilities available for teams, organizations, and customers.

Details:

• Filter usage analytics, spend logs, and activity metrics by agent ID
• View breakdowns on a per-agent basis
• Consistent filtering experience across all usage and analytics views

New Providers and Endpoints​

New Providers (5 new providers)​

New LLM API Endpoints (4 new endpoints)​

New Models / Updated Models​

New Model Support (270+ new models)​

Features​

• OpenAI
  • Add support for OpenAI GPT-5.2 models with reasoning_effort='xhigh' - PR #17836, PR #17875
  • Include 'user' param for responses API models - PR #17648
  • Use optimized async http client for text completions - PR #17831
• Azure
  • Add Azure GPT-5.2 models support - PR #17866
• Azure AI
  • Fix Azure AI Anthropic api-key header and passthrough cost calculation - PR #17656
  • Remove unsupported params from Azure AI Anthropic requests - PR #17822
• Anthropic
  • Prevent duplicate tool_result blocks with same tool - PR #17632
  • Handle partial JSON chunks in streaming responses - PR #17493
  • Preserve server_tool_use and web_search_tool_result in multi-turn conversations - PR #17746
  • Capture web_search_tool_result in streaming for multi-turn conversations - PR #17798
  • Add retrieve batches and retrieve file content support - PR #17700
• Bedrock
  • Add new Bedrock OSS models to model list - PR #17638
  • Add Bedrock Writer models (Palmyra-X4, Palmyra-X5) - PR #17685
  • Add EU Claude Opus 4.5 model - PR #17897
  • Add serviceTier support for Converse API - PR #17810
  • Fix header forwarding with custom API for Bedrock embeddings - PR #17872
• Gemini
  • Add support for computer use for Gemini - PR #17756
  • Handle context window errors - PR #17751
  • Add speechConfig to GenerationConfig for Gemini TTS - PR #17851
• Vertex AI
  • Add DeepSeek-V3.2 model support - PR #17770
  • Preserve systemInstructions for generate content request - PR #17803
• Mistral
  • Add Codestral 2508, Devstral 2512 models - PR #17801
• Cerebras
  • Add zai-glm-4.6 model support - PR #17683
  • Fix context window errors not recognized - PR #17587
• DeepSeek
  • Add native support for thinking and reasoning_effort params - PR #17712
• NVIDIA NIM Rerank
  • Add llama-3.2-nv-rerankqa-1b-v2 rerank model - PR #17670
• Fireworks AI
  • Add 227 new Fireworks AI models - PR #17692
• Dashscope
  • Fix default base_url error - PR #17584

Bug Fixes​

• Anthropic
  • Fix missing content in Anthropic to OpenAI conversion - PR #17693
  • Avoid error when we have just the tool_calls in input - PR #17753
• Azure
  • Fix error about encoding video id for Azure - PR #17708
• Azure AI
  • Fix LLM provider for azure_ai in model map - PR #17805
• Watsonx
  • Fix Watsonx Audio Transcription to only send supported params to API - PR #17840
• Router
  • Handle tools=None in completion requests - PR #17684
  • Add minimum request threshold for error rate cooldown - PR #17464

LLM API Endpoints​

Features​

• Responses API
  • Add usage details in responses usage object - PR #17641
  • Fix error for response API polling - PR #17654
  • Fix streaming tool_calls being dropped when text + tool_calls - PR #17652
  • Transform image content in tool results for Responses API - PR #17799
  • Fix responses api not applying tpm rate limits on api keys - PR #17707
• Containers API
  • Allow using LIST, Create Containers using custom-llm-provider - PR #17740
  • Add new container API file management + UI Interface - PR #17745
• Rerank API
  • Add support for forwarding client headers in /rerank endpoint - PR #17873
• Files API
  • Add support for expires_after param in Files endpoint - PR #17860
• Video API
  • Use litellm params for all videos APIs - PR #17732
  • Respect videos content db creds - PR #17771
• Embeddings API
  • Fix handling token array input decoding for embeddings - PR #17468
• Chat Completions API
  • Add v0 target storage support - store files in Azure AI storage and use with chat completions API - PR #17758
• generateContent API
  • Support model names with slashes on Gemini generateContent endpoints - PR #17743
• General
  • Use audio content for caching - PR #17651
  • Return 403 exception when calling GET responses API - PR #17629
  • Add nested field removal support to additional_drop_params - PR #17711
  • Async post_call_streaming_iterator_hook now properly iterates async generators - PR #17626

Bugs​

• General
  • Fix handle string content in is_cached_message - PR #17853

Management Endpoints / UI​

Features​

• UI Settings
  • Add Get and Update Backend Routes for UI Settings - PR #17689
  • UI Settings page implementation - PR #17697
  • Ensure Model Page honors UI Settings - PR #17804
  • Add All Proxy Models to Default User Settings - PR #17902
• Agent & Usage UI
  • Daily Agent Usage Backend - PR #17781
  • Agent Usage UI - PR #17797
  • Add agent cost tracking on UI - PR #17899
  • New Badge for Agent Usage - PR #17883
  • Usage Entity labels for filtering - PR #17896
  • Agent Usage Page minor fixes - PR #17901
  • Usage Page View Select component - PR #17854
  • Usage Page Components refactor - PR #17848
• Logs & Spend
  • Enhanced spend analytics in logs view - PR #17623
  • Add user info delete modal for user management - PR #17625
  • Show request and response details in logs view - PR #17928
• Virtual Keys
  • Fix x-litellm-key-spend header update - PR #17864
• Models & Endpoints
  • Model Hub Useful Links Rearrange - PR #17859
  • Create Team Model Dropdown honors Organization's Models - PR #17834
• SSO & Auth
  • Allow upserting user role when SSO provider role changes - PR #17754
  • Allow fetching role from generic SSO provider (Keycloak) - PR #17787
  • JWT Auth - allow selecting team_id from request header - PR #17884
  • Remove SSO Config Values from Config Table on SSO Update - PR #17668
• Teams
  • Attach team to org table - PR #17832
  • Expose the team alias when authenticating - PR #17725
• MCP Server Management
  • Add extra_headers and allowed_tools to UpdateMCPServerRequest - PR #17940
• Notifications
  • Show progress and pause on hover for Notifications - PR #17942
• General
  • Allow Root Path to Redirect when Docs not on Root Path - PR #16843
  • Show UI version number on top left near logo - PR #17891
  • Re-organize left navigation with correct categories and agents on root - PR #17890
  • UI Playground - allow custom model names in model selector dropdown - PR #17892

Bugs​

• UI Fixes
  • Fix links + old login page deprecation message - PR #17624
  • Filtering for Chat UI Endpoint Selector - PR #17567
  • Race Condition Handling in SCIM v2 - PR #17513
  • Make /litellm_model_cost_map public - PR #16795
  • Custom Callback on UI - PR #17522
  • Add User Writable Directory to Non Root Docker for Logo - PR #17180
  • Swap URL Input and Display Name inputs - PR #17682
  • Change deprecation banner to only show on /sso/key/generate - PR #17681
  • Change credential encryption to only affect db credentials - PR #17741
• Auth & Routes
  • Return 403 instead of 503 for unauthorized routes - PR #17723
  • AI Gateway Auth - allow using wildcard patterns for public routes - PR #17686

AI Integrations​

New Integrations (4 new integrations)​

Logging​

• Langfuse
  • Propagate Langfuse trace_id - PR #17669
  • Prefer standard trace id for Langfuse logging - PR #17791
  • Move query params to create_pass_through_route call in Langfuse passthrough - PR #17660
  • Add support for custom masking function - PR #17826
• Prometheus
  • Add 'exception_status' to prometheus logger - PR #17847
• OpenTelemetry
  • Add latency metrics (TTFT, TPOT, Total Generation Time) to OTEL payload - PR #17888
• General
  • Add polling via cache feature for async logging - PR #16862

Guardrails​

• HiddenLayer
  • Add HiddenLayer Guardrail Hooks - PR #17728
• Pillar Security
  • Add opt-in evidence results for Pillar Security guardrail during monitoring - PR #17812
• PANW Prisma AIRS
  • Add configurable fail-open, timeout, and app_user tracking - PR #17785
• Presidio
  • Add support for configurable confidence score thresholds and scope in Presidio PII masking - PR #17817
• LiteLLM Content Filter
  • Mask all regex pattern matches, not just first - PR #17727
• Regex Guardrails
  • Add enhanced regex pattern matching for guardrails - PR #17915
• Gray Swan Guardrail
  • Add passthrough mode for model response - PR #17102

Prompt Management​

• General
  • New API for integrating prompt management providers - PR #17829

Spend Tracking, Budgets and Rate Limiting​

• Service Tier Pricing - Extract service_tier from response/usage for OpenAI flex pricing - PR #17748
• Agent Cost Tracking - Track agent_id in SpendLogs - PR #17795
• Tag Activity - Deduplicate /tag/daily/activity metadata - PR #16764
• Rate Limiting - Dynamic Rate Limiter - allow specifying ttl for in memory cache - PR #17679

MCP Gateway​

• Chat Completions Integration - Add support for using MCPs on /chat/completions - PR #17747
• UI Session Permissions - Fix UI session MCP permissions across real teams - PR #17620
• OAuth Callback - Fix MCP OAuth callback routing and URL handling - PR #17789
• Tool Name Prefix - Fix MCP tool name prefix - PR #17908

Agent Gateway (A2A)​

• Cost Per Query - Add cost per query for agent invocations - PR #17774
• Token Counting - Add token counting non streaming + streaming - PR #17779
• Cost Per Token - Add cost per token pricing for A2A - PR #17780
• LangGraph Provider - Add LangGraph provider for Agent Gateway - PR #17783
• Bedrock & LangGraph Agents - Allow using Bedrock AgentCore, LangGraph agents with A2A Gateway - PR #17786
• Agent Management - Allow adding LangGraph, Bedrock Agent Core agents - PR #17802
• Azure Foundry Agents - Add Azure AI Foundry Agents support - PR #17845
• Azure Foundry UI - Allow adding Azure Foundry Agents on UI - PR #17909
• Azure Foundry Fixes - Ensure Azure Foundry agents work correctly - PR #17943

Performance / Loadbalancing / Reliability improvements​

• Memory Leak Fix - Cut memory leak in half - PR #17784
• Spend Logs Memory - Reduce memory accumulation of spend_logs - PR #17742
• Router Optimization - Replace time.perf_counter() with time.time() - PR #17881
• Filter Internal Params - Filter internal params in fallback code - PR #17941
• Gunicorn Suggestion - Suggest Gunicorn instead of uvicorn when using max_requests_before_restart - PR #17788
• Pydantic Warnings - Mitigate PydanticDeprecatedSince20 warnings - PR #17657
• Python 3.14 Support - Add Python 3.14 support via grpcio version constraints - PR #17666
• OpenAI Package - Bump openai package to 2.9.0 - PR #17818

Documentation Updates​

• Contributing - Update clone instructions to recommend forking first - PR #17637
• Getting Started - Improve Getting Started page and SDK documentation structure - PR #17614
• JSON Mode - Make it clearer how to get Pydantic model output - PR #17671
• drop_params - Update litellm docs for drop_params - PR #17658
• Environment Variables - Document missing environment variables and fix incorrect types - PR #17649
• SumoLogic - Add SumoLogic integration documentation - PR #17647
• SAP Gen AI - Add SAP Gen AI provider documentation - PR #17667
• Authentication - Add Note for Authentication - PR #17733
• Known Issues - Adding known issues to 1.80.5-stable docs - PR #17738
• Supported Endpoints - Fix Supported Endpoints page - PR #17710
• Token Count - Document token count endpoint - PR #17772
• Overview - Made litellm proxy and SDK difference cleaner in overview with a table - PR #17790
• Containers API - Add docs for containers files API + code interpreter on LiteLLM - PR #17749
• Target Storage - Add documentation for target storage - PR #17882
• Agent Usage - Agent Usage documentation - PR #17931, PR #17932, PR #17934
• Cursor Integration - Cursor Integration documentation - PR #17855, PR #17939
• A2A Cost Tracking - A2A cost tracking docs - PR #17913
• Azure Search - Update azure search docs - PR #17726
• Milvus Client - Fix milvus client docs - PR #17736
• Streaming Logging - Remove streaming logging doc - PR #17739
• Integration Docs - Update integration docs location - PR #17644
• Links - Updated docs links for mistral and anthropic - PR #17852
• Community - Add community doc link - PR #17734
• Pricing - Update pricing for global.anthropic.claude-haiku-4-5-20251001-v1:0 - PR #17703
• gpt-image-1-mini - Correct model type for gpt-image-1-mini - PR #17635

Infrastructure / Deployment​

• Docker - Use python instead of wget for healthcheck in docker-compose.yml - PR #17646
• Helm Chart - Add extraResources support for Helm chart deployments - PR #17627
• Helm Versioning - Add semver prerelease suffix to helm chart versions - PR #17678
• Database Schema - Add storage_backend and storage_url columns to schema.prisma for target storage feature - PR #17936

New Contributors​

• @xianzongxie-stripe made their first contribution in PR #16862
• @krisxia0506 made their first contribution in PR #17637
• @chetanchoudhary-sumo made their first contribution in PR #17630
• @kevinmarx made their first contribution in PR #17632
• @expruc made their first contribution in PR #17627
• @rcII made their first contribution in PR #17626
• @tamirkiviti13 made their first contribution in PR #16591
• @Eric84626 made their first contribution in PR #17629
• @vasilisazayka made their first contribution in PR #16053
• @juliettech13 made their first contribution in PR #17663
• @jason-nance made their first contribution in PR #17660
• @yisding made their first contribution in PR #17671
• @emilsvennesson made their first contribution in PR #17656
• @kumekay made their first contribution in PR #17646
• @chenzhaofei01 made their first contribution in PR #17584
• @shivamrawat1 made their first contribution in PR #17733
• @ephrimstanley made their first contribution in PR #17723
• @hwittenborn made their first contribution in PR #17743
• @peterkc made their first contribution in PR #17727
• @saisurya237 made their first contribution in PR #17725
• @Ashton-Sidhu made their first contribution in PR #17728
• @CyrusTC made their first contribution in PR #17810
• @jichmi made their first contribution in PR #17703
• @ryan-crabbe made their first contribution in PR #17852
• @nlineback made their first contribution in PR #17851
• @butnarurazvan made their first contribution in PR #17468
• @yoshi-p27 made their first contribution in PR #17915

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.80.8-stable

pip install litellm==1.80.8

Key Highlights​

• Agent Gateway (A2A) - Invoke agents through the AI Gateway with request/response logging and access controls
• Guardrails API v2 - Generic Guardrail API with streaming support, structured messages, and tool call checks
• Customer (End User) Usage UI - Track and visualize end-user spend directly in the dashboard
• vLLM Batch + Files API - Support for batch and files API with vLLM deployments
• Dynamic Rate Limiting on Teams - Enable dynamic rate limits and priority reservation on team-level
• Google Cloud Chirp3 HD - New text-to-speech provider with Chirp3 HD voices

Agent Gateway (A2A)​

This release introduces A2A Agent Gateway for LiteLLM, allowing you to invoke and manage A2A agents with the same controls you have for LLM APIs.

As a LiteLLM Gateway Admin, you can now do the following:

• Request/Response Logging - Every agent invocation is logged to the Logs page with full request and response tracking.
• Access Control - Control which Team/Key can access which agents.

As a developer, you can continue using the A2A SDK, all you need to do is point you A2AClient to the LiteLLM proxy URL and your API key.

Works with the A2A SDK:

from a2a.client import A2AClient

client = A2AClient(
    base_url="http://localhost:4000",  # Your LiteLLM proxy
    api_key="sk-1234"                   # LiteLLM API key
)

response = client.send_message(
    agent_id="my-agent",
    message="What's the status of my order?"
)

Get started with Agent Gateway here: Agent Gateway Documentation

Customer (End User) Usage UI​

Users can now filter usage statistics by customers, providing the same granular filtering capabilities available for teams and organizations.

Details:

• Filter usage analytics, spend logs, and activity metrics by customer ID
• View customer-level breakdowns alongside existing team and user-level filters
• Consistent filtering experience across all usage and analytics views

New Providers and Endpoints​

New Providers (5 new providers)​

New LLM API Endpoints (2 new endpoints)​

New Models / Updated Models​

New Model Support (33 new models)​

Features​

• OpenAI

  • Add gpt-5.1-codex-max model pricing and configuration - PR #17541
  • Add xhigh reasoning effort for gpt-5.1-codex-max - PR #17585
  • Add clear error message for empty LLM endpoint responses - PR #17445

• Azure OpenAI

  • Allow reasoning_effort='none' for Azure gpt-5.1 models - PR #17311

• Anthropic

  • Add claude-opus-4-5 alias to pricing data - PR #17313
  • Parse <budget:thinking> blocks for opus 4.5 - PR #17534
  • Update new Anthropic features as reviewed - PR #17142
  • Skip empty text blocks in Anthropic system messages - PR #17442

• Bedrock

  • Add Nova embedding support - PR #17253
  • Add support for Bedrock Qwen 2 imported model - PR #17461
  • Bedrock OpenAI model support - PR #17368
  • Add support for file content download for Bedrock batches - PR #17470
  • Make streaming chunk size configurable in Bedrock API - PR #17357
  • Add experimental latest-user filtering for Bedrock - PR #17282
  • Handle Cohere v4 embed response dictionary format - PR #17220
  • Remove not compatible beta header from Bedrock - PR #17301
  • Add model price and details for Global Opus 4.5 Bedrock endpoint - PR #17380

• Gemini (Google AI Studio + Vertex AI)

  • Add better handling in image generation for Gemini models - PR #17292
  • Fix reasoning_content showing duplicate content in streaming responses - PR #17266
  • Handle partial JSON chunks after first valid chunk - PR #17496
  • Fix Gemini 3 last chunk thinking block - PR #17403
  • Fix Gemini image_tokens treated as text tokens in cost calculation - PR #17554
  • Make sure that media resolution is only for Gemini 3 model - PR #17137

• Vertex AI

  • Add Google Cloud Chirp3 HD support on /speech - PR #17391
  • Add BGE Embeddings support - PR #17362
  • Handle global location for Vertex AI image generation endpoint - PR #17255
  • Add Google Private API Endpoint to Vertex AI fields - PR #17382

• Z.AI (Zhipu AI)

  • Add Z.AI as built-in provider - PR #17307

• GitHub Copilot

  • Add Embedding API support - PR #17278
  • Preserve encrypted_content in reasoning items for multi-turn conversations - PR #17130

• Databricks

  • Update Databricks model pricing and add new models - PR #17277

• OVHcloud

  • Add support of audio transcription for OVHcloud - PR #17305

• Mistral

  • Add Mistral Large 3 model support - PR #17547

• Moonshot

  • Fix missing Moonshot turbo models and fix incorrect pricing - PR #17432

• Together AI

  • Add context window exception mapping for Together AI - PR #17284

• WatsonX

  • Allow passing zen_api_key dynamically - PR #16655
  • Fix Watsonx Audio Transcription API - PR #17326
  • Fix audio transcriptions, don't force content type in request headers - PR #17546

• Fireworks AI

  • Add new model fireworks_ai/kimi-k2-instruct-0905 - PR #17328
  • Add fireworks/deepseek-v3p2 - PR #17395

• DeepSeek

  • Support Deepseek 3.2 with Reasoning - PR #17384

• Nova Lite 2

  • Add Nova Lite 2 reasoning support with reasoningConfig - PR #17371

• Ollama

  • Fix auth not working with ollama.com - PR #17191

• Groq

  • Fix supports_response_schema before using json_tool_call workaround - PR #17438

• vLLM

  • Fix empty response + vLLM streaming - PR #17516

• Azure AI

  • Migrate Anthropic provider to Azure AI - PR #17202
  • Fix GA path for Azure OpenAI realtime models - PR #17260

• Bedrock TwelveLabs

  • Add support for TwelveLabs Pegasus video understanding - PR #17193

Bug Fixes​

• Bedrock

  • Fix extra_headers in messages API bedrock invoke - PR #17271
  • Fix Bedrock models in model map - PR #17419
  • Make Bedrock converse messages respect modify_params as expected - PR #17427
  • Fix Anthropic beta headers for Bedrock imported Qwen models - PR #17467
  • Preserve usage from JSON response for OpenAI provider in Bedrock - PR #17589

• SambaNova

  • Fix acompletion throws error with SambaNova models - PR #17217

• General

  • Fix AttributeError when metadata is null in request body - PR #17306
  • Fix 500 error for malformed request - PR #17291
  • Respect custom LLM provider in header - PR #17290
  • Replace deprecated .dict() with .model_dump() in streaming_handler - PR #17359

LLM API Endpoints​

Features​

• Responses API

  • Add cost tracking for responses API - PR #17258
  • Map output_tokens_details of responses API to completion_tokens_details - PR #17458
  • Add image generation support for Responses API - PR #16586

• Batch API

  • Add vLLM batch+files API support - PR #15823
  • Fix optional parameter default value - PR #17434
  • Add status parameter as optional for FileObject - PR #17431

• Video Generation API

  • Add passthrough cost tracking for Veo - PR #17296

• OCR API

  • Add missing OCR and aOCR to CallTypes enum - PR #17435

• General

  • Support routing to only websearch supported deployments - PR #17500

Bugs​

• General
  • Fix streaming error validation - PR #17242
  • Add length validation for empty tool_calls in delta - PR #17523

Management Endpoints / UI​

Features​

• New Login Page

  • New Login Page UI - PR #17443
  • Refactor /login route - PR #17379
  • Add auto_redirect_to_sso to UI Config - PR #17399
  • Add Auto Redirect to SSO to New Login Page - PR #17451

• Customer (End User) Usage

  • Customer (end user) Usage feature - PR #17498
  • Customer Usage UI - PR #17506
  • Add Info Banner for Customer Usage - PR #17598

• Virtual Keys

  • Standardize API Key vs Virtual Key in UI - PR #17325
  • Add User Alias Column to Internal User Table - PR #17321
  • Delete Credential Enhancements - PR #17317

• Models + Endpoints

  • Show all credential values on Edit Credential Modal - PR #17397
  • Change Edit Team Models Shown to Match Create Team - PR #17394
  • Support Images in Compare UI - PR #17562

• Callbacks

  • Show all callbacks on UI - PR #16335
  • Credentials to use React Query - PR #17465

• Management Routes

  • Allow admin viewer to access global tag usage - PR #17501
  • Allow wildcard routes for nonproxy admin (SCIM) - PR #17178
  • Return 404 when a user is not found on /user/info - PR #16850

• OCI Configuration

  • Enable Oracle Cloud Infrastructure configuration via UI - PR #17159

Bugs​

• UI Fixes

  • Fix Request and Response Panel JSONViewer - PR #17233
  • Adding Button Loading States to Edit Settings - PR #17236
  • Fix Various Text, button state, and test changes - PR #17237
  • Fix Fallbacks Immediately Deleting before API resolves - PR #17238
  • Remove Feature Flags - PR #17240
  • Fix metadata tags and model name display in UI for Azure passthrough - PR #17258
  • Change labeling around Vertex Fields - PR #17383
  • Remove second scrollbar when sidebar is expanded + tooltip z index - PR #17436
  • Fix Select in Edit Membership Modal - PR #17524
  • Change useAuthorized Hook to redirect to new Login Page - PR #17553

• SSO

  • Fix the generic SSO provider - PR #17227
  • Clear SSO integration for all users - PR #17287
  • Fix SSO users not added to Entra synced team - PR #17331

• Auth / JWT

  • JWT Auth - Allow using regular OIDC flow with user info endpoints - PR #17324
  • Fix litellm user auth not passing issue - PR #17342
  • Add other routes in JWT auth - PR #17345
  • Fix new org team validate against org - PR #17333
  • Fix litellm_enterprise ensure imported routes exist - PR #17337
  • Use organization.members instead of deprecated organization field - PR #17557

• Organizations/Teams

  • Fix organization max budget not enforced - PR #17334
  • Fix budget update to allow null max_budget - PR #17545

AI Integrations (2 new integrations)​

Logging (1 new integration)​

New Integration​

• Weave
  • Basic Weave OTEL integration - PR #17439

Improvements & Fixes​

• DataDog

  • Fix Datadog callback regression when ddtrace is installed - PR #17393

• Arize Phoenix

  • Fix clean arize-phoenix traces - PR #16611

• MLflow

  • Fix MLflow streaming spans for Anthropic passthrough - PR #17288

• Langfuse

  • Fix Langfuse logger test mock setup - PR #17591

• General

  • Improve PII anonymization handling in logging callbacks - PR #17207

Guardrails (1 new integration)​

New Integration​

• Generic Guardrail API
  • Generic Guardrail API - allows guardrail providers to add INSTANT support for LiteLLM w/out PR to repo - PR #17175
  • Guardrails API V2 - user api key metadata, session id, specify input type (request/response), image support - PR #17338
  • Guardrails API - add streaming support - PR #17400
  • Guardrails API - support tool call checks on OpenAI /chat/completions, OpenAI /responses, Anthropic /v1/messages - PR #17459
  • Guardrails API - new structured_messages param - PR #17518
  • Correctly map a v1/messages call to the anthropic unified guardrail - PR #17424
  • Support during_call event type for unified guardrails - PR #17514

Improvements & Fixes​

• Noma Guardrail

  • Refactor Noma guardrail to use shared Responses transformation and include system instructions - PR #17315

• Presidio

  • Handle empty content and error dict responses in guardrails - PR #17489
  • Fix Presidio guardrail test TypeError and license base64 decoding error - PR #17538

• Tool Permissions

  • Add regex-based tool_name/tool_type matching for tool-permission - PR #17164
  • Add images for tool permission guardrail documentation - PR #17322

• AIM Guardrails

  • Fix AIM guardrail tests - PR #17499

• Bedrock Guardrails

  • Fix Bedrock Guardrail indent and import - PR #17378

• General Guardrails

  • Mask all matching keywords in content filter - PR #17521
  • Ensure guardrail metadata is preserved in request_data - PR #17593
  • Fix apply_guardrail method and improve test isolation - PR #17555

Secret Managers​

• CyberArk

  • Allow setting SSL verify to false - PR #17433

• General

  • Make email and secret manager operations independent in key management hooks - PR #17551

Spend Tracking, Budgets and Rate Limiting​

• Rate Limiting

  • Parallel Request Limiter with /messages - PR #17426
  • Allow using dynamic rate limit/priority reservation on teams - PR #17061
  • Dynamic Rate Limiter - Fix token count increases/decreases by 1 instead of actual count + Redis TTL - PR #17558

• Spend Logs

  • Deprecate spend/logs & add spend/logs/v2 - PR #17167
  • Optimize SpendLogs queries to use timestamp filtering for index usage - PR #17504

• Enforce User Param

  • Enforce support of enforce_user_param to OpenAI post endpoints - PR #17407

MCP Gateway​

• MCP Configuration

  • Remove URL format validation for MCP server endpoints - PR #17270
  • Add stack trace to MCP error message - PR #17269

• MCP Tool Results

  • Preserve tool metadata in CallToolResult - PR #17561

Agent Gateway (A2A)​

• Agent Invocation

  • Allow invoking agents through AI Gateway - PR #17440
  • Allow tracking request/response in "Logs" Page - PR #17449

• Agent Access Control

  • Enforce Allowed agents by key, team + add agent access groups on backend - PR #17502

• Agent Gateway UI

  • Allow testing agents on UI - PR #17455
  • Set allowed agents by key, team - PR #17511

Performance / Loadbalancing / Reliability improvements​

• Audio/Speech Performance

  • Fix /audio/speech performance by using shared_sessions - PR #16739

• Memory Optimization

  • Prevent memory leak in aiohttp connection pooling - PR #17388
  • Lazy-load utils to reduce memory + import time - PR #17171

• Database

  • Update default database connection number - PR #17353
  • Update default proxy_batch_write_at number - PR #17355
  • Add background health checks to db - PR #17528

• Proxy Caching

  • Fix proxy caching between requests in aiohttp transport - PR #17122

• Session Management

  • Fix session consistency, move Lasso API version away from source code - PR #17316
  • Conditionally pass enable_cleanup_closed to aiohttp TCPConnector - PR #17367

• Vector Store

  • Fix vector store configuration synchronization failure - PR #17525

Documentation Updates​

• Provider Documentation

  • Add Azure AI Foundry documentation for Claude models - PR #17104
  • Document responses and embedding API for GitHub Copilot - PR #17456
  • Add gpt-5.1-codex-max to OpenAI provider documentation - PR #17602
  • Update Instructions For Phoenix Integration - PR #17373

• Guides

  • Add guide on how to debug gateway error vs provider error - PR #17387
  • Agent Gateway documentation - PR #17454
  • A2A Permission management documentation - PR #17515
  • Update docs to link agent hub - PR #17462

• Projects

  • Add Google ADK and Harbor to projects - PR #17352
  • Add Microsoft Agent Lightning to projects - PR #17422

• Cleanup

  • Cleanup: Remove orphan docs pages and Docusaurus template files - PR #17356
  • Remove source .env from docs - PR #17466

Infrastructure / CI/CD​

• Helm Chart

  • Add ingress-only labels - PR #17348

• Docker

  • Add retry logic to apk package installation in Dockerfile.non_root - PR #17596
  • Chainguard fixes - PR #17406

• OpenAPI Schema

  • Refactor add_schema_to_components to move definitions to components/schemas - PR #17389

• Security

  • Fix security vulnerability: update mdast-util-to-hast to 13.2.1 - PR #17601
  • Bump jws from 3.2.2 to 3.2.3 - PR #17494

New Contributors​

• @weichiet made their first contribution in PR #17242
• @AndyForest made their first contribution in PR #17220
• @omkar806 made their first contribution in PR #17217
• @v0rtex20k made their first contribution in PR #17178
• @hxomer made their first contribution in PR #17207
• @orgersh92 made their first contribution in PR #17316
• @dannykopping made their first contribution in PR #17313
• @rioiart made their first contribution in PR #17333
• @codgician made their first contribution in PR #17278
• @epistoteles made their first contribution in PR #17277
• @kothamah made their first contribution in PR #17368
• @flozonn made their first contribution in PR #17371
• @richardmcsong made their first contribution in PR #17389
• @matt-greathouse made their first contribution in PR #17384
• @mossbanay made their first contribution in PR #17380
• @mhielpos-asapp made their first contribution in PR #17376
• @Joilence made their first contribution in PR #17367
• @deepaktammali made their first contribution in PR #17357
• @axiomofjoy made their first contribution in PR #16611
• @DevajMody made their first contribution in PR #17445
• @andrewtruong made their first contribution in PR #17439
• @AnasAbdelR made their first contribution in PR #17490
• @dominicfeliton made their first contribution in PR #17516
• @kristianmitk made their first contribution in PR #17504
• @rgshr made their first contribution in PR #17130
• @dominicfallows made their first contribution in PR #17489
• @irfansofyana made their first contribution in PR #17467
• @GusBricker made their first contribution in PR #17191
• @OlivverX made their first contribution in PR #17255
• @withsmilo made their first contribution in PR #17585

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.80.5-stable

pip install litellm==1.80.5

Key Highlights​

• Gemini 3 - Day-0 support for Gemini 3 models with thought signatures
• Prompt Management - Full prompt versioning support with UI for editing, testing, and version history
• MCP Hub - Publish and discover MCP servers within your organization
• Model Compare UI - Side-by-side model comparison interface for testing
• Batch API Spend Tracking - Granular spend tracking with custom metadata for batch and file creation requests
• AWS IAM Secret Manager - IAM role authentication support for AWS Secret Manager
• Logging Callback Controls - Admin-level controls to prevent callers from disabling logging callbacks in compliance environments
• Proxy CLI JWT Authentication - Enable developers to authenticate to LiteLLM AI Gateway using the Proxy CLI
• Batch API Routing - Route batch operations to different provider accounts using model-specific credentials from your config.yaml

Prompt Management​

This release introduces LiteLLM Prompt Studio - a comprehensive prompt management solution built directly into the LiteLLM UI. Create, test, and version your prompts without leaving your browser.

You can now do the following on LiteLLM Prompt Studio:

• Create & Test Prompts: Build prompts with developer messages (system instructions) and test them in real-time with an interactive chat interface
• Dynamic Variables: Use {{variable_name}} syntax to create reusable prompt templates with automatic variable detection
• Version Control: Automatic versioning for every prompt update with complete version history tracking and rollback capabilities
• Prompt Studio: Edit prompts in a dedicated studio environment with live testing and preview

API Integration:

Use your prompts in any application with simple API calls:

response = client.chat.completions.create(
    model="gpt-4",
    extra_body={
        "prompt_id": "your-prompt-id",
        "prompt_version": 2,  # Optional: specify version
        "prompt_variables": {"name": "value"}  # Optional: pass variables
    }
)

Get started here: LiteLLM Prompt Management Documentation

Performance – /realtime 182× Lower p99 Latency​

This update reduces /realtime latency by removing redundant encodings on the hot path, reusing shared SSL contexts, and caching formatting strings that were being regenerated twice per request despite rarely changing.

Results​

Test Setup​

Model Compare UI​

New interactive playground UI enables side-by-side comparison of multiple LLM models, making it easy to evaluate and compare model responses.

Features:

• Compare responses from multiple models in real-time
• Side-by-side view with synchronized scrolling
• Support for all LiteLLM-supported models
• Cost tracking per model
• Response time comparison
• Pre-configured prompts for quick and easy testing

Details:

• Parameterization: Configure API keys, endpoints, models, and model parameters, as well as interaction types (chat completions, embeddings, etc.)

• Model Comparison: Compare up to 3 different models simultaneously with side-by-side response views

• Comparison Metrics: View detailed comparison information including:

  • Time To First Token
  • Input / Output / Reasoning Tokens
  • Total Latency
  • Cost (if enabled in config)

• Safety Filters: Configure and test guardrails (safety filters) directly in the playground interface

Get Started with Model Compare

New Providers and Endpoints​

New Providers​

New Models / Updated Models​

New Model Support​

Features​

• Gemini (Google AI Studio + Vertex AI)

  • Add Day 0 gemini-3-pro-preview support - PR #16719
  • Add support for Gemini 3 Pro Image model - PR #16938
  • Add reasoning_content to streaming responses with tools enabled - PR #16854
  • Add includeThoughts=True for Gemini 3 reasoning_effort - PR #16838
  • Support thought signatures for Gemini 3 in responses API - PR #16872
  • Correct wrong system message handling for gemma - PR #16767
  • Gemini 3 Pro Image: capture image_tokens and support cost_per_output_image - PR #16912
  • Fix missing costs for gemini-2.5-flash-image - PR #16882
  • Gemini 3 thought signatures in tool call id - PR #16895

• Azure

  • Add azure gpt-5.1 models - PR #16817
  • Add Azure models 2025 11 to cost maps - PR #16762
  • Update Azure Pricing - PR #16371
  • Add SSML Support for Azure Text-to-Speech (AVA) - PR #16747

• OpenAI

  • Support GPT-5.1 reasoning.effort='none' in proxy - PR #16745
  • Add gpt-5.1-codex and gpt-5.1-codex-mini models to documentation - PR #16735
  • Inherit BaseVideoConfig to enable async content response for OpenAI video - PR #16708

• Anthropic

  • Add support for strict parameter in Anthropic tool schemas - PR #16725
  • Add image as url support to anthropic - PR #16868
  • Add thought signature support to v1/messages api - PR #16812
  • Anthropic - support Structured Outputs output_format for Claude 4.5 sonnet and Opus 4.1 - PR #16949

• Bedrock

  • Haiku 4.5 correct Bedrock configs - PR #16732
  • Ensure consistent chunk IDs in Bedrock streaming responses - PR #16596
  • Add Claude 4.5 to US Gov Cloud - PR #16957
  • Fix images being dropped from tool results for bedrock - PR #16492

• Vertex AI

  • Add Vertex AI Image Edit Support - PR #16828
  • Update veo 3 pricing and add prod models - PR #16781
  • Fix Video download for veo3 - PR #16875

• Snowflake

  • Snowflake provider support: added embeddings, PAT, account_id - PR #15727

• OCI

  • Add oci_endpoint_id Parameter for OCI Dedicated Endpoints - PR #16723

• XAI

  • Add support for Grok 4.1 Fast models - PR #16936

• Together AI

  • Add GLM 4.6 from together.ai - PR #16942

• Cerebras

  • Fix Cerebras GPT-OSS-120B model name - PR #16939

Bug Fixes​

• OpenAI

  • Fix for 16863 - openai conversion from responses to completions - PR #16864
  • Revert "Make all gpt-5 and reasoning models to responses by default" - PR #16849

• General

  • Get custom_llm_provider from query param - PR #16731
  • Fix optional param mapping - PR #16852
  • Add None check for litellm_params - PR #16754

LLM API Endpoints​

Features​

• Responses API

  • Add Responses API support for gpt-5.1-codex model - PR #16845
  • Add managed files support for responses API - PR #16733
  • Add extra_body support for response supported api params from chat completion - PR #16765

• Batch API

  • Support /delete for files + support /cancel for batches - PR #16387
  • Add config based routing support for batches and files - PR #16872
  • Populate spend_logs_metadata in batch and files endpoints - PR #16921

• Search APIs

  • Search APIs - error in firecrawl-search "Invalid request body" - PR #16943

• Vector Stores

  • Fix vector store create issue - PR #16804
  • Team vector-store permissions now respected for key access - PR #16639

• Audio Transcription

  • Fix audio transcription cost tracking - PR #16478
  • Add missing shared_sessions to audio/transcriptions - PR #16858

• Video Generation API

  • Fix videos tagging - PR #16770

Bugs​

• General
  • Responses API cost tracking with custom deployment names - PR #16778
  • Trim logged response strings in spend-logs - PR #16654

Management Endpoints / UI​

Features​

• Proxy CLI Auth

  • Allow using JWTs for signing in with Proxy CLI - PR #16756

• Virtual Keys

  • Fix Key Model Alias Not Working - PR #16896

• Models + Endpoints

  • Add additional model settings to chat models in test key - PR #16793
  • Deactivate delete button on model table for config models - PR #16787
  • Change Public Model Hub to use proxyBaseUrl - PR #16892
  • Add JSON Viewer to request/response panel - PR #16687
  • Standarize icon images - PR #16837

• Teams

  • Teams table empty state - PR #16738

• Fallbacks

  • Fallbacks icon button tooltips and delete with friction - PR #16737

• MCP Servers

  • Delete user and MCP Server Modal, MCP Table Tooltips - PR #16751

• Callbacks

  • Expose backend endpoint for callbacks settings - PR #16698
  • Edit add callbacks route to use data from backend - PR #16699

• Usage & Analytics

  • Allow partial matches for user ID in User Table - PR #16952

• General UI

  • Allow setting base_url in API reference docs - PR #16674
  • Change /public fields to honor server root path - PR #16930
  • Correct ui build - PR #16702
  • Enable automatic dark/light mode based on system preference - PR #16748

Bugs​

• UI Fixes

  • Fix flaky tests due to antd Notification Manager - PR #16740
  • Fix UI MCP Tool Test Regression - PR #16695
  • Fix edit logging settings not appearing - PR #16798
  • Add css to truncate long request ids in request viewer - PR #16665
  • Remove azure/ prefix in Placeholder for Azure in Add Model - PR #16597
  • Remove UI Session Token from user/info return - PR #16851
  • Remove console logs and errors from model tab - PR #16455
  • Change Bulk Invite User Roles to Match Backend - PR #16906
  • Mock Tremor's Tooltip to Fix Flaky UI Tests - PR #16786
  • Fix e2e ui playwright test - PR #16799
  • Fix Tests in CI/CD - PR #16972

• SSO

  • Ensure role from SSO provider is used when a user is inserted onto LiteLLM - PR #16794
  • Docs - SSO - Manage User Roles via Azure App Roles - PR #16796

• Auth

  • Ensure Team Tags works when using JWT Auth - PR #16797
  • Fix key never expires - PR #16692

• Swagger UI

  • Fixes Swagger UI resolver errors for chat completion endpoints caused by Pydantic v2 $defs not being properly exposed in the OpenAPI schema - PR #16784

AI Integrations​

Logging​

• Arize Phoenix

  • Fix arize phoenix logging - PR #16301
  • Arize Phoenix - root span logging - PR #16949

• Langfuse

  • Filter secret fields form Langfuse - PR #16842

• General

  • Exclude litellm_credential_name from Sensitive Data Masker (Updated) - PR #16958
  • Allow admins to disable, dynamic callback controls - PR #16750

Guardrails​

• IBM Guardrails

  • Fix IBM Guardrails optional params, add extra_headers field - PR #16771

• Noma Guardrail

  • Use LiteLLM key alias as fallback Noma applicationId in NomaGuardrail - PR #16832
  • Allow custom violation message for tool-permission guardrail - PR #16916

• Grayswan Guardrail

  • Grayswan guardrail passthrough on flagged - PR #16891

• General Guardrails

  • Fix prompt injection not working - PR #16701

Prompt Management​

• Prompt Management
  • Allow specifying just prompt_id in a request to a model - PR #16834
  • Add support for versioning prompts - PR #16836
  • Allow storing prompt version in DB - PR #16848
  • Add UI for editing the prompts - PR #16853
  • Allow testing prompts with Chat UI - PR #16898
  • Allow viewing version history - PR #16901
  • Allow specifying prompt version in code - PR #16929
  • UI, allow seeing model, prompt id for Prompt - PR #16932
  • Show "get code" section for prompt management + minor polish of showing version history - PR #16941

Secret Managers​

• AWS Secrets Manager
  • Adds IAM role assumption support for AWS Secret Manager - PR #16887

MCP Gateway​

• MCP Hub - Publish/discover MCP Servers within a company - PR #16857
• MCP Resources - MCP resources support - PR #16800
• MCP OAuth - Docs - mcp oauth flow details - PR #16742
• MCP Lifecycle - Drop MCPClient.connect and use run_with_session lifecycle - PR #16696
• MCP Server IDs - Add mcp server ids - PR #16904
• MCP URL Format - Fix mcp url format - PR #16940

Performance / Loadbalancing / Reliability improvements​

• Realtime Endpoint Performance - Fix bottlenecks degrading realtime endpoint performance - PR #16670
• SSL Context Caching - Cache SSL contexts to prevent excessive memory allocation - PR #16955
• Cache Optimization - Fix cache cooldown key generation - PR #16954
• Router Cache - Fix routing for requests with same cacheable prefix but different user messages - PR #16951
• Redis Event Loop - Fix redis event loop closed at first call - PR #16913
• Dependency Management - Upgrade pydantic to version 2.11.0 - PR #16909

Documentation Updates​

• Provider Documentation

  • Add missing details to benchmark comparison - PR #16690
  • Fix anthropic pass-through endpoint - PR #16883
  • Cleanup repo and improve AI docs - PR #16775

• API Documentation

  • Add docs related to openai metadata - PR #16872
  • Update docs with all supported endpoints and cost tracking - PR #16872

• General Documentation

  • Add mini-swe-agent to Projects built on LiteLLM - PR #16971

Infrastructure / CI/CD​

• UI Testing

  • Break e2e_ui_testing into build, unit, and e2e steps - PR #16783
  • Building UI for Testing - PR #16968
  • CI/CD Fixes - PR #16937

• Dependency Management

  • Bump js-yaml from 3.14.1 to 3.14.2 in /tests/proxy_admin_ui_tests/ui_unit_tests - PR #16755
  • Bump js-yaml from 3.14.1 to 3.14.2 - PR #16802

• Migration

  • Migration job labels - PR #16831

• Config

  • This yaml actually works - PR #16757

• Release Notes

  • Add perf improvements on embeddings to release notes - PR #16697
  • Docs - v1.80.0 - PR #16694

• Investigation

  • Investigate issue root cause - PR #16859

New Contributors​

• @mattmorgis made their first contribution in PR #16371
• @mmandic-coatue made their first contribution in PR #16732
• @Bradley-Butcher made their first contribution in PR #16725
• @BenjaminLevy made their first contribution in PR #16757
• @CatBraaain made their first contribution in PR #16767
• @tushar8408 made their first contribution in PR #16831
• @nbsp1221 made their first contribution in PR #16845
• @idola9 made their first contribution in PR #16832
• @nkukard made their first contribution in PR #16864
• @alhuang10 made their first contribution in PR #16852
• @sebslight made their first contribution in PR #16838
• @TsurumaruTsuyoshi made their first contribution in PR #16905
• @cyberjunk made their first contribution in PR #16492
• @colinlin-stripe made their first contribution in PR #16895
• @sureshdsk made their first contribution in PR #16883
• @eiliyaabedini made their first contribution in PR #16875
• @justin-tahara made their first contribution in PR #16957
• @wangsoft made their first contribution in PR #16913
• @dsduenas made their first contribution in PR #16891

Known Issues​

• /audit and /user/available_users routes return 404. Fixed in PR #17337

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.80.0-stable

pip install litellm==1.80.0

Key Highlights​

• 🆕 Agent Hub Support - Register and make agents public for your organization
• RunwayML Provider - Complete video generation, image generation, and text-to-speech support
• GPT-5.1 Family Support - Day-0 support for OpenAI's latest GPT-5.1 and GPT-5.1-Codex models
• Prometheus OSS - Prometheus metrics now available in open-source version
• Vector Store Files API - Complete OpenAI-compatible Vector Store Files API with full CRUD operations
• Embeddings Performance - O(1) lookup optimization for router embeddings with shared sessions

Agent Hub​

This release adds support for registering and making agents public for your organization. This is great for Proxy Admins who want a central place to make agents built in their organization, discoverable to their users.

Here's the flow:

1. Add agent to litellm.
2. Make it public.
3. Allow anyone to discover it on the public AI Hub page.

Get Started with Agent Hub

Performance – /embeddings 13× Lower p95 Latency​

This update significantly improves /embeddings latency by routing it through the same optimized pipeline as /chat/completions, benefiting from all previously applied networking optimizations.

Results​

Test Setup​

🆕 RunwayML​

Complete integration for RunwayML's Gen-4 family of models, supporting video generation, image generation, and text-to-speech.

Supported Endpoints:

• /v1/videos - Video generation (Gen-4 Turbo, Gen-4 Aleph, Gen-3A Turbo)
• /v1/images/generations - Image generation (Gen-4 Image, Gen-4 Image Turbo)
• /v1/audio/speech - Text-to-speech (ElevenLabs Multilingual v2)

Quick Start:

curl --location 'http://localhost:4000/v1/videos' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer sk-1234' \
--data '{
    "model": "runwayml/gen4_turbo",
    "prompt": "A high quality demo video of litellm ai gateway",
    "input_reference": "https://example.com/image.jpg",
    "seconds": 5,
    "size": "1280x720"
}'

Get Started with RunwayML

Prometheus Metrics - Open Source​

Prometheus metrics are now available in the open-source version of LiteLLM, providing comprehensive observability for your AI Gateway without requiring an enterprise license.

Quick Start:

litellm_settings:
  success_callback: ["prometheus"]
  failure_callback: ["prometheus"]

Get Started with Prometheus

Vector Store Files API​

Complete OpenAI-compatible Vector Store Files API now stable, enabling full file lifecycle management within vector stores.

Supported Endpoints:

• POST /v1/vector_stores/{vector_store_id}/files - Create vector store file
• GET /v1/vector_stores/{vector_store_id}/files - List vector store files
• GET /v1/vector_stores/{vector_store_id}/files/{file_id} - Retrieve vector store file
• GET /v1/vector_stores/{vector_store_id}/files/{file_id}/content - Retrieve file content
• DELETE /v1/vector_stores/{vector_store_id}/files/{file_id} - Delete vector store file
• DELETE /v1/vector_stores/{vector_store_id} - Delete vector store

Quick Start:

curl --location 'http://localhost:4000/v1/vector_stores/vs_123/files' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer sk-1234' \
--data '{
    "file_id": "file_abc"
}'

Get Started with Vector Stores

New Providers and Endpoints​

New Providers​

New LLM API Endpoints​

New Models / Updated Models​

New Model Support​

Video Generation Models​

Image Generation Models​

Audio Models​

Features​

• OpenAI

  • Add GPT-5.1 family support with reasoning capabilities - PR #16598
  • Add support for reasoning_effort='none' for GPT-5.1 - PR #16658
  • Add verbosity parameter support for GPT-5 family models - PR #16660
  • Fix forward OpenAI organization for image generation - PR #16607

• Gemini (Google AI Studio + Vertex AI)

  • Add support for reasoning_effort='none' for Gemini models - PR #16548
  • Add all Gemini image models support in image generation - PR #16526
  • Add Gemini image edit support - PR #16430
  • Fix preserve non-ASCII characters in function call arguments - PR #16550
  • Fix Gemini conversation format issue with MCP auto-execution - PR #16592

• Bedrock

  • Add support for filtering knowledge base queries - PR #16543
  • Ensure correct aws_region is used when provided dynamically for embeddings - PR #16547
  • Add support for custom KMS encryption keys in Bedrock Batch operations - PR #16662
  • Add bearer token authentication support for AgentCore - PR #16556
  • Fix AgentCore SSE stream iterator to async for proper streaming support - PR #16293

• Anthropic

  • Add context management param support - PR #16528
  • Fix preserve $defs for Anthropic tools input schema - PR #16648
  • Fix support Anthropic tool_use and tool_result in token counter - PR #16351

• Vertex AI

  • Add Vertex Kimi-K2-Thinking support - PR #16671
  • Add vertex_credentials support to litellm.rerank() - PR #16479

• Mistral

  • Fix Magistral streaming to emit reasoning chunks - PR #16434

• Moonshot (Kimi)

  • Add Kimi K2 thinking model support - PR #16445

• SambaNova

  • Fix SambaNova API rejecting requests when message content is passed as a list format - PR #16612

• VLLM

  • Fix use vllm passthrough config for hosted vllm provider instead of raising error - PR #16537
  • Add headers to VLLM Passthrough requests with success event logging - PR #16532

• Azure

  • Fix improve Azure auth parameter handling for None values - PR #14436

• Groq

  • Fix parse failed chunks for Groq - PR #16595

• Voyage

  • Add Voyage 3.5 and 3.5-lite embeddings pricing and doc update - PR #16641

• Fal.ai

  • Add fal-ai/flux/schnell support - PR #16580
  • Add all Imagen4 variants of fal ai in model map - PR #16579

Bug Fixes​

• General
  • Fix sanitize null token usage in OpenAI-compatible responses - PR #16493
  • Fix apply provided timeout value to ClientTimeout.total - PR #16395
  • Fix raising wrong 429 error on wrong exception - PR #16482
  • Add new models, delete repeat models, update pricing - PR #16491
  • Update model logging format for custom LLM provider - PR #16485

LLM API Endpoints​

New Endpoints​

• GET /providers
  • Add GET list of providers endpoint - PR #16432

Features​

• Video Generation API

  • Allow internal users to access video generation routes - PR #16472

• Vector Stores API

  • Vector store files stable release with complete CRUD operations - PR #16643
    • POST /v1/vector_stores/{vector_store_id}/files - Create vector store file
    • GET /v1/vector_stores/{vector_store_id}/files - List vector store files
    • GET /v1/vector_stores/{vector_store_id}/files/{file_id} - Retrieve vector store file
    • GET /v1/vector_stores/{vector_store_id}/files/{file_id}/content - Retrieve file content
    • DELETE /v1/vector_stores/{vector_store_id}/files/{file_id} - Delete vector store file
    • DELETE /v1/vector_stores/{vector_store_id} - Delete vector store
  • Ensure users can access search_results for both stream + non-stream response - PR #16459

Bugs​

• Video Generation API

  • Fix use GET for /v1/videos/{video_id}/content - PR #16672

• General

  • Fix remove generic exception handling - PR #16599

Management Endpoints / UI​

Features​

• Proxy CLI Auth

  • Fix remove strict master_key check in add_deployment - PR #16453

• Virtual Keys

  • UI - Add Tags To Edit Key Flow - PR #16500
  • UI - Test Key Page show models based on selected endpoint - PR #16452
  • UI - Expose user_alias in view and update path - PR #16669

• Models + Endpoints

  • UI - Add LiteLLM Params to Edit Model - PR #16496
  • UI - Add Model use backend data - PR #16664
  • UI - Remove Description Field from LLM Credentials - PR #16608
  • UI - Add RunwayML on Admin UI supported models/providers - PR #16606
  • Infra - Migrate Add Model Fields to Backend - PR #16620
  • Add API Endpoint for creating model access group - PR #16663

• Teams

  • UI - Invite User Searchable Team Select - PR #16454
  • Fix use user budget instead of key budget when creating new team - PR #16074

• Budgets

  • UI - Move Budgets out of Experimental - PR #16544

• Guardrails

  • UI - Config Guardrails should not be deletable from table - PR #16540
  • Fix remove enterprise restriction from guardrails list endpoint - PR #15333

• Callbacks

  • UI - New Callbacks table - PR #16512
  • Fix delete callbacks failing - PR #16473

• Usage & Analytics

  • UI - Improve Usage Indicator - PR #16504
  • UI - Model Info Page Health Check - PR #16416
  • Infra - Show Deprecation Warning for Model Analytics Tab - PR #16417
  • Fix Litellm tags usage add request_id - PR #16111

• Health Check

  • Add Langfuse OTEL and SQS to Health Check - PR #16514

• General UI

  • UI - Normalize table action columns appearance - PR #16657
  • UI - Button Styles and Sizing in Settings Pages - PR #16600
  • UI - SSO Modal Cosmetic Changes - PR #16554
  • Fix UI logos loading with SERVER_ROOT_PATH - PR #16618
  • Fix remove misleading 'Custom' option mention from OpenAI endpoint tooltips - PR #16622

• SSO

  • Ensure role from SSO provider is used when a user is inserted onto LiteLLM - PR #16794

Bugs​

• Management Endpoints
  • Fix inconsistent error responses in customer management endpoints - PR #16450
  • Fix correct date range filtering in /spend/logs endpoint - PR #16443
  • Fix /spend/logs/ui Access Control - PR #16446
  • Add pagination for /spend/logs/session/ui endpoint - PR #16603
  • Fix LiteLLM Usage shows key_hash - PR #16471
  • Fix app_roles missing from jwt payload - PR #16448

Logging / Guardrail / Prompt Management Integrations​

New Integration​

• 🆕 Zscaler AI Guard
  • Add Zscaler AI Guard hook for security policy enforcement - PR #15691

Logging​

• Langfuse

  • Fix handle null usage values to prevent validation errors - PR #16396

• CloudZero

  • Fix updated spend would not be sent to CloudZero - PR #16201

Guardrails​

• IBM Detector
  • Ensure detector-id is passed as header to IBM detector server - PR #16649

Prompt Management​

• Custom Prompt Management
  • Add SDK focused examples for custom prompt management - PR #16441

Spend Tracking, Budgets and Rate Limiting​

• End User Budgets
  • Allow pointing max_end_user budget to an id, so the default ID applies to all end users - PR #16456

MCP Gateway​

• Configuration
  • Add dynamic OAuth2 metadata discovery for MCP servers - PR #16676
  • Fix allow tool call even when server name prefix is missing - PR #16425
  • Fix exclude unauthorized MCP servers from allowed server list - PR #16551
  • Fix unable to delete MCP server from permission settings - PR #16407
  • Fix avoid crashing when MCP server record lacks credentials - PR #16601

Agents​

• Agent Registration (A2A Spec)
  • Support agent registration + discovery following Agent-to-Agent specification - PR #16615

Performance / Loadbalancing / Reliability improvements​

• Embeddings Performance

  • Use router's O(1) lookup and shared sessions for embeddings - PR #16344

• Router Reliability

  • Support default fallbacks for unknown models - PR #16419

• Callback Management

  • Add atexit handlers to flush callbacks for async completions - PR #16487

General Proxy Improvements​

• Configuration Management
  • Fix update model_cost_map_url to use environment variable - PR #16429

Documentation Updates​

• Provider Documentation

  • Fix streaming example in README - PR #16461
  • Update broken Slack invite links to support page - PR #16546
  • Fix code block indentation for fallbacks page - PR #16542
  • Documentation code example corrections - PR #16502
  • Document reasoning_effort summary field options - PR #16549

• API Documentation

  • Add docs on APIs for model access management - PR #16673
  • Add docs for showing how to auto reload new pricing data - PR #16675
  • LiteLLM Quick start - show how model resolution works - PR #16602
  • Add docs for tracking callback failure - PR #16474

• General Documentation

  • Fix container api link in release page - PR #16440
  • Add softgen to projects that are using litellm - PR #16423

New Contributors​

• @artplan1 made their first contribution in PR #16423
• @JehandadK made their first contribution in PR #16472
• @vmiscenko made their first contribution in PR #16453
• @mcowger made their first contribution in PR #16429
• @yellowsubmarine372 made their first contribution in PR #16395
• @Hebruwu made their first contribution in PR #16201
• @jwang-gif made their first contribution in PR #15691
• @AnthonyMonaco made their first contribution in PR #16502
• @andrewm4894 made their first contribution in PR #16487
• @f14-bertolotti made their first contribution in PR #16485
• @busla made their first contribution in PR #16293
• @MightyGoldenOctopus made their first contribution in PR #16537
• @ultmaster made their first contribution in PR #14436
• @bchrobot made their first contribution in PR #16542
• @sep-grindr made their first contribution in PR #16622
• @pnookala-godaddy made their first contribution in PR #16607
• @dtunikov made their first contribution in PR #16592
• @lukapecnik made their first contribution in PR #16648
• @jyeros made their first contribution in PR #16618

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.79.3-stable

pip install litellm==1.79.3.rc.1

Key Highlights​

• LiteLLM Custom Guardrail - Built-in guardrail with UI configuration support
• Performance Improvements - /responses API 19× Lower Median Latency
• Veo3 Video Generation (Vertex AI + Google AI Studio) - Use OpenAI Video API to generate videos with Vertex AI and Google AI Studio Veo3 models

Built-in Guardrails on AI Gateway​

This release introduces built-in guardrails for LiteLLM AI Gateway, allowing you to enforce protections without depending on an external guardrail API.

• Blocking Keywords - Block known sensitive keywords like "litellm", "python", etc.
• Pattern Detection - Block known sensitive patterns like emails, Social Security Numbers, API keys, etc.
• Custom Regex Patterns - Define custom regex patterns for your specific use case.

Get started with the built-in guardrails on AI Gateway here.

Performance – /responses 19× Lower Median Latency​

This update significantly improves /responses latency by integrating our internal network management for connection handling, eliminating per-request setup overhead.

Results​

Test Setup​

New Models / Updated Models​

New Model Support​

OCR Models​

Search Models​

Features​

• Azure

  • Add Azure GPT-5-Pro Responses API support with reasoning capabilities - PR #16235
  • Add gpt-image-1-mini pricing for Azure with quality tiers (low/medium/high) - PR #16182
  • Add support for returning Azure Content Policy error information when exceptions from Azure OpenAI occur - PR #16231
  • Fix Azure GPT-5 incorrectly routed to O-series config (temperature parameter unsupported) - PR #16246
  • Fix Azure doesn't accept extra body param - PR #16116
  • Fix Azure DALL-E-3 health check content policy violation by using safe default prompt - PR #16329

• Bedrock

  • Fix empty assistant message handling in AWS Bedrock Converse API to prevent 400 Bad Request errors - PR #15850
  • Fix: Filter AWS authentication params from Bedrock InvokeModel request body - PR #16315
  • Fix Bedrock proxy adding name to file content, breaks when cache_control in use - PR #16275
  • Fix global.anthropic.claude-haiku-4-5-20251001-v1:0 supports_reasoning flag and update pricing - PR #16263

• Gemini (Google AI Studio + Vertex AI)

  • Add gemini live audio model cost in model map - PR #16183
  • Fix translation problem with Gemini parallel tool calls - PR #16194
  • Fix: Send Gemini API key via x-goog-api-key header with custom api_base - PR #16085
  • Fix image_config.aspect_ratio not working for gemini-2.5-flash-image - PR #15999
  • Fix Gemini minimal reasoning env overrides disabling thoughts - PR #16347
  • Fix cache_read_input_token_cost for gemini-2.5-flash - PR #16354

• Anthropic

  • Fix Anthropic token counting for VertexAI - PR #16171
  • Fix anthropic-adapter: properly translate Anthropic image format to OpenAI - PR #16202
  • Enable automated prompt caching message format for Claude on Databricks - PR #16200
  • Add support for Anthropic Memory Tool - PR #16115
  • Propagate cache creation/read token costs for model info to fix Anthropic long context cost calculations - PR #16376

• Vertex AI

  • Add Vertex MiniMAX m2 model support - PR #16373
  • Correctly map 429 Resource Exhausted to RateLimitError - PR #16363
  • Add vertex_credentials support to litellm.rerank() for Vertex AI - PR #16266

• Databricks

  • Fix databricks streaming - PR #16368

• Deepgram

  • Return the diarized transcript when it's required in the request - PR #16133

• Fireworks

  • Update Fireworks audio endpoints to new api.fireworks.ai domains - PR #16346

• Cohere

  • Add cohere embed-v4.0 model support - PR #16358

• Watsonx

  • Support reasoning_effort for watsonx chat models - PR #16261

• OpenAI

  • Remove automatic summary from reasoning_effort transformation - PR #16210

• XAI

  • Remove Grok 4 Models Reasoning Effort Parameter - PR #16265

• Hosted VLLM

  • Fix HostedVLLMRerankConfig will not be used - PR #16352

New Provider Support​

• Bedrock Agentcore
  • Add Bedrock Agentcore as a provider on LiteLLM Python SDK and LiteLLM AI Gateway - PR #16252

LLM API Endpoints​

Features​

• OCR API

  • Add VertexAI OCR provider support + cost tracking - PR #16216
  • Add Azure AI Doc Intelligence OCR support - PR #16219

• Search API

  • Add firecrawl search API support with tiered pricing - PR #16257
  • Add searxng search API provider - PR #16259

• Responses API

  • Support responses API streaming in langfuse otel - PR #16153
  • Pass extra_body parameters to provider in Responses API requests - PR #16320

• Container API

  • Add E2E Container API Support - PR #16136
  • Update container documentation to be similar to others - PR #16327

• Video Generation API

  • Add Vertex and Gemini Videos API with Cost Tracking + UI support - PR #16323
  • Add custom_llm_provider support for video endpoints (non-generation) - PR #16121

• Audio API

  • Add gpt-4o-transcribe cost tracking - PR #16412

• Vector Stores

  • Milvus - search vector store support + support multi-part form data on passthrough - PR #16035
  • Azure AI Vector Stores - support "virtual" indexes + create vector store on passthrough API - PR #16160
  • Milvus - Passthrough API support - adds create + read vector store support via passthrough API's - PR #16170

• Embeddings API

  • Use valid CallTypes enum value in embeddings endpoint - PR #16328

• Rerank API

  • Generalize tiered pricing in generic cost calculator - PR #16150

Bugs​

• General
  • Fix index field not populated in streaming mode with n>1 and tool calls - PR #15962
  • Pass aws_region_name in litellm_params - PR #16321
  • Add retry-after header support for errors 502, 503, 504 - PR #16288

Management Endpoints / UI​

Features​

• Virtual Keys

  • UI - Delete Team Member with friction - PR #16167
  • UI - Litellm test key audio support - PR #16251
  • UI - Test Key Page Revert Model To Single Select - PR #16390

• Models + Endpoints

  • UI - Add Model Existing Credentials Improvement - PR #16166
  • UI - Add Azure AD Token field and Azure API Key optional - PR #16331
  • UI - Fixed Label for vLLM in Model Create Flow - PR #16285
  • UI - Include Model Access Group Models on Team Models Table - PR #16298
  • Fix /model_group/info Returning Entire Model List for SSO Users - PR #16296
  • Litellm non root docker Model Hub Table fix - PR #16282

• Guardrails

  • UI - Fix regression where Guardrail Entity Could not be selected and entity was not displayed - PR #16165
  • UI - Guardrail Info Page Show PII Config - PR #16164
  • Change guardrail_information to list type - PR #16127
  • UI - LiteLLM Guardrail - ensure you can see UI Friendly name for PII Patterns - PR #16382
  • UI - Guardrails - LiteLLM Content Filter, Allow Viewing/Editing Content Filter Settings - PR #16383
  • UI - Guardrails - allow updating guardrails through UI. Ensure litellm_params actually get updated in memory - PR #16384

• SSO Settings

  • Support dot notation on ui sso - PR #16135
  • UI - Prevent trailing slash in sso proxy base url input - PR #16244
  • UI - SSO Proxy Base URL input validation and remove normalizing / - PR #16332
  • UI - Surface SSO Create errors on create flow - PR #16369

• Usage & Analytics

  • UI - Tag Usage Top Model Table View and Label Fix - PR #16249
  • UI - Litellm usage date picker - PR #16264

• Cache Settings

  • UI - Cache Settings Redis Add Semantic Cache Settings - PR #16398

Bugs​

• General
  • UI - Remove encoding_format in request for embedding models - PR #16367
  • UI - Revert Changes for Test Key Multiple Model Select - PR #16372
  • UI - Various Small Issues - PR #16406

AI Integrations​

Logging​

• Langfuse

  • Fix langfuse input tokens logic for cached tokens - PR #16203

• Opik

  • Fix the bug with not incorrect attachment to existing trace & refactor - PR #15529

• S3

  • S3 logger, add support for ssl_verify when using minio logger - PR #16211
  • Strip base64 in s3 - PR #16157
  • Add allowing Key based prefix to s3 path - PR #16237
  • Add Prometheus metric to track callback logging failures in S3 - PR #16209

• OpenTelemetry

  • OTEL - Log Cost Breakdown on OTEL Logger - PR #16334

• DataDog

  • Add DD Agent Host support for datadog callback - PR #16379

Guardrails​

• Noma

  • Revert Noma Apply Guardrail implementation - PR #16214
  • Litellm noma guardrail support images - PR #16199

• PANW Prisma AIRS

  • PANW prisma airs guardrail deduplication and enhanced session tracking - PR #16273

• LiteLLM Custom Guardrail

  • Add LiteLLM Gateway built in guardrail - PR #16338
  • UI - Allow configuring LiteLLM Custom Guardrail - PR #16339
  • Bug Fix: Content Filter Guard - PR #16414

Secret Managers​

• CyberArk

  • Add CyberArk Secrets Manager Integration - PR #16278
  • Cyber Ark - Add Key Rotations support - PR #16289

• HashiCorp Vault

  • Add configurable mount name and path prefix for HashiCorp Vault - PR #16253
  • Secret Manager - Hashicorp, add auth via approle - PR #16374

• AWS Secrets Manager

  • Add tags and descriptions support to aws secrets manager - PR #16224

• Custom Secret Manager

  • Add Custom Secret Manager - Allow users to define and write a custom secret manager - PR #16297

• General

  • Email Notifications - Ensure Users get Key Rotated Email - PR #16292
  • Fix verify ssl on sts boto3 - PR #16313

Spend Tracking, Budgets and Rate Limiting​

• Cost Tracking
  • Fix OpenAI Responses API streaming tests usage field names and cost calculation - PR #16236

MCP Gateway​

• Configuration
  • Configure static mcp header - PR #16179
  • Persist mcp credentials in db - PR #16308

Performance / Loadbalancing / Reliability improvements​

• Memory Leak Fixes

  • Resolve memory accumulation caused by Pydantic 2.11+ deprecation warnings - PR #16110

• Session Management

  • Add shared_session support to responses API - PR #16260

• Error Handling

  • Gracefully handle connection closed errors during streaming - PR #16294
  • Handle None values in daily spend sort key - PR #16245

• Configuration

  • Remove minimum validation for cache control injection index - PR #16149
  • Improve clearing logic - only remove unvisited endpoints - PR #16400

• Redis

  • Handle float redis_version from AWS ElastiCache Valkey - PR #16207

• Hooks

  • Add parallel execution handling in during_call_hook - PR #16279

• Infrastructure

  • Install runtime node for prisma - PR #16410

Documentation Updates​

• Provider Documentation

  • Docs - v1.79.1 - PR #16163
  • Fix broken link on model_management.md - PR #16217
  • Fix image generation response format - use 'images' array instead of 'image' object - PR #16378

• General Documentation

  • Add minimum resource requirement for production - PR #16146
  • Add benchmark comparison with other AI gateways - PR #16248
  • LiteLLM content filter guard documentation - PR #16413
  • Fix typo of the word orginal - PR #16255

• Security

  • Remove tornado test files (including test.key), fixes Python 3.13 security issues - PR #16342

New Contributors​

• @steve-gore-snapdocs made their first contribution in PR #16149
• @timbmg made their first contribution in PR #16120
• @Nivg made their first contribution in PR #16202
• @pablobgar made their first contribution in PR #16194
• @AlanPonnachan made their first contribution in PR #16150
• @Chesars made their first contribution in PR #16236
• @bowenliang123 made their first contribution in PR #16255
• @dean-zavad made their first contribution in PR #16199
• @alexkuzmik made their first contribution in PR #15529
• @Granine made their first contribution in PR #16281
• @Oodapow made their first contribution in PR #16279
• @jgoodyear made their first contribution in PR #16275
• @Qanpi made their first contribution in PR #16321
• @ShimonMimoun made their first contribution in PR #16313
• @andriykislitsyn made their first contribution in PR #16288
• @reckless-huang made their first contribution in PR #16263
• @chenmoneygithub made their first contribution in PR #16368
• @stembe-digitalex made their first contribution in PR #16354
• @jfcherng made their first contribution in PR #16352
• @xingyaoww made their first contribution in PR #16246
• @emerzon made their first contribution in PR #16373
• @wwwillchen made their first contribution in PR #16376
• @fabriciojoc made their first contribution in PR #16203
• @jroberts2600 made their first contribution in PR #16273

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.79.1-stable

pip install litellm==1.80.0

Key Highlights​

• Container API Support - End-to-end OpenAI Container API support with proxy integration, logging, and cost tracking
• FAL AI Image Generation - Native support for FAL AI image generation models with cost tracking
• UI Enhancements - Guardrail Playground, Cache Settings, Tag Routing, SSO Settings
• Batch API Rate Limiting - Input-based rate limits support for Batch API requests
• Vector Store Expansion - Milvus vector store support and Azure AI virtual indexes
• Memory Leak Fixes - Resolved issues accounting for 90% of memory leaks on Python SDK & AI Gateway

Dependency Upgrades​

• Dependencies
  • Build(deps): bump starlette from 0.47.2 to 0.49.1 - PR #16027
  • Build(deps): bump fastapi from 0.116.1 to 0.120.1 - PR #16054
  • Build(deps): bump hono from 4.9.7 to 4.10.3 in /litellm-js/spend-logs - PR #15915

New Models / Updated Models​

New Model Support​

Features​

• Anthropic

  • Extended Claude 3-7 Sonnet deprecation date from 2026-02-01 to 2026-02-19 - PR #15976
  • Extended Claude Opus 4-0 deprecation date from 2025-03-01 to 2026-05-01 - PR #15976
  • Removed Claude Haiku 3-5 deprecation date (previously 2025-03-01) - PR #15976
  • Added Claude Opus 4-1, Claude Opus 4-0 20250513, Claude Sonnet 4 20250514 deprecation dates - PR #15976
  • Added web search support for Claude Opus 4-1 - PR #15976

• Bedrock

  • Fix empty assistant message handling in AWS Bedrock Converse API to prevent 400 Bad Request errors - PR #15850
  • Allow using ARNs when generating images via Bedrock - PR #15789
  • Add per model group header forwarding for Bedrock Invoke API - PR #16042
  • Preserve Bedrock inference profile IDs in health checks - PR #15947
  • Added fallback logic for detecting file content-type when S3 returns generic type - When using Bedrock with S3-hosted files, if the S3 object's Content-Type is not correctly set (e.g., binary/octet-stream instead of image/png), Bedrock can now handle it correctly - PR #15635

• Azure

  • Add deprecation dates for Azure OpenAI models (gpt-4o-2024-08-06, gpt-4o-2024-11-20, gpt-4.1 series, o3-2025-04-16, text-embedding-3-small) - PR #15976
  • Fix Azure OpenAI ContextWindowExceededError mapping from Azure errors - PR #15981
  • Add handling for v1 under Azure API versions - PR #15984
  • Fix azure doesn't accept extra body param - PR #16116

• OpenAI

  • Add deprecation dates for gpt-3.5-turbo-1106, gpt-4-0125-preview, gpt-4-1106-preview, o1-mini-2024-09-12 - PR #15976
  • Add extended Sora-2 modality support (text + image inputs) - PR #15976
  • Updated OpenAI Sora-2-Pro pricing to $0.30/video/second - PR #15976

• OpenRouter

  • Add Claude Haiku 4.5 pricing for OpenRouter - PR #15909
  • Add base_url config with environment variables documentation - PR #15946

• Mistral

  • Add codestral-embed-2505 embedding model - PR #16071

• Gemini (Google AI Studio + Vertex AI)

  • Fix gemini request mutation for tool use - PR #16002
  • Add gemini-embedding-001 pricing entry for Google GenAI API - PR #16078
  • Changes to fix frequency_penalty and presence_penalty issue for gemini-2.5-pro model - PR #16041

• DeepInfra

  • Add vision support for Qwen/Qwen3-chat-32b model - PR #15976

• Vercel AI Gateway

  • Fix vercel_ai_gateway entry for glm-4.6 (moved from vercel_ai_gateway/glm-4.6 to vercel_ai_gateway/zai/glm-4.6) - PR #16084

• Fireworks

  • Don't add "accounts/fireworks/models" prefix for Fireworks Provider - PR #15938

• Cohere

  • Add OpenAI-compatible annotations support for Cohere v2 citations - PR #16038

• Deepgram

  • Handle Deepgram detected language when available - PR #16093

Bug Fixes​

• Xai
  • Add Xai websearch cost tracking - PR #16001

New Provider Support​

• FAL AI

  • Add FAL AI Image Generation support - PR #16067

• OCI (Oracle Cloud Infrastructure)

  • Add OCI Signer Authentication support - PR #16064

LLM API Endpoints​

Features​

• Container API

  • Add end-to-end OpenAI Container API support to LiteLLM SDK - PR #16136
  • Add proxy support for container APIs - PR #16049
  • Add logging support for Container API - PR #16049
  • Add cost tracking support for containers with documentation - PR #16117

• Responses API

  • Respect LiteLLM-Disable-Message-Redaction header for Responses API - PR #15966
  • Add /openai routes for responses API (Azure OpenAI SDK Compatibility) - PR #15988
  • Redact reasoning summaries in ResponsesAPI output when message logging is disabled - PR #15965
  • Support text.format parameter in Responses API for providers without native ResponsesAPIConfig - PR #16023
  • Add LLM provider response headers to Responses API - PR #16091

• Video Generation API

  • Add custom_llm_provider support for video endpoints (non-generation) - PR #16121
  • Fix documentation for videos - PR #15937
  • Add OpenAI client usage documentation for videos and fix navigation visibility - PR #15996

• Moderations API

  • Moderations endpoint now respects api_base configuration parameter - PR #16087

• Vector Stores

  • Milvus - search vector store support - PR #16035
  • Azure AI Vector Stores - support "virtual" indexes + create vector store on passthrough API - PR #16160

• Passthrough Endpoints

  • Support multi-part form data on passthrough - PR #16035

Management Endpoints / UI​

Features​

• Virtual Keys

  • Validation for Proxy Base URL in SSO Settings - PR #16082
  • Test Key UI Embeddings support - PR #16065
  • Add Key Type Select in Key Settings - PR #16034
  • Key Already Exist Error Notification - PR #15993

• Models + Endpoints

  • Changed API Base from Select to Input in New LLM Credentials - PR #15987
  • Remove limit from admin UI numerical input - PR #15991
  • Config Models should not be editable - PR #16020
  • Add tags in model creation - PR #16138
  • Add Tags to update model - PR #16140

• Guardrails

  • Add Apply Guardrail Testing Playground - PR #16030
  • Config Guardrails should not be editable and guardrail info fix - PR #16142

• Cache Settings

  • Allow setting cache settings on UI - PR #16143

• Routing

  • Allow setting all routing strategies, tag filtering on UI - PR #16139

• Admin Settings

  • Add license metadata to health/readiness endpoint - PR #15997
  • Litellm Backend SSO Changes - PR #16029

Logging / Guardrail / Prompt Management Integrations​

Features​

• OpenTelemetry

  • Enable OpenTelemetry context propagation by external tracers - PR #15940
  • Ensure error information is logged on OTEL - PR #15978

• Langfuse

  • Fix duplicate trace in langfuse_otel - PR #15931
  • Support tool usage messages with Langfuse OTEL integration - PR #15932

• DataDog

  • Ensure key's metadata + guardrail is logged on DD - PR #15980

• Opik

  • Enhance requester metadata retrieval from API key auth - PR #15897
  • User auth key metadata Documentation - PR #16004

• SQS

  • Add Base64 handling for SQS Logger - PR #16028

• General

  • Fix: User API key and team id and user id missing from custom callback is not misfiring - PR #15982

Guardrails​

• IBM Guardrails

  • Update IBM Guardrails to correctly use SSL Verify argument - PR #15975
  • Add additional detail to ibm_guardrails.md documentation - PR #15971

• Model Armor

  • Support during_call for model armor guardrails - PR #15970

• Lasso Security

  • Upgrade to Lasso API v3 and fix ULID generation - PR #15941

• PANW Prisma AIRS

  • Add per-request profile overrides to PANW Prisma AIRS - PR #16069

• Grayswan

  • Improve Grayswan guardrail documentation - PR #15875

• Pillar AI

  • Graceful degradation for pillar service when using litellm - PR #15857

• General

  • Ensure Key Guardrails are applied - PR #16025

Prompt Management​

• GitLab
  • Add GitlabPromptCache and enable subfolder access - PR #15712

Spend Tracking, Budgets and Rate Limiting​

• Cost Tracking

  • Fix spend tracking for OCR/aOCR requests (log pages_processed + recognize OCRResponse) - PR #16070

• Rate Limiting

  • Add support for Batch API Rate limiting - PR1 adds support for input based rate limits - PR #16075
  • Handle multiple rate limit types per descriptor and prevent IndexError - PR #16039

MCP Gateway​

• OAuth
  • Add support for dynamic client registration - PR #15921
  • Respect X-Forwarded- headers in OAuth endpoints - PR #16036

Performance / Loadbalancing / Reliability improvements​

• Memory Leak Fixes

  • Fix: prevent httpx DeprecationWarning memory leak in AsyncHTTPHandler - PR #16024
  • Fix: resolve memory accumulation caused by Pydantic 2.11+ deprecation warnings - PR #16110
  • Fix(apscheduler): prevent memory leaks from jitter and frequent job intervals - PR #15846

• Configuration

  • Remove minimum validation for cache control injection index - PR #16149
  • Fix prompt_caching.md: wrong prompt_tokens definition - PR #16044

Documentation Updates​

• Provider Documentation

  • Use custom-llm-provider header in examples - PR #16055
  • Litellm docs readme fixes - PR #16107
  • Readme fixes add supported providers - PR #16109

• Model References

  • Add supports vision field to qwen-vl models in model_prices_and_context_window.json - PR #16106

• General Documentation

  • 1-79-0 docs - PR #15936
  • Add minimum resource requirement for production - PR #16146

New Contributors​

• @RobGeada made their first contribution in PR #15975
• @shanto12 made their first contribution in PR #15946
• @dima-hx430 made their first contribution in PR #15976
• @m-misiura made their first contribution in PR #15971
• @ylgibby made their first contribution in PR #15947
• @Somtom made their first contribution in PR #15909
• @rodolfo-nobrega made their first contribution in PR #16023
• @bernata made their first contribution in PR #15997
• @AlbertDeFusco made their first contribution in PR #15881
• @komarovd95 made their first contribution in PR #15789
• @langpingxue made their first contribution in PR #15635
• @OrionCodeDev made their first contribution in PR #16070
• @sbinnee made their first contribution in PR #16078
• @JetoPistola made their first contribution in PR #16106
• @gvioss made their first contribution in PR #16093
• @pale-aura made their first contribution in PR #16084
• @tanvithakur94 made their first contribution in PR #16041
• @li-boxuan made their first contribution in PR #16044
• @1stprinciple made their first contribution in PR #15938
• @raghav-stripe made their first contribution in PR #16137
• @steve-gore-snapdocs made their first contribution in PR #16149

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.79.0-stable

pip install litellm==1.79.0

Major Changes​

• Cohere models will now be routed to Cohere v2 API by default - PR #15722

Key Highlights​

• Search APIs - Native /v1/search endpoint with support for Perplexity, Tavily, Parallel AI, Exa AI, DataforSEO, and Google PSE with cost tracking
• Vector Stores - Vertex AI Search API integration as vector store through LiteLLM with passthrough endpoint support
• Guardrails Expansion - Apply guardrails across Responses API, Image Gen, Text completions, Audio transcriptions, Audio Speech, Rerank, and Anthropic Messages API via unified apply_guardrails function
• New Guardrail Providers - Gray Swan, Dynamo AI, IBM Guardrails, Lasso Security v3, and Bedrock Guardrail apply_guardrail endpoint support
• Video Generation API - Native support for OpenAI Sora-2 and Azure Sora-2 (Pro, Pro-High-Res) with cost tracking and logging support
• Azure AI Speech (TTS) - Native Azure AI Speech integration with cost tracking for standard and HD voices

New Models / Updated Models​

New Model Support​

Features​

• Anthropic

  • Fix cache_control incorrectly applied to all content items instead of last item only - PR #15699
  • Forward anthropic-beta headers to Bedrock, VertexAI - PR #15700
  • Change max_tokens value to match max_output_tokens for claude sonnet - PR #15715

• Bedrock

  • Add AWS us-gov-west-1 Claude 3.7 Sonnet costs - PR #15775
  • Fix the date for sonnet 3.7 in govcloud - PR #15800
  • Use proper bedrock model name in health check - PR #15808
  • Support for embeddings_by_type Response Format in Bedrock Cohere Embed v1 - PR #15707
  • Add titan image generations with cost tracking - PR #15916

• Gemini

  • Add imageConfig parameter for gemini-2.5-flash-image - PR #15530
  • Replace deprecated gemini-1.5-pro-preview-0514 - PR #15852
  • Update vertex ai gemini costs - PR #15911

• Ollama

  • Set 'think' to False when reasoning effort is minimal/none/disable - PR #15763
  • Handle parsing ollama chunk error - PR #15717

• Vertex AI

  • Add mistral medium 3 and Codestral 2 on vertex - PR #15887

• Databricks

  • Allow prompt caching to be used for Anthropic Claude on Databricks - PR #15801

• Azure

  • Add Azure AVA TTS integration - PR #15749
  • Add Azure AVA (Speech AI) Cost Tracking - PR #15754
  • Azure AI Speech - Ensure voice is mapped from request body to SSML body, allow sending role and style - PR #15810
  • Add Azure support for video generation functionality (Sora-2) - PR #15901

• OpenAI

  • OpenAI videos refactoring - PR #15900

• General

  • Read from custom-llm-provider header - PR #15528

LLM API Endpoints​

Features​

• Responses API

  • Add gpt 4.1 pricing for response endpoint - PR #15593
  • Fix Incorrect status value in responses api with gemini - PR #15753
  • Simplify reasoning item handling for gpt-5-codex - PR #15815
  • ErrorEvent ValidationError when OpenAI Responses API returns nested error structure - PR #15804
  • Fix reasoning item ID auto-generation causing encrypted content verification errors - PR #15782
  • Support tags in metadata - PR #15867
  • Security: prevent User A from retrieving User B's response, if response.id is leaked - PR #15757

• Batch API

  • Add pre and post call for list batches - PR #15673
  • Add function responsible to call precall - PR #15636
  • Fix "User default_user_id does not have access to the object" when object not in db - PR #15873

• OCR API

  • Add Azure AI - OCR to docs - PR #15768
  • Add mode + Health check support for OCR models - PR #15767

• Search API

  • Add def search() APIs for Web Search - Perplexity API - PR #15769
  • Add Tavily Search API - PR #15770
  • Add Parallel AI - Search API - PR #15772
  • Add EXA AI Search API to LiteLLM - PR #15774
  • Add /search endpoint on LiteLLM Gateway - PR #15780
  • Add DataforSEO Search API - PR #15817
  • Add Google PSE Search Provider - PR #15816
  • Add cost tracking for Search API requests - Google PSE, Tavily, Parallel AI, Exa AI - PR #15821
  • Backend: Allow storing configured Search APIs in DB - PR #15862
  • Exa Search API - ensure request params are sent to Exa AI - PR #15855

• Vector Stores

  • Support Vertex AI Search API as vector store through LiteLLM - PR #15781
  • Azure AI - Search Vector Stores - PR #15873
  • VertexAI Search Vector Store - Passthrough endpoint support + Vector store search Cost tracking support - PR #15824
  • Don't raise error if managed object is not found - PR #15873
  • Show config.yaml vector stores on UI - PR #15873
  • Cost tracking for search spend - PR #15859

• Images API

  • Pass user-defined headers and extra_headers to image-edit calls - PR #15811

• Video Generation API

  • Add Azure support for video generation functionality (Sora-2, Sora-2-Pro, Sora-2-Pro-High-Res) - PR #15901
  • OpenAI video generation refactoring (Sora-2) - PR #15900

• Bedrock /invoke

  • Fix: Hooks broken on /bedrock passthrough due to missing metadata - PR #15849

• Realtime API

  • Fix: OpenAI Realtime API integration fails due to websockets.exceptions.PayloadTooBig error - PR #15751

Management Endpoints / UI​

Features​

• Passthrough

  • Set auth on passthrough endpoints, on the UI - PR #15778
  • Fix pass-through endpoint budget enforcement bug - PR #15805

• Organizations

  • Allow org admins to create teams on UI - PR #15924

• Search Tools

  • UI - Search Tools, allow adding search tools on UI + testing search - PR #15871
  • UI - Add logos for search providers - PR #15872

• General

  • Fix routing for custom server root path - PR #15701

Logging / Guardrail / Prompt Management Integrations​

Features​

• OpenTelemetry

  • Fix OpenTelemetry Logging functionality - PR #15645
  • Fix issue where headers were not being split correctly - PR #15916

• Sentry

  • Add SENTRY_ENVIRONMENT configuration for Sentry integration - PR #15760

• Helicone

  • Fix JSON serialization error in Helicone logging by removing OpenTelemetry span from metadata - PR #15728

• MLFlow

  • Fix MLFlow tags - split request_tags into (key, val) if request_tag has colon - PR #15914

• General

  • Rename configured_cold_storage_logger to cold_storage_custom_logger - PR #15798

Guardrails​

• Gray Swan

  • Add GraySwan Guardrails support - PR #15756
  • Rename GraySwan to Gray Swan - PR #15771

• Dynamo AI

  • New Guardrail - Dynamo AI Guardrail - PR #15920

• IBM Guardrails

  • IBM Guardrails integration - PR #15924

• Lasso Security

  • Add v3 API Support - PR #12452
  • Fixed lasso import config, redis cluster hash tags for test keys - PR #15917

• Bedrock Guardrails

  • Implement Bedrock Guardrail apply_guardrail endpoint support - PR #15892

• General

  • Guardrails - Responses API, Image Gen, Text completions, Audio transcriptions, Audio Speech, Rerank, Anthropic Messages API support via the unified apply_guardrails function - PR #15706

Spend Tracking, Budgets and Rate Limiting​

• Rate Limiting
  • Support absolute RPM/TPM in priority_reservation - PR #15813
  • Org level tpm/rpm limits + Team tpm/rpm validation when assigned to org - PR #15549

MCP Gateway​

• OAuth
  • Auth Header Fix for MCP Tool Call - PR #15736
  • Add response_type + PKCE parameters to OAuth authorization endpoint - PR #15720

Performance / Loadbalancing / Reliability improvements​

• Database

  • Minimize the occurrence of deadlocks - PR #15281

• Redis

  • Apply max_connections configuration to Redis async client - PR #15797

• Caching

  • Add documentation for enable_caching_on_provider_specific_optional_params setting - PR #15885

Documentation Updates​

• Provider Documentation
  • Update worker recommendation - PR #15702
  • Fix the wrong request body in json mode doc - PR #15729
  • Add details in docs - PR #15721
  • Add responses api on openai docs - PR #15866
  • Add OpenAI responses api - PR #15868

New Contributors​

• @tlecomte made their first contribution in PR #15528
• @tomhaynes made their first contribution in PR #15645
• @talalryz made their first contribution in PR #15720
• @1vinodsingh1 made their first contribution in PR #15736
• @nuernber made their first contribution in PR #15775
• @Thomas-Mildner made their first contribution in PR #15760
• @javiergarciapleo made their first contribution in PR #15721
• @lshgdut made their first contribution in PR #15717
• @kk-wangjifeng made their first contribution in PR #15530
• @anthonyivn2 made their first contribution in PR #15801
• @romanglo made their first contribution in PR #15707
• @mythral made their first contribution in PR #15859
• @mubashirosmani made their first contribution in PR #15866
• @CAFxX made their first contribution in PR #15281
• @reflection made their first contribution in PR #15914
• @shadielfares made their first contribution in PR #15917

PR Count Summary​

10/26/2025​

• New Models / Updated Models: 20
• LLM API Endpoints: 29
• Management Endpoints / UI: 5
• Logging / Guardrail / Prompt Management Integrations: 10
• Spend Tracking, Budgets and Rate Limiting: 2
• MCP Gateway: 2
• Performance / Loadbalancing / Reliability improvements: 3
• Documentation Updates: 5

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.78.5-stable

pip install litellm==1.78.5

Key Highlights​

• Native OCR Endpoints - Native /v1/ocr endpoint support with cost tracking for Mistral OCR and Azure AI OCR
• Global Vendor Discounts - Specify global vendor discount percentages for accurate cost tracking and reporting
• Team Spending Reports - Team admins can now export detailed spending reports for their teams
• Claude Haiku 4.5 - Day 0 support for Claude Haiku 4.5 across Bedrock, Vertex AI, and OpenRouter with 200K context window
• GPT-5-Codex - Support for GPT-5-Codex via Responses API on OpenAI and Azure
• Performance Improvements - Major router optimizations: O(1) model lookups, 10-100x faster shallow copy, 30-40% faster timing calls, and O(n) to O(1) hash generation

New Models / Updated Models​

New Model Support​

Features​

• OpenAI

  • GPT-5 return reasoning content via /chat/completions + GPT-5-Codex working on Claude Code - PR #15441

• Anthropic

  • Reduce claude-4-sonnet max_output_tokens to 64k - PR #15409
  • Added claude-haiku-4.5 - PR #15579
  • Add support for thinking blocks and redacted thinking blocks in Anthropic v1/messages API - PR #15501

• Bedrock

  • Add anthropic.claude-haiku-4-5-20251001-v1:0 on Bedrock, VertexAI - PR #15581
  • Add Claude Haiku 4.5 support for Bedrock global and US regions - PR #15650
  • Add Claude Haiku 4.5 support for Bedrock Other regions - PR #15653
  • Add JP Cross-Region Inference jp.anthropic.claude-haiku-4-5-20251001 - PR #15598
  • Fix: bedrock-pricing-geo-inregion-cross-region / add Global Cross-Region Inference - PR #15685
  • Fix: Support us-gov prefix for AWS GovCloud Bedrock models - PR #15626
  • Fix GPT-OSS in Bedrock now supports streaming. Revert fake streaming - PR #15668

• Gemini

  • Feat(pricing): Add Gemini 2.5 Flash Image (Nano Banana) in GA - PR #15557
  • Fix: Gemini 2.5 Flash Image should not have supports_web_search=true - PR #15642
  • Remove penalty params as supported params for gemini preview model - PR #15503

• Ollama

  • Fix(ollama/chat): correctly map reasoning_effort to think in requests - PR #15465

• OpenRouter

  • Add anthropic/claude-sonnet-4.5 to OpenRouter cost map - PR #15472
  • Prompt caching for anthropic models with OpenRouter - PR #15535
  • Get completion cost directly from OpenRouter - PR #15448
  • Fix OpenRouter Claude Opus 4 model naming - PR #15495

• CometAPI

  • Fix(cometapi): improve CometAPI provider support (embeddings, image generation, docs) - PR #15591

• Lemonade

  • Adding new models to the lemonade provider - PR #15554

• Watson X

  • Fix (pricing): Fix pricing for watsonx model family for various models - PR #15670

• Vercel AI Gateway

  • Add glm-4.6 model to pricing configuration - PR #15679

• Vertex AI

  • Add Vertex AI Discovery Engine Rerank Support - PR #15532

Bug Fixes​

• Anthropic

  • Fix: Pricing for Claude Sonnet 4.5 in US regions is 10x too high - PR #15374

• OpenRouter

  • Change gpt-5-codex support in model_price json - PR #15540

• Bedrock

  • Fix filtering headers for signature calcs - PR #15590

• General

  • Add native reasoning and streaming support flag for gpt-5-codex - PR #15569

LLM API Endpoints​

Features​

• Responses API

  • Responses API - enable calling anthropic/gemini models in Responses API streaming in openai ruby sdk + DB - sanity check pending migrations before startup - PR #15432
  • Add support for responses mode in health check - PR #15658

• OCR API

  • Feat: Add native litellm.ocr() functions - PR #15567
  • Feat: Add /ocr route on LiteLLM AI Gateway - Adds support for native Mistral OCR calling - PR #15571
  • Feat: Add Azure AI Mistral OCR Integration - PR #15572
  • Feat: Native /ocr endpoint support - PR #15573
  • Feat: Add Cost Tracking for /ocr endpoints - PR #15678

• /generateContent

  • Fix: GEMINI - CLI - add google_routes to llm_api_routes - PR #15500
  • Fix Pydantic validation error for citationMetadata.citationSources in Google GenAI responses - PR #15592

• Images API

  • Fix: Dall-e-2 for Image Edits API - PR #15604

• Bedrock Passthrough

  • Feat: Allow calling /invoke, /converse routes through AI Gateway + models on config.yaml - PR #15618

Bugs​

• General
  • Fix: Convert object to a correct type - PR #15634
  • Bug Fix: Tags as metadata dicts were raising exceptions - PR #15625
  • Add type hint to function_to_dict and fix typo - PR #15580

Management Endpoints / UI​

Features​

• Virtual Keys

  • Docs: Key Rotations - PR #15455
  • Fix: UI - Key Max Budget Removal Error Fix - PR #15672
  • litellm_Key Settings Max Budget Removal Error Fix - PR #15669

• Teams

  • Feat: Allow Team Admins to export a report of the team spending - PR #15542

• Passthrough

  • Feat: Passthrough - allow admin to give access to specific passthrough endpoints - PR #15401

• SCIM v2

  • Feat(scim_v2.py): if group.id doesn't exist, use external id + Passthrough - ensure updates and deletions persist across instances - PR #15276

• SSO

  • Feat: UI SSO - Add PKCE for OKTA SSO - PR #15608
  • Fix: Separate OAuth M2M authentication from UI SSO + Handle Introspection endpoint for Oauth2 - PR #15667
  • Fix/entraid app roles jwt claim clean - PR #15583

Logging / Guardrail / Prompt Management Integrations​

Guardrails​

• General

  • Fix apply_guardrail endpoint returning raw string instead of ApplyGuardrailResponse - PR #15436
  • Fix: Ensure guardrail memory sync after database updates - PR #15633
  • Feat: add guardrail for image generation - PR #15619
  • Feat: Add Guardrails for /v1/messages and /v1/responses API - PR #15686

• Pillar Security

  • Feature: update pillar security integration to support no persistence mode in litellm proxy - PR #15599

Prompt Management​

• General
  • Small fix code snippet custom_prompt_management.md - PR #15544

Spend Tracking, Budgets and Rate Limiting​

• Cost Tracking

  • Feat: Cost Tracking - specify a global vendor discount for costs - PR #15546
  • Feat: UI - Allow setting Provider Discounts on UI - PR #15550

• Budgets

  • Fix: improve budget clarity - PR #15682

Performance / Loadbalancing / Reliability improvements​

• Router Optimizations

  • Perf(router): use shallow copy instead of deepcopy for model aliases - 10-100x faster than deepcopy on nested dict structures - PR #15576
  • Perf(router): optimize string concatenation in hash generation - Improves time complexity from O(n²) to O(n) - PR #15575
  • Perf(router): optimize model lookups with O(1) data structures - Replace O(n) scans with index map lookups - PR #15578
  • Perf(router): optimize model lookups with O(1) index maps - Use model_id_to_deployment_index_map and model_name_to_deployment_indices for instant lookups - PR #15574
  • Perf(router): optimize timing functions in completion hot path - Use time.perf_counter() for duration measurements and time.monotonic() for timeout calculations, providing 30-40% faster timing calls - PR #15617

• SSL/TLS Performance

  • Feat(ssl): add configurable ECDH curve for TLS performance - Configure via ssl_ecdh_curve setting to disable PQC on OpenSSL 3.x for better performance - PR #15617

• Token Counter

  • Fix(token-counter): extract model_info from deployment for custom_tokenizer - PR #15680

• Performance Metrics

  • Add: perf summary - PR #15458

• CI/CD

  • Fix: CI/CD - Missing env key & Linter type error - PR #15606

Documentation Updates​

• Provider Documentation

  • Litellm docs 10 11 2025 - PR #15457
  • Docs: add ecs deployment guide - PR #15468
  • Docs: Update benchmark results - PR #15461
  • Fix: add missing context to benchmark docs - PR #15688

• General

  • Fixed a few typos - PR #15267

New Contributors​

• @jlan-nl made their first contribution in PR #15374
• @ImadSaddik made their first contribution in PR #15267
• @huangyafei made their first contribution in PR #15472
• @mubashir1osmani made their first contribution in PR #15468
• @kowyo made their first contribution in PR #15465
• @dhruvyad made their first contribution in PR #15448
• @davizucon made their first contribution in PR #15544
• @FelipeRodriguesGare made their first contribution in PR #15540
• @ndrsfel made their first contribution in PR #15557
• @shinharaguchi made their first contribution in PR #15598
• @TensorNull made their first contribution in PR #15591
• @TeddyAmkie made their first contribution in PR #15583
• @aniketmaurya made their first contribution in PR #15580
• @eddierichter-amd made their first contribution in PR #15554
• @konekohana made their first contribution in PR #15535
• @Classic298 made their first contribution in PR #15495
• @afogel made their first contribution in PR #15599
• @orolega made their first contribution in PR #15633
• @LucasSugi made their first contribution in PR #15634
• @uc4w6c made their first contribution in PR #15619
• @Sameerlite made their first contribution in PR #15658
• @yuneng-jiang made their first contribution in PR #15672
• @Nikro made their first contribution in PR #15680

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.78.0-stable

pip install litellm==1.78.0.post1

Key Highlights​

• MCP Gateway - Control Tool Access by Team, Key - Control MCP tool access by team/key.
• Performance Improvements - 70% Lower p99 Latency
• GPT-5 Pro & GPT-Image-1-Mini - Day 0 support for OpenAI's GPT-5 Pro (400K context) and gpt-image-1-mini image generation
• EnkryptAI Guardrails - New guardrail integration for content moderation
• Tag-Based Budgets - Support for setting budgets based on request tags

MCP Gateway - Control Tool Access by Team, Key​

Proxy admins can now control MCP tool access by team or key. This makes it easy to grant different teams selective access to tools from the same MCP server.

For example, you can now give your Engineering team access to list_repositories, create_issue, and search_code tools, while Sales only gets search_code and close_issue tools.

This makes it easier for Proxy Admins to govern MCP Tool Access.

Get Started

Performance - 70% Lower p99 Latency​

This release cuts p99 latency by 70% on LiteLLM AI Gateway, making it even better for low-latency use cases.

These gains come from two key enhancements:

Reliable Sessions

Added support for shared sessions with aiohttp. The shared_session parameter is now consistently used across all calls, enabling connection pooling.

Faster Routing

A new model_name_to_deployment_indices hash map replaces O(n) list scans in _get_all_deployments() with O(1) hash lookups, boosting routing performance and scalability.

As a result, performance improved across all latency percentiles:

• Median latency: 110 ms → 100 ms (−9.1%)
• p95 latency: 440 ms → 150 ms (−65.9%)
• p99 latency: 810 ms → 240 ms (−70.4%)
• Average latency: 310 ms → 111.73 ms (−64.0%)

Test Setup​

Locust

• Concurrent users: 1,000
• Ramp-up: 500

System Specs

• Database was used
• CPU: 4 vCPUs
• Memory: 8 GB RAM
• LiteLLM Workers: 4
• Instances: 4

Configuration (config.yaml)

View the complete configuration: gist.github.com/AlexsanderHamir/config.yaml

Load Script (no_cache_hits.py)

View the complete load testing script: gist.github.com/AlexsanderHamir/no_cache_hits.py

New Models / Updated Models​

New Model Support​