LangChain affected by SSRF via image_url token counting in ChatOpenAI.get_num_tokens_from_messages

Server-Side Request Forgery (SSRF) in ChatOpenAI Image Token Counting

Summary

The ChatOpenAI.get_num_tokens_from_messages() method fetches arbitrary image_url values without validation when computing token counts for vision-enabled models. This allows attackers to trigger Server-Side Request Forgery (SSRF) attacks by providing malicious image URLs in user input.

Severity

Low - The vulnerability allows SSRF attacks but has limited impact due to:

• Responses are not returned to the attacker (blind SSRF)
• Default 5-second timeout limits resource exhaustion
• Non-image responses fail at PIL image parsing

Impact

An attacker who can control image URLs passed to get_num_tokens_from_messages() can:

• Trigger HTTP requests from the application server to arbitrary internal or external URLs
• Cause the server to access internal network resources (private IPs, cloud metadata endpoints)
• Cause minor resource consumption through image downloads (bounded by timeout)

Note: This vulnerability occurs during token counting, which may happen outside of model invocation (e.g., in logging, metrics, or token budgeting flows).

Details

The vulnerable code path:

1. get_num_tokens_from_messages() processes messages containing image_url content blocks
2. For images without detail: "low", it calls _url_to_size() to fetch the image and compute token counts
3. _url_to_size() performs httpx.get(image_source) on any URL without validation
4. Prior to the patch, there was no SSRF protection, size limits, or explicit timeout

File: libs/partners/openai/langchain_openai/chat_models/base.py

Patches

The vulnerability has been patched in langchain-openai==1.1.9 (requires langchain-core==1.2.11).

The patch adds:

1. SSRF validation using langchain_core._security._ssrf_protection.validate_safe_url() to block:
   • Private IP ranges (RFC 1918, loopback, link-local)
   • Cloud metadata endpoints (169.254.169.254, etc.)
   • Invalid URL schemes
2. Explicit size limits (50 MB maximum, matching OpenAI's payload limit)
3. Explicit timeout (5 seconds, same as httpx.get default)
4. Allow disabling image fetching via allow_fetching_images=False parameter

Workarounds

If you cannot upgrade immediately:

1. Sanitize input: Validate and filter image_url values before passing messages to token counting or model invocation
2. Use network controls: Implement egress filtering to prevent outbound requests to private IPs