Token count discrepancy between tiktoken and API response when messages contain tool calls

[yangchou19]

Description

I've encountered a significant discrepancy between token counts calculated by tiktoken locally and the actual token usage returned by the OpenAI API, specifically when messages contain tool calls.

Environment

• tiktoken version: 3.11
• Python version: 0.7.0
• Model: gpt-4.1-mini

Steps to Reproduce

1. Prepare a message payload that includes tool calls (function calling)
2. Calculate token count using tiktoken locally
3. Send the same payload to OpenAI API
4. Compare tiktoken's result with the usage field in API response

Expected Behavior

The token count calculated by tiktoken should be close to (or exactly match) the token usage reported by the API.

Actual Behavior

There is a large discrepancy between tiktoken's calculation and the API's reported token usage when tool calls are present in the message body.

tiktoken calculation

encoding = tiktoken.encoding_for_model("gpt-4")

Result

tiktoken_count = 47,194

API response shows: api_count = 140,384

Difference: -93,190

[Hitarth-Jeavio]

In my case, it is also not working with the gpt-5-mini model.
tiktoken reports 279,880 tokens, but the API returns 314,943 tokens.

[yangchou19]

Here's my message structure:

Message List:

[
  {"role": "system"},
  {"role": "user"},
  {"role": "assistant", "tool_calls": [...]},  // 1 tool call
  {"role": "tool"},
  {"role": "assistant", "tool_calls": [...]},  // 3 parallel tool calls
  {"role": "tool"},
  {"role": "tool"},
  {"role": "tool"}
]

Summary:

• 8 messages total
• 4 tool calls (1 single + 3 parallel)
• Each tool_calls object includes id, type, function (with name and arguments), and additional metadata fields

[milistu]

Hi @yangchou19!

I’m not sure how you’re counting tokens locally, but if you’re only counting message tokens, or if you’re adding function/tool calls as raw strings before token calculation, that would explain the discrepancy.

Function (tool) calling is processed internally in a specific way, and this affects how tokens are counted by the model.

There’s a relevant explanation in the OpenAI Cookbook that shows how to calculate tokens using the tiktoken library, including a section on token counting with function/tool calls:
https://developers.openai.com/cookbook/examples/how_to_count_tokens_with_tiktoken/#7-counting-tokens-for-chat-completions-with-tool-calls

Note that this guide refers to the chat.completions endpoint. The newer responses endpoint handles things slightly differently, so some variation is expected.

I hope this helps explain why you’re seeing different token counts locally versus when calling the API.