Fix streaming + tool calling interleaving in OpenAI provider #2

New Issue

Open

opened 2026-09-18 01:12:20 -04:00 by assistant · 0 comments

assistant commented 2026-09-18 01:12:20 -04:00

Copy Link

Copy Source

Summary

The OpenAI provider's streaming loop incorrectly handles interleaved tool_call and content deltas, causing premature termination and missed chunks.

Root Cause

The streaming loop uses chunk.done to detect completion, but OpenAI streams at the choice level:

• finish_reason indicates when the LLM is done
• tool_calls and content deltas arrive independently and out of order

Current code:

1. Waits for entire tool_calls array before executing tools
2. Sets terminal = true on wrong condition (never properly set via finish_reason)
3. May stop consuming chunks after tool execution begins

Required Changes

• Detect proper termination via finish_reason === 'stop' | 'tool_calls'
• Accumulate tool_call deltas by index until all complete
• Execute tools only after all deltas for a given tool are received
• Continue consuming content deltas after tools execute
• Reset state correctly when switching between tool_calls and content

Acceptance Criteria

• Tools execute reliably even with interleaved deltas
• Content following tools is captured correctly
• Stream terminates at proper finish_reason, not on missing chunk.done
• No chunks are missed or duplicated during tool execution cycles

Related

• #1 Response streaming + tool calling ends prematurely or without follow-up text

## Summary The OpenAI provider's streaming loop incorrectly handles interleaved tool_call and content deltas, causing premature termination and missed chunks. ## Root Cause The streaming loop uses `chunk.done` to detect completion, but OpenAI streams at the choice level: - `finish_reason` indicates when the LLM is done - `tool_calls` and `content` deltas arrive independently and out of order Current code: 1. Waits for entire `tool_calls` array before executing tools 2. Sets `terminal = true` on wrong condition (never properly set via finish_reason) 3. May stop consuming chunks after tool execution begins ## Required Changes - Detect proper termination via `finish_reason === 'stop' | 'tool_calls'` - Accumulate tool_call deltas by index until all complete - Execute tools only after all deltas for a given tool are received - Continue consuming content deltas after tools execute - Reset state correctly when switching between tool_calls and content ## Acceptance Criteria - Tools execute reliably even with interleaved deltas - Content following tools is captured correctly - Stream terminates at proper finish_reason, not on missing `chunk.done` - No chunks are missed or duplicated during tool execution cycles ## Related - #1 Response streaming + tool calling ends prematurely or without follow-up text

ztimson added reference fix/streaming-tool-interleaving 2026-09-18 14:01:51 -04:00

Sign in to join this conversation.

fix/streaming-tool-interleaving

Branches Tags

Clear current reference

master

fix/streaming-tool-interleaving

Clear current reference

1.7.2

1.7.1

1.7.0

1.6.13

1.6.12

1.6.11

1.6.10

1.6.9

1.6.8

1.6.7

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.0

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.13

1.2.12

1.2.11

1.2.10

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.0

1.0.6

1.0.5

1.0.4

1.0.3

1.0.2

1.0.1

1.0.0

0.9.0

0.8.16

0.8.15

0.8.14

0.8.13

0.8.12

0.8.11

0.8.10

0.8.9

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.11

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7-rc.1

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.6

0.5.5

0.5.4

0.5.3

0.5.2

0.5.1

0.4.1

0.4.0

0.3.0

0.2.7

0.2.6

0.2.5

0.2.4

0.2.3

0.2.2

0.2.1

0.2.0

0.1.22

0.1.21

0.1.20

0.1.19

0.1.18

0.1.16

0.1.15

0.1.14

0.1.13

0.1.12

0.1.11

0.1.10

0.1.9

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

ztimson (Zakary Timson)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ztimson/ai-utils#2