Prerequisites
Feature Description
I would like webui to be robust to connection interruptions. If I am streaming a response to my phone and the network changes, the response is lost immediately, which is not a good user experience.
Motivation
On the ChatGPT app, it is nice to start something, close the app, and come back later when the response is done.
I would enjoy having a similar level of polish in webui.
Possible Implementation
I can think of two approaches that make sense:
- We could fix the trailing assistant prefill API so that it works with reasoning models.
This is possibly the least invasive approach, although it doesn't allow background chats to work if the client pauses in the background, like it does on iOS.
Essentially: when the client detects the connection is interrupted, it would send the chat history plus the incomplete response back to the server and say "continue" automatically. There is no technical reason that this couldn't work that I'm aware of. The LLM does not care whether it is naturally generating tokens in the middle of reasoning or not, or whether it has simply been given a partially-prefilled response and told to continue. It cannot know. It's just a matter of hooking up the template machinery correctly so that a client can send either a partial reasoning_content or a complete reasoning_content and a partial response text and have the server resume.
webui already supports a manual continue button for non-reasoning models only, and this restriction is because of limitations in the current llama-server implementation.
- The other approach is to allow the server to support SSE resumption.
The underlying chat stream is an SSE stream, even if the POST prevents the client from using EventSource. The server could optionally accept a random UUID as a session_id in the chat request to recognize this extension of the chat API and be able to later identify this generation request. When provided with a session_id, the server would include a sequentially autoincrementing integer in the id: field for each SSE event.
Normally, the server stops generating a response the instant the client disconnects. When there is a session_id, the server would not give up on generating the response just because the client disconnects. This maintains compatibility with the existing expectation, because clients would have to opt into this new behavior. Instead of quitting, the server would keep generating into a session_id-specific buffer.
The client would reconnect by providing both the session ID and event ID, and the server could send down what it missed. When reconnecting, the client would not provide the conversational context.
Realistically, even a long response is going to be a few kilobytes at most, so the server could maintain an in-memory buffer of the entire session response until the response is complete and acknowledged by the client, and I struggle to imagine anyone ever noticing the memory usage involved. This would simplify being able to forward the rest of the response after the last event ID. Or we could even do away with event IDs entirely and just re-send the entire response as it exists. Of course, if the server is not done generating, it would continue streaming new chunks as they are available. Once a response is complete and the client disconnects, the server would delete the session memory immediately.
There would ideally also be a new endpoint that is used to cancel generation for a particular session ID, since disconnecting would no longer achieve that for these upgraded connections, and the user might want to click the stop button, or they might edit and re-send a previous message while a response was still generating.
One could also imagine having some timeout knobs: one timeout that deletes a session if the client has been gone for a certain number of seconds, and another timeout that stops generation after the client has been gone for a certain number of seconds. I could go into more detail about the use cases for these if needed, but they are just nice-to-haves.
Since random UUIDs have 122 bits of randomness, collisions are statistically impossible, which makes them perfect for use as a client-generated secure session ID that no one else can guess, and they would be ephemeral to each response in a conversation.
Both options have their pros and cons, and there could even be benefits to implementing both. There are potential use cases for assistant prefill on reasoning models, independent of webui, but webui could fall back to assistant prefill if the server crashed and restarted (and therefore has no memory of the session that was in progress). This would lend even greater robustness to the client.
It is also worth noting that with client-side tools like we have today, the server would still only generate one turn, so a response that requires calling a tool would block on that tool call until the client returns, provides the result, and then calls for a new response, but this trade off is fine. In the future, if we have server side tool calling, then ideally the session would be continuous across responses and tool calls, only ending when a tool is called that the server cannot handle (a client side tool) or when the model hands control back to the user to wait for the next user message.
EDIT:
An LLM has convinced me that assistant prefill is not always the same as the original stream would have been, so that is one thing to note for the above proposal, which was completely written by hand from my own thoughts, before I checked with an LLM on that detail.
(Details are in this collapsed block to be less annoying.)
Because the server is not resuming from the model’s KV/cache state. It is rebuilding state from text.
Those are not always equivalent. Main reasons:
- The prompt may not be reconstructed identically.
- In current llama.cpp, assistant-prefill is a special path, not “replay the exact same structured turn.”
- For reasoning models, reasoning_content, handling, tool-call parsing, and generation prompt handling can change what tokens the model sees.
- Parsing/rendering can be lossy.
- If the original stream was parsed into reasoning_content and content, then later re-rendered through a template, that is not guaranteed to reproduce the exact original token sequence.
- Even whitespace or delimiter differences matter.
- The interruption can happen between semantic boundaries.
- The client may only have received text up to some chunk boundary, not necessarily the exact full parser state the server had internally at that moment.
- Sampler state is not just “visible text so far.”
- Some samplers depend only on prior tokens, so replay can match.
- But if anything about tokenization/prompt reconstruction changes, repetition penalties, DRY logic, grammar/tool-call state, or reasoning parser state can diverge.
- Floating-point / batching effects can differ.
- llama.cpp already documents that prompt reuse and different batching are not guaranteed bit-for-bit identical.
- Recomputing from scratch can produce slightly different logits even before sampling.
So, in the ideal case:
- same exact token history
- same template rendering
- same parser mode
- same sampler config
- same seed
- same numerical path
then yes, the state should effectively match.
But in practice, especially for reasoning models in the current server, that “same exact token history” condition is exactly what is not guaranteed. That is why session resume is exact, while assistant-prefill continuation is only potentially exact.
So, assistant prefill continuation is probably more of a "best effort" way to handle things, and I would definitely favor the continuous approach with SSE resumption, but I don't think assistant prefill continuation would be a bad option.
Prerequisites
Feature Description
I would like
webuito be robust to connection interruptions. If I am streaming a response to my phone and the network changes, the response is lost immediately, which is not a good user experience.Motivation
On the ChatGPT app, it is nice to start something, close the app, and come back later when the response is done.
I would enjoy having a similar level of polish in
webui.Possible Implementation
I can think of two approaches that make sense:
This is possibly the least invasive approach, although it doesn't allow background chats to work if the client pauses in the background, like it does on iOS.
Essentially: when the client detects the connection is interrupted, it would send the chat history plus the incomplete response back to the server and say "continue" automatically. There is no technical reason that this couldn't work that I'm aware of. The LLM does not care whether it is naturally generating tokens in the middle of reasoning or not, or whether it has simply been given a partially-prefilled response and told to continue. It cannot know. It's just a matter of hooking up the template machinery correctly so that a client can send either a partial reasoning_content or a complete reasoning_content and a partial response text and have the server resume.
webuialready supports a manual continue button for non-reasoning models only, and this restriction is because of limitations in the currentllama-serverimplementation.The underlying chat stream is an SSE stream, even if the
POSTprevents the client from usingEventSource. The server could optionally accept a random UUID as asession_idin the chat request to recognize this extension of the chat API and be able to later identify this generation request. When provided with asession_id, the server would include a sequentially autoincrementing integer in theid:field for each SSE event.Normally, the server stops generating a response the instant the client disconnects. When there is a
session_id, the server would not give up on generating the response just because the client disconnects. This maintains compatibility with the existing expectation, because clients would have to opt into this new behavior. Instead of quitting, the server would keep generating into asession_id-specific buffer.The client would reconnect by providing both the session ID and event ID, and the server could send down what it missed. When reconnecting, the client would not provide the conversational context.
Realistically, even a long response is going to be a few kilobytes at most, so the server could maintain an in-memory buffer of the entire session response until the response is complete and acknowledged by the client, and I struggle to imagine anyone ever noticing the memory usage involved. This would simplify being able to forward the rest of the response after the last event ID. Or we could even do away with event IDs entirely and just re-send the entire response as it exists. Of course, if the server is not done generating, it would continue streaming new chunks as they are available. Once a response is complete and the client disconnects, the server would delete the session memory immediately.
There would ideally also be a new endpoint that is used to cancel generation for a particular session ID, since disconnecting would no longer achieve that for these upgraded connections, and the user might want to click the stop button, or they might edit and re-send a previous message while a response was still generating.
One could also imagine having some timeout knobs: one timeout that deletes a session if the client has been gone for a certain number of seconds, and another timeout that stops generation after the client has been gone for a certain number of seconds. I could go into more detail about the use cases for these if needed, but they are just nice-to-haves.
Since random UUIDs have 122 bits of randomness, collisions are statistically impossible, which makes them perfect for use as a client-generated secure session ID that no one else can guess, and they would be ephemeral to each response in a conversation.
Both options have their pros and cons, and there could even be benefits to implementing both. There are potential use cases for assistant prefill on reasoning models, independent of
webui, butwebuicould fall back to assistant prefill if the server crashed and restarted (and therefore has no memory of the session that was in progress). This would lend even greater robustness to the client.It is also worth noting that with client-side tools like we have today, the server would still only generate one turn, so a response that requires calling a tool would block on that tool call until the client returns, provides the result, and then calls for a new response, but this trade off is fine. In the future, if we have server side tool calling, then ideally the session would be continuous across responses and tool calls, only ending when a tool is called that the server cannot handle (a client side tool) or when the model hands control back to the user to wait for the next user message.
EDIT:
An LLM has convinced me that assistant prefill is not always the same as the original stream would have been, so that is one thing to note for the above proposal, which was completely written by hand from my own thoughts, before I checked with an LLM on that detail.
(Details are in this collapsed block to be less annoying.)
So, assistant prefill continuation is probably more of a "best effort" way to handle things, and I would definitely favor the continuous approach with SSE resumption, but I don't think assistant prefill continuation would be a bad option.