I'm building a real-time chat application using FastAPI WebSockets and LangGraph (LangChain) where the assistant can return both text and relevant financial charts/images retrieved from a Weaviate vector store.
The text response is streamed token-by-token beautifully, but the images are currently only sent after the full text response has finished (response_end), causing them to appear below the completed message instead of embedded inline where they are referenced in the analysis.
Here's the current flow (simplified):
User sends query → processed via LangGraph.
Text tokens are streamed via {"type": "token", "content": "..."}
When LangGraph finishes, I collect any returned images (base64 from Weaviate) in collected_images.
Send {"type": "response_end"}
Then loop through collected_images and send each via {"type": "image_b64", ...}
Result: All images appear at the bottom, even if the LLM says "As you can see in the chart above..."
Goal: Have images appear inline, exactly where they belong in the text (e.g., after a paragraph mentioning "performance trend"), ideally while text is still streaming or at least seamlessly embedded.