If I stream the data directly using an adapter, it works perfectly. However, when I try to implement a custom TransformStream() to hook into the chunk buffer, decode the text, run regex, and re-encode it, the stream either locks up, drops text blocks chunks randomly, or causes a buffer overflow error on the client side:

TypeError: ReadableStream pipeline crashed or terminated unexpectedly mid-flight.

My Code Setup (app/api/chat/route.ts):

TypeScript

import { NextResponse } from 'next/server';

export async function POST(req: Request) {
  const { messages } = await req.json();

  // Hypothetical AI provider stream response
  const aiStream = await callAIProviderStream(messages); 

  // Custom transform stream to modify chunks mid-flight
  const transformStream = new TransformStream({
    transform(chunk, controller) {
      const text = new TextDecoder().decode(chunk);
      
      // Attempting to mutate text string before pushing to client
      const cleanedText = text.replace(/\[source:\s*\d+\]/g, ''); 
      
      controller.enqueue(new TextEncoder().encode(cleanedText));
    }
  });

  const mutatedStream = aiStream.pipeThrough(transformStream);

  return new NextResponse(mutatedStream, {
    headers: { 'Content-Type': 'text/event-stream' },
  });
}

What I have tried:

1. Verified that the raw aiStream works fine without the pipeThrough handler.

2. Tried utilizing string concatenation on a local variable inside the transform block, but it breaks the real-time nature of the stream by waiting for the full buffer to resolve.

How do I properly intercept, mutate, and forward raw text chunks in Next.js without fragmenting multi-byte characters or breaking the streaming interface?