Context

I'm building an AI voice interface through Twilio. When the user speaks, it starts the pipeline of recognise speech -> generate answer -> turn to audio -> playback to user

Twiml for websocket bidirectional MediaStream with

I get the audio of what the user says through the connected websocket, and on the same connection I send audio payloads with the AI responses.

I'm sending audio payloads as Media Messages as described in docs (1 payload per sentence to be played back):

{
  "event": "media",
  "streamSid": "MZ18ad3ab5a668481ce02b83e7395059f0",
  "media": {
    "payload": "a3242sadfasfa423242... (the encoded audio for a sentence)"
  }
}

and I'm marking the sentence immediately to know when the playback has finished:

{ 
 "event": "mark",
 "streamSid": "MZ18ad3ab5a668481ce02b83e7395059f0",
 "mark": {
   "name": "*Sentence that will be played back*"
 }
}

If the user says something, I send an interrupt payload to stop the current playback, similar to how you would want your interlocutor to stop speaking if you start saying something:

{ 
 "event": "clear",
 "streamSid": "MZ18ad3ab5a668481ce02b83e7395059f0",
}

When you interrupt an in-progress playback, you get back the a mark event you sent for the sentence that was interrupted AND for the next sentences in the queue that did not get played back

The problem

There is no clear distinction between a mark emitted event when the playback was interrupted and a mark emitted event when the playback simply finished succesfully.

Another related issue is that if the playback of the sentence was interrupted, you can't know how much of the sentence the user has listened to. I need this information because it is relevant to the AI context for the next answers. Example ideal context:

User: Hello how are you?
AI: Hello! How can I hel*interrupt*
User: Good! Please tell me square root of pi!
Ai: The square root of ...

What I tried to solve the issue

I tried to compute sentence duration, to know when to expect the mark event:

1. Sentence is 10 seconds
2. I send the payload at 14:00
3. No other sentences are in the playback queue
4. Expect mark to come by 14:10
5. If mark comes earlier -> it was interrupted

However this is pretty flaky due to network delays

Does anybody know of a better solution for this issue?