Delay reduction

[amohajerani]

Based on the youtube demo video, the agent has a 4-5 second delay in conversing with the human. Are there any solutions to reduce this delay to half?

[clemlesne]

Hello, in the last versions delay has been redureced from the time I recorded the demo.

The delay mainly come from two things:

• The fact that Communication Services is sequential in the way it forwards the audio (it technically foarwards only the text, not the audio, and once the entire audio is transformed, after waited for a specified blank time) -- gpt-4o-realtime is a part of the solution (Implement GPT-4o-realtime API for inbound voice #317), but not yet perfect
• The LLM, more specifically the delay between API call and first sentence infered, can be long (as the sentences are sent one by one once they are made avalable), even longer if it hallucinate and returns empty answers (it happens regularly, and the applicatoipn retries the call)

From now, the only impactful thing you can do is the LLM part. This can be acheieve by a PTU on Azure or using a less smart model like gpt-4o-mini (selected by default on the latest versions). With a PTU on Azure OpenAI, you can divide by 2 the latency in some case.

[clemlesne]

I created an issue to enhance the doc: #328