The /chat endpoint is currently using FastAPI's background task to execute the search process and LLM calls in order to get around Turn.io's 20 second limitation. This is not a robust solution since FastAPI's background task does not guarantee execution order.
Suitable solutions for productions are:
- Websockets (but Turn.io does not support this currently)
- Proper message queue system like Celery, RMQ, etc.
- Preventing users from requesting another message until the current background task finishes.
NB: This does not affect the frontend UI since there is no way to call the /chat endpoint with a WA ID and Turn API key from the UI.
The
/chatendpoint is currently using FastAPI's background task to execute the search process and LLM calls in order to get around Turn.io's 20 second limitation. This is not a robust solution since FastAPI's background task does not guarantee execution order.Suitable solutions for productions are:
NB: This does not affect the frontend UI since there is no way to call the
/chatendpoint with a WA ID and Turn API key from the UI.