Created an Assistant for Activity Reports at Our Company

[sashokbg]

Hello, I have managed to create an experimental assistant that can interact with the APIs of a small program we made for the activity reports of our employees.

In the video bellow you can see how a typical interaction between a client and the assistant can go.

I encountered lot's of issues with hallucinated function calls and with wrong parameter inference, so a simple and elegant in my humble opinion solution was to ask the user for confirmation before each function call that modifies the database.

This ping-pong of conversation with validations feels like talking to a real person who tries to better understand you whenever you have expressed something in an ambiguous manner.
It also allows for the conversation to continue even when errors occur either on the assistant side or when a function call returns HTTP 400 / 500 errors.

You can see the full conversation here:

https://www.youtube.com/watch?v=vtFhx6U0GPo

Big thank you to the team working on the model !
Keep up the good work

[musab-mk]

That looks amazing! We're currently trying to fix the hallucination issue. So hopefully its gonna be fixed. Though, i must admit that its not an easy problem. Cause, there is a balance between guessing and asking for clarification. If it always asks for clarification, thats also another convenience problem. Maybe RLHF or DPO might be necessary for that kind of Human preferences.

[sashokbg]

Hello @musab-mk yes it looks like one of the big challenges of LLMs.
Either way functionary works very well and as shown in the POC it can actually be used to accomplish real tasks.
I will keep an eye on your project and continue testing whenever I have some free time.
Thanks once again

[musab-mk]

We've just retrained the model to reduce the issue that you mentioned. its released as 2.1 . BUT we still observed the same issue you mentioned. We're currently preparing data to just solve your issue :) We hope to finish it soon.

[musab-mk]

Are you utilizing non-function calls? For example chats without functions? Or is it mainly with functions?

[sashokbg]

I am sorry @musab-mk but I am not sure I understand the question :(

Here are the two main classes. It is very inspired by the examples you gave in the Readme:

https://github.com/sashokbg/functionary-crapp-bot/blob/main2/assistant.py

https://github.com/sashokbg/functionary-crapp-bot/blob/main2/server2.py

[musab-mk]

I meant, The model is also capable of chatting only without calling functions. Do you also see that its a use-case for you? Or are you using prompts that would result in code calls %100 of the time?
But i see now that you're mainly using function calls.

[musab-mk]

Also i want to say that we've not included more than two system messages on the training. So maybe its better to put all of the system stuff in single system message. Though if it works, then great! :)