Show current LLM model used

[FadhlanR]

[github-actions]

Host Test Results

    1 files  ±0      1 suites  ±0   20m 42s ⏱️ +17s
726 tests +1  724 ✔️ +1  2 💤 ±0  0 ❌ ±0 
731 runs  +1  729 ✔️ +1  2 💤 ±0  0 ❌ ±0 

Results for commit 7421df3. ± Comparison against base commit b650668.

♻️ This comment has been updated with latest results.

[IanCal]

For an initial list, I've used a filtered set here https://openrouter.ai/models?fmt=cards&order=newest&supported_parameters=tools

openai/gpt-4o

openai/gpt-4o-mini

anthropic/claude-3.5-sonnet

google/gemini-pro-1.5

We can try out other models and add them if they seem to be working, this gives a reasonable list of some popular models right now, which have

O1 will be good to add, but it is not currently generally available via openrouter and the newer google models are all marked experimental (which would be fine, but they make them free then heavily rate limit them so they quickly break).

[FadhlanR]

For an initial list, I've used a filtered set here https://openrouter.ai/models?fmt=cards&order=newest&supported_parameters=tools

openai/gpt-4o

openai/gpt-4o-mini

anthropic/claude-3.5-sonnet

google/gemini-pro-1.5

We can try out other models and add them if they seem to be working, this gives a reasonable list of some popular models right now, which have

O1 will be good to add, but it is not currently generally available via openrouter and the newer google models are all marked experimental (which would be fine, but they make them free then heavily rate limit them so they quickly break).

Currently, I obtain the list by calling openai.models.list(). With the list you provided, does this mean we no longer need to call openai.models.list()? Instead, we can set the list in the runtime common.

[IanCal]

Currently, I obtain the list by calling openai.models.list(). With the list you provided, does this mean we no longer need to call openai.models.list()? Instead, we can set the list in the runtime common.

Yep, this can be how we have the default list.

[FadhlanR]

Not sure if this is the right place to put DEFAULT_LLM and DEFAULT_LLM_LIST constants but I put them here to make it available for host and ai-bot.

[jurgenwerk]

I chose Claude but bot says:

[FadhlanR]

I chose Claude but bot says:

Were you trying it locally and running the latest ai-bot code?

[jurgenwerk]

I am running cs-7714-show-current-llm-model-used branch locally, with a restarted bot

[jurgenwerk]

For this particular model, I get the following error in the ai bot:

Request used 884 prompt tokens and 0
Attempt 1 failed: TypeError: response.error.includes is not a function
    at fetchGenerationCost (/Users/jurgen/development/boxel/packages/ai-bot/lib/ai-billing.ts:92:40)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async retry (/Users/jurgen/development/boxel/packages/runtime-common/utils.ts:7:22)
    at async saveUsageCost (/Users/jurgen/development/boxel/packages/ai-bot/lib/ai-billing.ts:22:21)
Attempt 2 failed: TypeError: response.error.includes is not a function
    at fetchGenerationCost (/Users/jurgen/development/boxel/packages/ai-bot/lib/ai-billing.ts:92:40)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async retry (/Users/jurgen/development/boxel/packages/runtime-common/utils.ts:7:22)
    at async saveUsageCost (/Users/jurgen/development/boxel/packages/ai-bot/lib/ai-billing.ts:22:21)
    ```

[FadhlanR]

@jurgenwerk I just tested it again in my local with two models and it worked. I couldn't reproduce it. It seems the value of response.error is not a string, I guess we need to capture the error structure so we can update how we handle the error in fetchGenerationCost.

[jurgenwerk]

I don't know what went wrong previously bot for the purposes of some other task I nuked all my local dbs and started anew in the meanwhile, and now this feature works for me 🥂 However I noticed a bug - switching doesn't seem to work ok:

[FadhlanR]

I don't know what went wrong previously bot for the purposes of some other task I nuked all my local dbs and started anew in the meanwhile, and now this feature works for me 🥂 However I noticed a bug - switching doesn't seem to work ok:

I noticed that too, it seems the ai bot will answer based on the previous conversation. If you ask the ai to forget the previous conversation, it will give you the right answer.

[lukemelia]

Should we be reading this from room state instead?

[FadhlanR]

We could read this from the room state, but I chose to read it from the event list to make testing easier. Additionally, this approach is consistent with how we read skills from the room.

[IanCal]

I tried with a local key and can see the requests going to the right providers even if they respond with what has been said before.

I found that when changing the room however the model didn't change in the dropdown

Screencast.from.09-01-25.15.49.19.webm

[IanCal]

I've not been able to get sonnet working either, not sure why but I think it's unrelated to this work

edit - found the issue. If the last message is from the assistant, claude breaks. I'll create a test and a fix in a different PR, it's a small change.

[FadhlanR]

I found that when changing the room however the model didn't change in the dropdown

Fixed! please check it again.