You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As part of building reasoning-gym we are evaluating the performance of frontier AI systems on the datasets. This issue relates to the running evaluations for the o1 model. o1 evaluations have already been recorded for several of the datasets. You can find the remaining ones in the following google sheet
For consistency we have used the Open Router provider for all our evaluations and this will be required for this evaluation. We have a separate evaluation repository where we store the results of the model evals for each dataset. Please upload results here. If you would like to take a subset of these evaluations e.g (algorithmic) feel free to create a sub-issue. You should be able to use the following evaluation script to run your evaluations.
The text was updated successfully, but these errors were encountered:
As part of building reasoning-gym we are evaluating the performance of frontier AI systems on the datasets. This issue relates to the running evaluations for the o1 model. o1 evaluations have already been recorded for several of the datasets. You can find the remaining ones in the following google sheet
For consistency we have used the Open Router provider for all our evaluations and this will be required for this evaluation. We have a separate evaluation repository where we store the results of the model evals for each dataset. Please upload results here. If you would like to take a subset of these evaluations e.g (algorithmic) feel free to create a sub-issue. You should be able to use the following evaluation script to run your evaluations.
The text was updated successfully, but these errors were encountered: