Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
I've put together a new MultiPL-E release, v3 and this updates the harness to point to the new prompts. There are fixes to old prompts as well as support for new languages, both described on the MultiPL-E dataset page. The nature of fixes is that some PLs have their scores go up a little. I've tested with with StarCoder2-15B (using VLLM) and will continue testing other models.
- Loading branch information