Skip to content

Commit

Permalink
Update MultiPL-E to v3 prompts
Browse files Browse the repository at this point in the history
I've put together a new MultiPL-E release, v3 and this updates the harness to point to the new prompts.

There are fixes to old prompts as well as support for new languages, both described on the MultiPL-E dataset page. The nature of fixes is that some PLs have their scores go up a little. I've tested with with StarCoder2-15B (using VLLM) and will continue testing other models.
  • Loading branch information
arjunguha authored Jul 12, 2024
1 parent 334efb7 commit 8eed623
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion bigcode_eval/tasks/multiple.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ class GeneralMultiPLE(Task):

DATASET_PATH = "nuprl/MultiPL-E"
DATASET_NAME = None
DATASET_REVISION = "d23b094346c5dbda1080a74bb2a24c18adbf7409"
DATASET_REVISION = "8a4cb75204eb3d5855a81778db6b95bfc80c9136"

def __init__(self, language):
self.language = language
Expand Down

0 comments on commit 8eed623

Please sign in to comment.