Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple-E Go test file name suffix does not contain _test.go #224

Open
hitesh-1997 opened this issue Apr 20, 2024 · 0 comments · May be fixed by #225
Open

Multiple-E Go test file name suffix does not contain _test.go #224

hitesh-1997 opened this issue Apr 20, 2024 · 0 comments · May be fixed by #225

Comments

@hitesh-1997
Copy link

hitesh-1997 commented Apr 20, 2024

Hi Team,
I was using the bigcode-evaluation-harness to evaluate generation for go on Multiple-E dataset and found that, all the evaluation had output ? command-line-arguments [no test files] although status_code = 0.
On debugging further, it looks like we set self.language here instead of prompt_name['langugage'] in the problem dict to process execution downstream, and when language is checked in evaluators here, it is appended without _test.go suffix leading to non detecting any test files.

To make it easy to repro this, I have added a video below which evaluate one go generation test case (used deepseek coder to generate this)

generations_go_example.json

[
	[
        "package strlen_test\n\nimport (\n    \"testing\"\n    \"fmt\"\n)\n\n// Return length of given string\n// >>> strlen(\"\")\n// 0\n// >>> strlen(\"abc\")\n// 3\nfunc strlen(myString string) int {\n    return len(myString)\n}\n"
    ]
]
bigcode_go_test_file_name_issue.mp4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant