You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As described in #1324 , I'm trying to create a scorer which just returns strings. I hence passed an empry list to metrics (since I don't want to reduce anything). I get this error from the resulting log when I try to open it:
at $TypeError: Cannot read properties of undefined (reading 'reducer')
at ResultsPanel (https://file+.vscode-resource.vscode-cdn.net/Users/max/Documents/git_repos/stories-cip/.venv/lib/python3.11/site-packages/inspect_ai/_view/www/dist/assets/index.js:60545:42)
at renderWithHooks (https://file+.vscode-resource.vscode-cdn.net/Users/max/Documents/git_repos/stories-cip/.venv/lib/python3.11/site-packages/inspect_ai/_view/www/dist/assets/index.js:3533:25)
at updateFunctionComponent (https://file+.vscode-resource.vscode-cdn.net/Users/max/Documents/git_repos/stories-cip/.venv/lib/python3.11/site-packages/inspect_ai/_view/www/dist/assets/index.js:5030:20)
at beginWork (https://file+.vscode-resource.vscode-cdn.net/Users/max/Documents/git_repos/stories-cip/.venv/lib/python3.11/site-packages/inspect_ai/_view/www/dist/assets/index.js:5685:18)
at performUnitOfWork (https://file+.vscode-resource.vscode-cdn.net/Users/max/Documents/git_repos/stories-cip/.venv/lib/python3.11/site-packages/inspect_ai/_view/www/dist/assets/index.js:8750:18)
at workLoopSync (https://file+.vscode-resource.vscode-cdn.net/Users/max/Documents/git_repos/stories-cip/.venv/lib/python3.11/site-packages/inspect_ai/_view/www/dist/assets/index.js:8649:41)
at renderRootSync (https://file+.vscode-resource.vscode-cdn.net/Users/max/Documents/git_repos/stories-cip/.venv/lib/python3.11/site-packages/inspect_ai/_view/www/dist/assets/index.js:8633:11)
at performWorkOnRoot (https://file+.vscode-resource.vscode-cdn.net/Users/max/Documents/git_repos/stories-cip/.venv/lib/python3.11/site-packages/inspect_ai/_view/www/dist/assets/index.js:8335:44)
at performWorkOnRootViaSchedulerTask (https://file+.vscode-resource.vscode-cdn.net/Users/max/Documents/git_repos/stories-cip/.venv/lib/python3.11/site-packages/inspect_ai/_view/www/dist/assets/index.js:9175:7)
at MessagePort.performWorkUntilDeadline (https://file+.vscode-resource.vscode-cdn.net/Users/max/Documents/git_repos/stories-cip/.venv/lib/python3.11/site-packages/inspect_ai/_view/www/dist/assets/index.js:191:50)
Here is my scorer, I believe this reproduces with any scorer if you make metrics empty:
@scorer(metrics=[])defscenario_parser_v0() ->Scorer:
"""Returns a scorer that returns a dictionary containing the model's question and answer. Fills out metadata with parse_error if the model's output does not contain both a <scenario> and a <question> tag."""asyncdefscenario_parser_v0(state: TaskState, target: Target) ->Score:
model_output=state.output.completionscenario_match=re.search(
r"<scenario>(.*?)</scenario>", model_output, re.DOTALL
)
question_match=re.search(
r"<question>(.*?)</question>", model_output, re.DOTALL
)
scenario=scenario_match.group(1).strip() ifscenario_matchelseNonequestion=question_match.group(1).strip() ifquestion_matchelseNoneparse_error=scenarioisNoneorquestionisNonereturnScore(
value=1.0,
answer=model_output,
metadata={
"parse_error": parse_error,
"question": questionifquestionelse"",
"answer": scenarioifscenarioelse"",
},
)
returnscenario_parser_v0
Should probably error the user earlier, when they first create the scorer or (as in my usecase) allow for an empty list of metrics.
The text was updated successfully, but these errors were encountered:
As described in #1324 , I'm trying to create a scorer which just returns strings. I hence passed an empry list to metrics (since I don't want to reduce anything). I get this error from the resulting log when I try to open it:
Here is my scorer, I believe this reproduces with any scorer if you make metrics empty:
Should probably error the user earlier, when they first create the scorer or (as in my usecase) allow for an empty list of metrics.
The text was updated successfully, but these errors were encountered: