feat: add JGLUE tasks #469
Conversation
| def correlation_metric(golds: list[int], predictions: list[str], **kwargs): | ||
| def convert_to_float(score): | ||
| try: | ||
| return float(score) | ||
| except ValueError: | ||
| return None | ||
|
|
||
| predicted_score = convert_to_float(predictions[0]) | ||
| gold_score = convert_to_float(golds[0]) | ||
|
|
||
| return { | ||
| "predicted_score": predicted_score, | ||
| "gold_score": gold_score, | ||
| } | ||
|
|
||
|
|
||
| def spearman_corpus_metric(items): | ||
| predicted_scores, gold_scores = zip( | ||
| *[ | ||
| (item["predicted_score"], item["gold_score"]) | ||
| for item in items | ||
| if (item["gold_score"] is not None and item["predicted_score"] is not None) | ||
| ] | ||
| ) | ||
| r, _ = spearmanr(predicted_scores, gold_scores) | ||
| if np.isnan(r): | ||
| return 0.0 | ||
| frac = len(predicted_scores) / len(items) |
There was a problem hiding this comment.
@NathanHB I believe we could add these 2 to core metrics, wdyt?
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Sorry for the delay—I was on vacation last week and didn't check anything on GitHub. Wishing you a Happy New Year! It seems my earlier explanation may have caused some confusion. I am in the process of creating the |
|
Thanks for the explanation! Do you have any other implementation against which you could check your results? You'll need to run the code quality too :) |
|
You can also add the 2 metrics I highlighted to the core metrics file if you want, as they are very valuable |
I’ll start by fixing the CI tonight and then transfer the metrics to the core. I might also fix the |
|
Hm, would you have a simpler way to make sure your results are within range? Maybe a paper reported results with their implementation and you could try to reproduce it? |
JGLUE is a widely used test set in the Japanese LLM research community, consisting of five sub-tests (with MARC-ja removed due to a request from Amazon):
JSTS
JNLI
JSQuAD
JCommonsenseQA
Finished Issue #455