Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Difference between cli and corpus_score #259

Open
VarunGumma opened this issue Mar 20, 2024 · 1 comment
Open

Difference between cli and corpus_score #259

VarunGumma opened this issue Mar 20, 2024 · 1 comment

Comments

@VarunGumma
Copy link

I am attempting to compute chrF++ for a set of predictions and references. If I use sacrebleu cli (sacrebleu ref.eng_Latn.tok < pred.eng_Latn.tok -m bleu chrf --chrf-word-order 2), I find a significant difference when I use corpus_score with CHRF(word_order=2).corpus_score(preds, refs). I have double-checked the data in both cases, and it is correct and the same, so no issues there. Any reason why this is happening? Similarly, the BLEU scores (with BLEU().corpus_score(preds, refs)) also varies significantly. Are there some default params that I am missing?

@nkrasner
Copy link

nkrasner commented May 31, 2024

I think this is related to #220 . I was having the same issue, transposing the references as they mentioned fixed my issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants