Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove the flores101 related tokenizer logging message when the selected tokenizer is flores200 #219

Open
hadyelsahar opened this issue Oct 24, 2022 · 0 comments
Open

Remove the flores101 related tokenizer logging message when the selected tokenizer is flores200 #219

hadyelsahar opened this issue Oct 24, 2022 · 0 comments
Labels
enhancement

Comments

@hadyelsahar
Copy link

This line outputs the following message below which might confuse the user to think that flores101 tokenizer is activated even when the tokenizer flores200 is selected. Better trigger the warning only when spm or flores101 tokenizers are selected

sacrebleu/sacrebleu/tokenizers/tokenizer_spm.py

Line 37 in 734669c

sacrelogger.warn("Tokenizer 'spm' has been changed to 'flores101', and may be removed in the future.")

Outputted message

2022-10-24 12:27:01 | WARNING | sacrebleu | Tokenizer 'spm' has been changed to 'flores101', and may be removed in the future.
Pogayo added a commit to Pogayo/sacrebleu that referenced this issue Jul 15, 2023
@Pogayo
Update tokenizer_spm.py to issue warning message only when 'spm' is used
9c96505 
An attempt at fixing issue 219
mjpost#219
@Pogayo Pogayo mentioned this issue Jul 15, 2023
Merged
martinpopel pushed a commit that referenced this issue Jul 17, 2023
@Pogayo
Update tokenizer_spm.py to issue warning message only when 'spm' is u…
bafca9d 
…sed (#238)

* Update tokenizer_spm.py to issue warning message only when 'spm' is used

An attempt at fixing issue 219
#219
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ozancaglayan @hadyelsahar