Added early stopping capability with multibleu.perl (extra commits removed) #19
Conversation
kelvinxu
changed the title
| @@ -336,6 +343,14 @@ def main(self): | |||
| self.step % self.state['hookFreq'] == 0 and \ | |||
| self.hooks: | |||
| [fn() for fn in self.hooks] | |||
|
|
|||
| if self.state['bleu_val_frequency'] is not None and \ | |||
There was a problem hiding this comment.
This is not good: Groundhog's main_loop should keep being task agnostic. Can you call you BLUE score computation with the standard validation callback?
There was a problem hiding this comment.
I agree this is not ideal. But initially I thought it'd be nice to both track the standard cross entropy in the validation call back + compute bleu (occasionally) so we can compare how well we are optimizing both. Do you think we should have both in one validation call?
There was a problem hiding this comment.
I think it makes sense, as both are validation costs. Alternatively you can do bleu-related things using the hook mechanism (which is currently used to print samples).
There was a problem hiding this comment.
Alright. I'm gonna implement a normal validation function using cross entropy, since that seems to be missing. Might be interesting to see how well it correlates with BLEU.
Kelvin
There was a problem hiding this comment.
Nice! But if just moved the code that calls BLEU computation from main_loop into the validation callback, we could merge this pull request.
There was a problem hiding this comment.
I see, I'll move this by the end of the week.
Notes:
It assumed utf8 encoding meant chinese character based bleu, which might not be right for other languages.