Diff based culling

[maxnth]

Use case

When correcting ground truth in big datasets it's often useful to check the diff between a very good prediction and the ground truth in LAREX and correct it if necessary. Culling the correction data set of all files which don't contain any diff between prediction and ground truth makes this a lot easier.

Implementation

The CLI should accept:

• a list of PAGE XML files and two indices (for TextEquiv/@index) with denominate prediction and ground truth
• two lists of files with one index each in case GT and Pred are stored in two different XML files
• whether to apply Unicode normalization / regularization
• An output directory