A question about the split of dataset

[wuxchcandoit]

Hi!
In the paper, dataset is split into two parts, training slides and testing slides. In the code, testing dataset is evaluated after every epoch of training. It seems that testing dataset is used as validation dataset. Do the classification results come from results of the last evaluation? Which do the classification results come from, results of the last evaluation or the best evaluation?

[binli123]

Hi, the test set should be reserved before the training, and the training process further splits the training data into a training set and a validation set. After the training is done you can then evaluate with the best model on the test data.

[binli123]

I incorporated the training/testing into the same pipeline in the latest commit. You can set --eval_scheme=5-fold-cv-standalone-test which will perform a train/valid/test like this:

A standalone test set consisting of 20% samples is reserved, remaining 80% of samples are used to construct a 5-fold cross-validation.
For each fold, the best model and corresponding threshold are saved.
After the 5-fold cross-validation, 5 best models along with the corresponding optimal thresholds are obtained which are used to perform inference on the reserved test set. A final prediction for a test sample is the majority vote of the 5 models.
For a binary classification, accuracy and balanced accuracy scores are computed. For a multi-label classification, hamming loss (smaller the better) and subset accuracy are computed.

You can also simply run a 5-fold cv --eval_scheme=5-fold-cv

There were some issues with the testing script when loading pretrained weights (i.e., sometimes the weights are not fully loaded or there are missing weights, setting strict=False can reveal the problems.). The purpose of the testing script is to generate the heatmap, you should now read the performance directly from the training script. I will fix the issues in a couple of days.