The Webapp
The simulator allows investigating the performance of a matcher and PAD, separately and then fused into the selected scenario, by taking as input the individual ROC (Receiver Operating Characteristic) curves, and provides the ROC of their fusion according to two parameters:
- The prior probability of being attacked by spoofs w
- The specific operational point chosen for the PAD, set by APCER ≤ p% or BPCER ≤ p%
To use the simulator, we provided a web application, Bio-WISE, which performs such fusion by simply selecting the setting in terms of attack probability (w) and operational point for the PAD (APCER ≤ p% or BPCER ≤ p%), and the input data related to both matcher and PAD through a csv file.
The following sections explain details about the required input file, the fusion settings, and the functionalities of the webapp.
After a brief description about the simulator, you can find the input area, which explains the two enabled settings for the input data. As it is possible to observe, there are two available settings:
- Performance simulation from ROCs of both matcher and PAD
- Performance simulation from estimated scores from matcher and PAD
As previously cited, data from matcher and PAD can be inserted through a csv file, containing values related to a single type of data on different lines (or rows). Therefore, each of the lines contains data about a single parameter, which values are separated by a comma (space after the comma is tolerated as well).
The first setting (i.e., computation from ROCs) requires a csv file with data on five different rows:
- FMR (False Match Rate, previously known as False Acceptance Rate or FAR), identifying the percentage the proportion of impostor samples classified as genuine by the matcher at different decision thresholds
- FNMR (False Non-Match Rate, previously known as False Rejection Rate or FRR), identifying the proportion of genuine samples classified as impostor by the matcher at different decision thresholds
- IAPMR (Impostor Attack Presentation Match Rate, previously known as Spoof False Acceptance Rate or SFAR), identifying the proportion of impostor attack presentation using the same PAI (Presentation Attach Instrument) species in which the target reference is matched.
- APCER (Attack Presentation Classification Error Rate, previously known as False Acceptance Rate or FAR in PADs), identifying the percentage the proportion of impostor samples classified as genuine by the matcher at different decision thresholds
- BPCER (Bona-fide Presentation Classification Error Rate, previously known as False Rejection Rate or FRR in PADs), identifying the proportion of genuine samples classified as impostors by the matcher at different decision thresholds
All these values can either be inserted as percentages (e.g., 0, 3, 30.1, 43, 45.3, 67, 90.9, 100) or fractions between 0 and 1, where 1 corresponds to 100% (e.g., 0, 0.03, 0.301, 0.43, 0.453, 0.0.67, 0.909, 1). The webapp will automatically recognize the format by assuming that data is presented as percentages whenever any of the values is greater than 1, as fractions otherwise.
However, it is also possible to use the simulator as a simple performance estimator of either matcher or PAD, by avoiding inserting all the lines on the csv files:
- If only 2 lines are inserted, they will be considered as FMR and FNMR and the related performance will be estimated
- If only 3 lines are inserted, they will be considered as FMR, FNMR, and IAPMR and the full matcher performance will be estimated
- If all the 5 lines are inserted, therefore the performance of both matcher and PAD are estimated as well as of the whole system created by their fusion
The second setting (i.e., computation from raw scores) requires a csv file with data on four different rows:
- Scores related to the verified identity, identifying the scores associated from the matcher to each sample with respect to the verified identity
- IDs, identifying if each sample is related to a genuine attempt (value=1) or to an impostor one (value=0)
- Liveness scores, identifying the scores associated from the PAD to each sample with respect to its likelihood to be a live samples
- Liveness IDs, identifying if each sample is live (value=1) or a fake (i.e., spoofed) sample (value=0)
The simulator will automatically estimate FMR, FNMR, IAPMR, APCER, and BPCER from these data, assuming a decision threshold step equal to 0.01 (or 1%), evaluating all the thresholds between 0 and 1 (or 0% and 100%).
As in the previous setting, it is possible to insert only 2 lines (i.e., scores related to the verified identity and the IDs) for estimating performance related to the only matcher, or all the 4 lines to perform the full assessment.
While in the first line and the third line (scores related to the verified identity and to their liveness, respectively) it is possible to insert data as in the previous setting, in the second and the fourth line (IDs and liveness IDs, respectively) it is necessary to use a binary value (0 or 1) which asserts if such sample is related to a genuine sample (value=1) or to an impostor sample (value=0).
After the selection of the required data settings, it is possible to insert the related csv file through the Input file button. Note that, for further analysis, the extracted or estimated FMR, FNMR, IAPMR, APCER, and BPCER will be memorized. Therefore, it is possible to perform the simulation in different scenarios without inserting the csv file again, and using the default data setting (i.e., from ROCs).
However, if you have any doubts about how to set the file for the required setting, you can select such a setting through the radio buttons and then download an example file through the Example button.
In order to perform the simulation, you must select the scenario.
Therefore, you must select 2 different parameters:
- The prior probability of being attacked by spoofs w
- The specific operational point chosen for the PAD, set by APCER ≤ p% or BPCER ≤ p%
Note that, once you choose between APCER ≤ p% and BPCER ≤ p%, the parameter which has not been chosen will be determined accordingly. For example, if you select APCER ≤ 5%, then BPCER will be assumed to be the value for which APCER corresponds to such value.
Furthermore, note that whenever the selected operational point is not present for the chosen parameter, the smaller value closest to the value p will be considered.
Note that scenario selection is not required if you need to analyze the matcher-only or the PAD-only performance, without fusing them.
Once you uploaded the file and selected the simulation scenario, you can finally perform your simulation. The webapp is able to provide the results into two different ways:
- You can download a csv file containing the results into a textual form through the "Report generation button
- You can display results graphically on the webapp through the "Performance evaluation" button
Note that in the second case, you can simply download the produced images by right-clicking with your mouse/pad and selecting "Save as" (or a similar sentence) from the window which will be shown.
Go to the other sections of the Wiki for further information about the simulator.