Skip to content

Commit

Permalink
Testing aspect of page for Jasmin
Browse files Browse the repository at this point in the history
  • Loading branch information
greenw0lf committed Jul 5, 2024
1 parent 904d57b commit 9d5ae3e
Show file tree
Hide file tree
Showing 2 changed files with 18 additions and 10 deletions.
18 changes: 11 additions & 7 deletions UT/CommonVoice/cv.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,21 +8,25 @@ Multiple versions of the dataset are available, released by Mozilla every few mo

<br>

Here is a matrix with **WER** results and the **time** each model/configuration spent transcribing the dataset:
Here is a matrix with **WER** results and the **time** each model/configuration spent transcribing the test subset:

|Model|WER|Time|
|---|---|---|
|Kaldi_NL|20.7%|8h:15m:54s*|
|faster-whisper v2|5.6%|1h:58m:37s|
|*faster-whisper v3*|**4.3%**|1h:55m:20s|
|[Kaldi_NL](https://github.com/opensource-spraakherkenning-nl/Kaldi_NL)|20.7%|8h:15m:54s*|
|[faster-whisper v2](https://github.com/SYSTRAN/faster-whisper/)|5.6%|1h:58m:37s|
|**faster-whisper v3**|**4.3%**|1h:55m:20s|
|faster-whisper v2 w/ VAD|5.6%|1h:58m:50s|
|faster-whisper v3 w/ VAD|4.4%|2h:01m:33s|
|XLS-R FT on Dutch|6.5%|1h:04m:00s|
|*MMS - 102 languages*|13.4%|**0h:37m:50s**|
|MMS - 1162 languages|9.5%|0:53m:56s|
|[XLS-R FT on Dutch](https://huggingface.co/jonatasgrosman/wav2vec2-xls-r-1b-dutch)|6.5%|1h:04m:00s|
|[**MMS - 102 languages**](https://huggingface.co/facebook/mms-1b-fl102)|13.4%|**0h:37m:50s**|
|[MMS - 1162 languages](https://huggingface.co/facebook/mms-1b-all)|9.5%|0:53m:56s|

\* Most of the time was spent by the configuration running the speaker diarization module.

<br>

There is a clear discrepancy between the WER scores of the newer ASR models and the baseline which is explained by the fact that data from Common Voice has been used to train the newer models, whereas that is not the case for our baseline.


### Normalization

Expand Down
10 changes: 7 additions & 3 deletions UT/Jasmin/jasmin_res.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
[Go back](./jasmin.md)

## Jasmin Dutch results
Here is a matrix with **WER** results of the baseline model, Kaldi_NL, as well as different models tested on the **Dutch** part of the corpus:
Here is a matrix with **WER** results of the baseline model, Kaldi_NL, as well as different models tested on **Dutch read speech**:

|Model\Dataset|Jasmin_q_1|Jasmin_q_2|Jasmin_q_3|Jasmin_q_4|Jasmin_q_5|
|Model\Dataset|Native Children|Native Teenagers|Non-native Minors|Non-native Adults|Native Elderly|
|---|---|---|---|---|---|
|Kaldi_NL|28.1%|16.2%|43.6%|45.3%|20.9%|
|Whisper v2|22.6%|18.0%|36.5%|37.3%|22.2%|
Expand All @@ -18,7 +18,11 @@ Here is a matrix with **WER** results of the baseline model, Kaldi_NL, as well a
|MMS - 102 languages|31.6%|20.3%|54.2%|55.1%|23.9%|
|MMS - 1162 languages|28.9%|20.0%|50.1%|54.0%|28.3%|

|Model\Dataset|Jasmin_p_1|Jasmin_p_2|Jasmin_p_3|Jasmin_p_4|Jasmin_p_5|
<br>

And for **Dutch conversational speech**:

|Model\Dataset|Native Children|Native Teenagers|Non-native Minors|Non-native Adults|Native Elderly|
|---|---|---|---|---|---|
|Kaldi_NL|55.4%|62.4%|69.1%|60.0%|44.0%|
|Whisper v2|95.8%|107.4%|124.0%|88.1%|61.9%|
Expand Down

0 comments on commit 9d5ae3e

Please sign in to comment.