is there a way to load tokenizer_config.json by djl?

[gabohouhou]

from class HuggingFaceTokenizer i can find the way to load tokenizer.json, but model tokenizer often use 2 files ：tokenizer.json and tokenizer_config.json。is there a way to load tokenizer_config.json？

[frankfliu]

You can create your own TranslatorFactory, read tokenizer_config.json and pass the options to HuggingfaceTokenizer.

Here is the place you can inject code: https://github.com/deepjavalibrary/djl/blob/master/extensions/tokenizers/src/main/java/ai/djl/huggingface/translator/TextEmbeddingTranslatorFactory.java#L64

If you don't want to touch TranslatorFactory, you can manually read tokenizer_config.json and extract arguments and pass them in to Criteria, for example: Criteria.optArgument("doLowerCase", "true")