SaulLu 1025a9b742 add a warning in `SpmConverter` for sentencepiece's model using the byte fallback feature (#16629) ... * update proto sentencepiece model * Revert "update proto sentencepiece model" This reverts commit b07f671747. * add check * add test * Revert "Revert "update proto sentencepiece model"" This reverts commit 46108257b8. * test for log level * test for log level 2 * warning at the warning level * clean * format * add explanation in docstring		2022-04-11 11:06:10 +02:00
..
tests_samples	…
add_distilbert_like_config.json	…
dummy-config.json	…
dummy_feature_extractor_config.json	…
empty.txt	…
input.txt	…
merges.txt	…
preprocessor_config.json	…
sample_text.txt	…
sample_text_no_unicode.txt	…
spiece.model	…
test_entity_vocab.json	…
test_sentencepiece.model	…
test_sentencepiece_bpe.model	…
test_sentencepiece_no_bos.model	…
test_sentencepiece_with_bytefallback.model	…
vocab.json	…
vocab.txt	…