dnrops
/
transformers
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
transformers/tests/fixtures
History
SaulLu 1025a9b742
add a warning in `SpmConverter` for sentencepiece's model using the byte fallback feature (#16629) 
* update proto sentencepiece model

* Revert "update proto sentencepiece model"

This reverts commit b07f671747.

* add check

* add test

* Revert "Revert "update proto sentencepiece model""

This reverts commit 46108257b8.

* test for log level

* test for log level 2

* warning at the warning level

* clean

* format

* add explanation in docstring
 		2022-04-11 11:06:10 +02:00
..
tests_samples Fix img classification tests (#13456) 2021-09-07 05:58:45 -04:00
add_distilbert_like_config.json Add model like (#14992) 2022-01-24 15:25:10 -05:00
dummy-config.json AutoConfig + other Auto classes honor model_type 2020-01-11 02:46:17 +00:00
dummy_feature_extractor_config.json [AutoProcessor] Add Wav2Vec2WithLM & small fix (#14675) 2021-12-08 15:51:28 +01:00
empty.txt GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
input.txt GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
merges.txt [AutoTokenizer] Allow creation of tokenizers by tokenizer type (#13668) 2021-09-22 00:29:38 +02:00
preprocessor_config.json Auto processor (#14465) 2021-11-22 12:17:38 -05:00
sample_text.txt GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
sample_text_no_unicode.txt [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
spiece.model GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
test_entity_vocab.json Feature/fix slow test in mluke (#14749) 2021-12-22 06:35:59 -05:00
test_sentencepiece.model GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
test_sentencepiece_bpe.model Conversion from slow to fast for BPE spm vocabs contained an error. (#10120) 2021-02-13 08:24:53 -05:00
test_sentencepiece_no_bos.model [pegasus] Faster tokenizer tests (#7672) 2020-10-09 11:10:32 -04:00
test_sentencepiece_with_bytefallback.model add a warning in `SpmConverter` for sentencepiece's model using the byte fallback feature (#16629) 2022-04-11 11:06:10 +02:00
vocab.json [AutoTokenizer] Allow creation of tokenizers by tokenizer type (#13668) 2021-09-22 00:29:38 +02:00
vocab.txt [AutoTokenizer] Allow creation of tokenizers by tokenizer type (#13668) 2021-09-22 00:29:38 +02:00