Arthur 15cfe38942 [`Core tokenization`] `add_dummy_prefix_space` option to help with latest issues (#28010) ... * add add_dummy_prefix_space option to slow * checking kwargs might be better. Should be there for all spm tokenizer IMO * nits * fix copies * more copied * nits * add prefix space * nit * nits * Update src/transformers/convert_slow_tokenizer.py * fix inti * revert wrong styling * fix * nits * style * updates * make sure we use slow tokenizer for conversion instead of looking for the decoder * support llama ast well * update llama tokenizer fast * nits * nits nits nits * update the doc * update * update to fix tests * skip unrelated tailing test * Update src/transformers/convert_slow_tokenizer.py * add proper testing * test decode as well * more testing * format * fix llama test * Apply suggestions from code review		2024-02-20 12:50:31 +01:00
..
__init__.py	Move test model folders (#17034)	2022-05-03 14:42:02 +02:00
test_modeling_flax_t5.py	Update all references to canonical models (#29001)	2024-02-16 08:16:58 +01:00
test_modeling_t5.py	Update all references to canonical models (#29001)	2024-02-16 08:16:58 +01:00
test_modeling_tf_t5.py	Update all references to canonical models (#29001)	2024-02-16 08:16:58 +01:00
test_tokenization_t5.py	[`Core tokenization`] `add_dummy_prefix_space` option to help with latest issues (#28010)	2024-02-20 12:50:31 +01:00