15cfe38942
* add add_dummy_prefix_space option to slow * checking kwargs might be better. Should be there for all spm tokenizer IMO * nits * fix copies * more copied * nits * add prefix space * nit * nits * Update src/transformers/convert_slow_tokenizer.py * fix inti * revert wrong styling * fix * nits * style * updates * make sure we use slow tokenizer for conversion instead of looking for the decoder * support llama ast well * update llama tokenizer fast * nits * nits nits nits * update the doc * update * update to fix tests * skip unrelated tailing test * Update src/transformers/convert_slow_tokenizer.py * add proper testing * test decode as well * more testing * format * fix llama test * Apply suggestions from code review |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_feature_extraction_seamless_m4t.py | ||
| test_modeling_seamless_m4t.py | ||
| test_processor_seamless_m4t.py | ||
| test_tokenization_seamless_m4t.py | ||