Arthur b44567538b [`NllbTokenizer`] refactor with added tokens decoder (#27717) ... * refactor with addedtokens decoder * style * get rid of lang code to id * style * keep some things for BC * update tests * add the mask token at the end of the vocab * nits * nits * fix final tests * style * nits * Update src/transformers/models/nllb/tokenization_nllb_fast.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * nits * style? * Update src/transformers/convert_slow_tokenizer.py * make it a tad bit more custom * ruff please stop Co-Authored by avidale <dale.david@mail.ru> * Update Co-authored-by: avidale <dale.david@mail.ru> * Update Co-authored-by: avidale <dale.david@mail.ru> * oupts * ouft * nites * test * fix the remaining failing tests * style * fix failing test * ficx other test * temp dir + test the raw init * update test * style --------- Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>		2024-02-13 03:49:20 +01:00
..
__init__.py	NLLB tokenizer (#18126)	2022-07-18 08:12:34 -04:00
test_tokenization_nllb.py	[`NllbTokenizer`] refactor with added tokens decoder (#27717)	2024-02-13 03:49:20 +01:00