transformers/tests/models/codegen
st81 8d6b509611
Add token type ids to CodeGenTokenizer (#29265)
* Add create token type ids to CodeGenTokenizer

* Fix inconsistent length of token type ids

* Format source codes

* Fix inconsistent order of methods

* Update docstring

* add test_tokenizer_integration test

* Format source codes

* Add `copied from` comment to CodeGenTokenizerFast

* Add doc of create_token_type_ids_from_sequences

* Make return_token_type_ids False by default

* Make test_tokenizer_integration as slow test

* Add return_token_type_ids to tokenizer init arg

* Add test for tokenizer's init return_token_type_ids

* Format source codes
2024-04-17 12:19:18 +02:00
..
__init__.py Add CodeGen model (#17443) 2022-06-24 17:10:38 +02:00
test_modeling_codegen.py Remove static pretrained maps from the library's internals (#29112) 2024-03-25 10:33:38 +01:00
test_tokenization_codegen.py Add token type ids to CodeGenTokenizer (#29265) 2024-04-17 12:19:18 +02:00