dnrops
/
transformers
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
transformers/tests
History
Yu Liu 762cba3bda
Albert pretrain datasets/ datacollator (#6168) 
* add dataset for albert pretrain

* datacollator for albert pretrain

* naming, comprehension, file reading change

* data cleaning is no needed after this modification

* delete prints

* fix a bug

* file structure change

* add tests for albert datacollator

* remove random seed

* add back len and get item function

* sample file for testing and test code added

* format change for black

* more format change

* Style

* var assignment issue resolve

* add back wrongly deleted DataCollatorWithPadding in init file

* Style

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>
 		2020-09-10 07:56:29 -04:00
..
fixtures Albert pretrain datasets/ datacollator (#6168) 2020-09-10 07:56:29 -04:00
__init__.py GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
conftest.py enable easy checkout switch (#5645) 2020-07-31 04:34:46 -04:00
test_activations.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_benchmark.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_benchmark_tf.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_cli.py [transformers-cli] fix logger getter (#6777) 2020-08-27 20:01:17 -04:00
test_configuration_auto.py Move tests/utils.py -> transformers/testing_utils.py (#5350) 2020-07-01 10:31:17 -04:00
test_configuration_common.py Pass kwargs to configuration (#3147) 2020-03-05 17:16:57 -05:00
test_data_collator.py Albert pretrain datasets/ datacollator (#6168) 2020-09-10 07:56:29 -04:00
test_doc_samples.py Move tests/utils.py -> transformers/testing_utils.py (#5350) 2020-07-01 10:31:17 -04:00
test_hf_api.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_hf_argparser.py parse arguments from dict (#4869) 2020-07-31 04:44:23 -04:00
test_logging.py adding TRANSFORMERS_VERBOSITY env var (#6961) 2020-09-09 04:08:01 -04:00
test_model_card.py GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
test_modeling_albert.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_auto.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_modeling_bart.py prepare_seq2seq_batch makes labels/ decoder_input_ids made later. (#6654) 2020-08-28 11:15:17 -04:00
test_modeling_bert.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_camembert.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_common.py Funnel transformer (#6908) 2020-09-08 08:08:08 -04:00
test_modeling_ctrl.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_distilbert.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_dpr.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_electra.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_encoder_decoder.py [Generate] Facilitate PyTorch generate using `ModelOutputs` (#6735) 2020-09-01 12:38:25 +02:00
test_modeling_flaubert.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_funnel.py Funnel transformer (#6908) 2020-09-08 08:08:08 -04:00
test_modeling_gpt2.py [Generate] Facilitate PyTorch generate using `ModelOutputs` (#6735) 2020-09-01 12:38:25 +02:00
test_modeling_longformer.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_lxmert.py fix (#6946) 2020-09-04 16:08:54 +02:00
test_modeling_marian.py Fix marian slow test (#6854) 2020-08-31 16:10:43 -04:00
test_modeling_mbart.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_modeling_mobilebert.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_openai.py [Generate] Facilitate PyTorch generate using `ModelOutputs` (#6735) 2020-09-01 12:38:25 +02:00
test_modeling_pegasus.py Pegasus finetune script: add --adafactor (#6811) 2020-08-29 17:43:32 -04:00
test_modeling_reformer.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_roberta.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_t5.py [Generate] Facilitate PyTorch generate using `ModelOutputs` (#6735) 2020-09-01 12:38:25 +02:00
test_modeling_tf_albert.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_modeling_tf_auto.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_modeling_tf_bert.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_modeling_tf_camembert.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_tf_common.py test_tf_common: remove un_used mixin class parameters (#6866) 2020-09-02 10:54:40 -04:00
test_modeling_tf_ctrl.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_tf_distilbert.py test_tf_common: remove un_used mixin class parameters (#6866) 2020-09-02 10:54:40 -04:00
test_modeling_tf_electra.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_tf_flaubert.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_tf_gpt2.py [Generate] Facilitate PyTorch generate using `ModelOutputs` (#6735) 2020-09-01 12:38:25 +02:00
test_modeling_tf_longformer.py test_tf_common: remove un_used mixin class parameters (#6866) 2020-09-02 10:54:40 -04:00
test_modeling_tf_lxmert.py Adding the LXMERT pretraining model (MultiModal languageXvision) to HuggingFace's suite of models (#5793) 2020-09-03 04:02:25 -04:00
test_modeling_tf_mobilebert.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_modeling_tf_openai.py [Generate] Facilitate PyTorch generate using `ModelOutputs` (#6735) 2020-09-01 12:38:25 +02:00
test_modeling_tf_roberta.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_tf_t5.py [Generate] Facilitate PyTorch generate using `ModelOutputs` (#6735) 2020-09-01 12:38:25 +02:00
test_modeling_tf_transfo_xl.py test_tf_common: remove un_used mixin class parameters (#6866) 2020-09-02 10:54:40 -04:00
test_modeling_tf_xlm.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_tf_xlm_roberta.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_modeling_tf_xlnet.py test_tf_common: remove un_used mixin class parameters (#6866) 2020-09-02 10:54:40 -04:00
test_modeling_transfo_xl.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_xlm.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_xlm_roberta.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_modeling_xlnet.py Black 20 release 2020-08-26 17:20:22 +02:00
test_onnx.py Fix flaky ONNX tests (#6531) 2020-08-17 09:04:35 -04:00
test_optimization.py Format 2020-08-27 18:31:51 +02:00
test_optimization_tf.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_pipelines.py feat: allow prefix for any generative model (#5885) 2020-09-07 03:03:45 -04:00
test_tokenization_albert.py [HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510) 2020-06-15 17:12:51 -04:00
test_tokenization_auto.py [from_pretrained] Allow tokenizer_type ≠ model_type (#6995) 2020-09-09 04:22:59 -04:00
test_tokenization_bart.py [tests] fix typos in inputs (#6818) 2020-08-30 18:19:57 +08:00
test_tokenization_bert.py Add strip_accents to basic BertTokenizer. (#6280) 2020-08-06 18:52:28 +08:00
test_tokenization_bert_japanese.py Support additional dictionaries for BERT Japanese tokenizers (#6515) 2020-08-17 12:00:23 +08:00
test_tokenization_common.py Batch encore plus and overflowing tokens fails when non existing overflowing tokens for a sequence (#6677) 2020-09-09 06:55:17 -04:00
test_tokenization_ctrl.py [HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510) 2020-06-15 17:12:51 -04:00
test_tokenization_distilbert.py Move tests/utils.py -> transformers/testing_utils.py (#5350) 2020-07-01 10:31:17 -04:00
test_tokenization_dpr.py Fix tests imports dpr (#5576) 2020-07-07 16:35:12 +02:00
test_tokenization_fast.py Fix style (#6803) 2020-08-28 15:02:25 -04:00
test_tokenization_funnel.py Funnel transformer (#6908) 2020-09-08 08:08:08 -04:00
test_tokenization_gpt2.py [HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510) 2020-06-15 17:12:51 -04:00
test_tokenization_lxmert.py Adding the LXMERT pretraining model (MultiModal languageXvision) to HuggingFace's suite of models (#5793) 2020-09-03 04:02:25 -04:00
test_tokenization_marian.py rename prepare_translation_batch -> prepare_seq2seq_batch (#6103) 2020-08-11 15:57:07 -04:00
test_tokenization_mbart.py prepare_seq2seq_batch makes labels/ decoder_input_ids made later. (#6654) 2020-08-28 11:15:17 -04:00
test_tokenization_openai.py [HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510) 2020-06-15 17:12:51 -04:00
test_tokenization_pegasus.py prepare_seq2seq_batch makes labels/ decoder_input_ids made later. (#6654) 2020-08-28 11:15:17 -04:00
test_tokenization_reformer.py Black 20 release 2020-08-26 17:20:22 +02:00
test_tokenization_roberta.py prepare_seq2seq_batch makes labels/ decoder_input_ids made later. (#6654) 2020-08-28 11:15:17 -04:00
test_tokenization_t5.py [tests] fix typos in inputs (#6818) 2020-08-30 18:19:57 +08:00
test_tokenization_transfo_xl.py Transformer-XL: Improved tokenization with sacremoses (#6322) 2020-08-28 09:56:17 -04:00
test_tokenization_utils.py Fixes to make life easier with the nlp library (#6423) 2020-08-12 08:00:56 -04:00
test_tokenization_xlm.py Move tests/utils.py -> transformers/testing_utils.py (#5350) 2020-07-01 10:31:17 -04:00
test_tokenization_xlm_roberta.py Move tests/utils.py -> transformers/testing_utils.py (#5350) 2020-07-01 10:31:17 -04:00
test_tokenization_xlnet.py Move tests/utils.py -> transformers/testing_utils.py (#5350) 2020-07-01 10:31:17 -04:00
test_trainer.py [testing] replace hardcoded paths to allow running tests from anywhere (#6523) 2020-08-27 12:22:18 -04:00
test_trainer_distributed.py Add tests to Trainer (#6605) 2020-08-20 11:13:50 -04:00