2d2ed2cc18
[T5] Fix speed degradation bug t5 ( #10496 )
...
* fix speed degradation bug t5
* fix for all models
* fix code quality
2021-03-03 12:42:41 +03:00
894db6701e
Bugfix: Removal of padding_idx in BartLearnedPositionalEmbedding ( #10200 )
...
* Assumption of padding_idx <2 might not stand
* Use offset instead of 2
* Fix with black
* Change behavior to warning instead for backward compatibility.
* Fix with black
* Remove warning
* Make padding_idx non-required
* padding_idx fix for blenderbot
* padding_idx fix for blenderbot_small
* padding_idx fix for led
* padding_idx fix for mbart
* Remove extra whitespaces
* padding_idx fix for template
* Fix padding_idx passed to nn.Embedding mistake
* Fixed padding_idx passed to positional embedding in template
* Remove padding_idx from pytorch learned positional embeddings
* Remove accidentally added quotes
* Remove padding_idx from tf learned positional embeddings
* Remove zeroing of weights in __init__
Co-authored-by: Wang Ming Rui <mingrui.wang@C02CJTUYMD6M.local>
2021-02-25 14:33:13 +03:00
83d803ba02
Making TF BART-like models XLA and AMP compliant ( #10191 )
...
* Update BART
* Update Blenderbot
* Update BlenderbotSmall
* Update Marian
* Update MBart
* Update MBart
* Update Pegasus
* Update template
* Fix Marian and Pegasus
* Apply style
* Default initializer
* Default initializer
* Default initializer
* Remove int32 casts
* Fix template
* Remove more cast
2021-02-17 17:48:56 +01:00
31b0560ab4
Add AMP for Albert ( #10141 )
2021-02-15 17:18:33 +01:00
570218878a
Fix TF template ( #10189 )
...
* Fix template
* Update Seq2Seq tests
2021-02-15 09:21:57 -05:00
8e13b73593
Update README.md
2021-02-11 18:35:27 +03:00
d6b4f48ecb
Update ADD_BIG_BIRD.md
2021-02-11 18:34:17 +03:00
4cda2d73ef
Update ADD_BIG_BIRD.md
2021-02-09 19:58:35 +03:00
b82fe7d258
Replace strided slice with tf.expand_dims ( #10078 )
...
* Replace tf.newaxis -> tf.expand_dims
* Fix tests
* Fix tests
* Use reshape when a tensors needs a double expand
* Fix GPT2
* Fix GPT2
2021-02-09 11:48:28 -05:00
c9df1b1d53
Model templates ( #10072 )
2021-02-08 09:07:02 -05:00
cdd8659231
Fix TF template ( #10069 )
...
* Fix template
* Fix template
2021-02-08 08:10:50 -05:00
31563e056d
Restore TF embeddings and attention layers to their previous version ( #9890 )
...
* Refacto BERT
* Restore all the concerned models
* Remove print
* Update template
* Apply Sylvain's and Morgan's comments
* Fix cast
* Put the cast inside call
* Remove cond in ebds
* Fix funnel
* Restore previous dot product (attention_scores) computation
* Add ConvBERT and BART
* Make all the S2S models ONNX compliant
* Fix test
* Fix check copies
2021-02-08 14:36:30 +03:00
ae37ceacbd
Fix typo ( #10064 )
2021-02-08 06:02:05 -05:00
8ea412a86f
[examples] make run scripts executable ( #10037 )
...
* make executable
* make executable
* same for the template
* cleanup
2021-02-05 15:51:18 -08:00
89be094e29
[Templates] Add template "call-for-model" markdown and "call-for-big-bird" markdown ( #9921 )
...
* add big bird
* change teacher to mentor
* add proposal template
* adapt template
* delete old template
* correct some links
* finish template
* create big bird from template
* add big bird
* improve boxes
* finish boxes
* add pointers for BigBird
* finish big bird
* up
* up
* up
* up
* apply lysandres and sylvains suggestions
* delete bogus file
* correct markdown
* try different style
* try different style
* finalize
2021-02-05 15:47:54 +03:00
e89c959af9
Fix model templates ( #9999 )
2021-02-04 07:47:26 -05:00
00031785a8
BartForCausalLM analogs to `ProphetNetForCausalLM` ( #9128 )
...
* initiliaze bart4causalLM
* create BartDecoderWrapper, setters/getters
* delete spaces
* forward and additional methods
* update cache function, loss function, remove ngram* params in data class.
* add bartcausallm, bartdecoder testing
* correct bart for causal lm
* remove at
* add mbart as well
* up
* fix typo
* up
* correct
* add pegasusforcausallm
* add blenderbotforcausallm
* add blenderbotsmallforcausallm
* add marianforcausallm
* add test for MarianForCausalLM
* add Pegasus test
* add BlenderbotSmall test
* add blenderbot test
* fix a fail
* fix an import fail
* a fix
* fix
* Update modeling_pegasus.py
* fix models
* fix inputs_embeds setting getter
* adapt tests
* correct repo utils check
* finish test improvement
* fix tf models as well
* make style
* make fix-copies
* fix copies
* run all tests
* last changes
* fix all tests
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
2021-02-04 11:56:12 +03:00
0f4dc5d864
fix typo in naming ( #9944 )
2021-02-02 12:22:42 +03:00
538b3b4607
[Tokenizer Utils Base] Make pad function more flexible ( #9928 )
...
* change tokenizer requirement
* split line
* Correct typo from list to str
* improve style
* make other function pretty as well
* add comment
* correct typo
* add new test
* pass tests for tok without padding token
* Apply suggestions from code review
2021-02-02 10:35:27 +03:00
0e3be1ac8f
Add new model docs ( #9667 )
...
* add new model logic
* fix docs
* change structure
* improve add_new_model
* push new changes
* up
* up
* correct spelling
* improve docstring
* correct line length
* update readme
* correct links
* correct typos
* only add rst file for now
* Apply suggestions from code review 1
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: Bram Vanroy <Bram.Vanroy@UGent.be>
* Apply suggestions from code review
Co-authored-by: Bram Vanroy <Bram.Vanroy@UGent.be>
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: Stefan Schweter <stefan@schweter.it>
Co-authored-by: Bram Vanroy <Bram.Vanroy@UGent.be>
* Apply suggestions from code review
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: Pierric Cistac <Pierrci@users.noreply.github.com>
* finish adding all suggestions
* make style
* apply Niels feedback
* Apply suggestions from code review
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* apply sylvains suggestions
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: Bram Vanroy <Bram.Vanroy@UGent.be>
Co-authored-by: Stefan Schweter <stefan@schweter.it>
Co-authored-by: Pierric Cistac <Pierrci@users.noreply.github.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2021-02-01 17:55:10 +03:00
d85691ac75
Doc title in the template ( #9910 )
2021-02-01 03:05:31 -05:00
b4e559cfa1
Deprecate model_path in Trainer.train ( #9854 )
2021-01-28 08:32:46 -05:00
2ee9f9b69e
Fix computation of attention_probs when head_mask is provided. ( #9853 )
...
* Fix computation of attention_probs when head_mask is provided.
Signed-off-by: Morgan Funtowicz <funtowiczmo@gmail.com>
* Apply changes to the template
Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>
2021-01-28 06:11:52 -05:00
763ece2fea
Fix model templates ( #9842 )
2021-01-27 08:20:58 -05:00
bd701ab1a0
Fix template ( #9840 )
2021-01-27 07:40:30 -05:00
4adbdce5ee
Clean TF Bert ( #9788 )
...
* Start cleaning BERT
* Clean BERT and all those depends of it
* Fix attribute name
* Apply style
* Apply Sylvain's comments
* Apply Lysandre's comments
* remove unused import
2021-01-27 11:28:11 +01:00
a1720694a5
Remove a TF usage warning and rework the documentation ( #9756 )
...
* Rework documentation
* Update the template
* Trigger CI
* Restore the warning but with the TF logger
* Update convbert doc
2021-01-27 10:45:42 +01:00
f2fabedbab
Setup logging with a stdout handler ( #9816 )
2021-01-27 03:39:11 -05:00
897a24c869
Fix head_mask for model templates
2021-01-26 11:02:48 +01:00
10e5f28212
Improve pytorch examples for fp16 ( #9796 )
...
* Pad to 8x for fp16 multiple choice example (#9752 )
* Pad to 8x for fp16 squad trainer example (#9752 )
* Pad to 8x for fp16 ner example (#9752 )
* Pad to 8x for fp16 swag example (#9752 )
* Pad to 8x for fp16 qa beam search example (#9752 )
* Pad to 8x for fp16 qa example (#9752 )
* Pad to 8x for fp16 seq2seq example (#9752 )
* Pad to 8x for fp16 glue example (#9752 )
* Pad to 8x for fp16 new ner example (#9752 )
* update script template #9752
* Update examples/multiple-choice/run_swag.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Update examples/question-answering/run_qa.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Update examples/question-answering/run_qa_beam_search.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* improve code quality #9752
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2021-01-26 04:47:07 -05:00
caf4abf768
Auto-resume training from checkpoint ( #9776 )
...
* Auto-resume training from checkpoint
* Update examples/text-classification/run_glue.py
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
* Roll out to other examples
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
2021-01-25 12:03:51 -05:00
a7dabfb3d1
Fix TF s2s models ( #9478 )
...
* Fix Seq2Seq models for serving
* Apply style
* Fix lonfgormer
* Fix mBart/Pegasus/Blenderbot
* Apply style
* Add a main intermediate layer
* Apply style
* Remove import
* Apply tf.function to Longformer
* Fix utils check_copy
* Update S2S template
* Fix BART + Blenderbot
* Fix BlenderbotSmall
* Fix BlenderbotSmall
* Fix BlenderbotSmall
* Fix MBart
* Fix Marian
* Fix Pegasus + template
* Apply style
* Fix common attributes test
* Forgot to fix the LED test
* Apply Patrick's comment on LED Decoder
2021-01-21 17:03:29 +01:00
3f290e6c84
Fix mixed precision in TF models ( #9163 )
...
* Fix Gelu precision
* Fix gelu_fast
* Naming
* Fix usage and apply style
* add TF gelu approximate version
* add TF gelu approximate version
* add TF gelu approximate version
* Apply style
* Fix albert
* Remove the usage of the Activation layer
2021-01-21 07:00:11 -05:00
7251a4736d
Fix template ( #9697 )
2021-01-20 09:04:53 -05:00
14042d560f
New TF embeddings (cleaner and faster) ( #9418 )
...
* Create new embeddings + add to BERT
* Add Albert
* Add DistilBert
* Add Albert + Electra + Funnel
* Add Longformer + Lxmert
* Add last models
* Apply style
* Update the template
* Remove unused imports
* Rename attribute
* Import embeddings in their own model file
* Replace word_embeddings per weight
* fix naming
* Fix Albert
* Fix Albert
* Fix Longformer
* Fix Lxmert Mobilebert and MPNet
* Fix copy
* Fix template
* Update the get weights function
* Update src/transformers/modeling_tf_utils.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Update src/transformers/models/electra/modeling_tf_electra.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* address Sylvain's comments
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2021-01-20 12:08:12 +01:00
7e662e6a3b
Fix model templates and use less than 119 chars ( #9684 )
...
* Fix model templates and use less than 119 chars
* Missing new line
2021-01-19 17:11:22 -05:00
b020a736c3
Update `past_key_values` in GPT-2 ( #9596 )
...
* Update past_key_values in gpt2 (#9391 )
* Update generation_utils, and rename some items
* Update modeling_gpt2 to avoid an error in gradient_checkpointing
* Remove 'reorder_cache' from util and add variations to XLNet, TransfoXL, GPT-2
* Change the location of '_reorder_cache' in modeling files
* Add '_reorder_cache' in modeling_ctrl
* Fix a bug of my last commit in CTRL
* Add '_reorder_cache' to GPT2DoubleHeadsModel
* Manage 'use_cache' in config of test_modeling_gpt2
* Clean up the doc string
* Update src/transformers/models/gpt2/modeling_gpt2.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Fix the doc string (GPT-2, CTRL)
* improve gradient_checkpointing_behavior
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
2021-01-19 16:00:15 +01:00
7f28613213
[TFBart] Split TF-Bart ( #9497 )
...
* make templates ready
* make add_new_model_command_ready
* finish tf bart
* prepare tf mbart
* finish tf bart
* add tf mbart
* add marian
* prep pegasus
* add tf pegasus
* push blenderbot tf
* add blenderbot
* add blenderbot small
* clean-up
* make fix copy
* define blend bot tok
* fix
* up
* make style
* add to docs
* add copy statements
* overwrite changes
* improve
* fix docs
* finish
* fix last slow test
* fix missing git conflict line
* fix blenderbot
* up
* fix blenderbot small
* load changes
* finish copied from
* upload fix
2021-01-12 02:06:32 +01:00
1e3c362235
Fix template ( #9512 )
2021-01-11 08:03:28 -05:00
1243ee7d0c
Full rework of the TF input/output embeddings and bias resizing ( #9193 )
...
* Start rework resizing
* Rework bias/decoder resizing
* Full resizing rework
* Full resizing rework
* Start to update the models with the new approach
* Finish to update the models
* Update all the tests
* Update the template
* Fix tests
* Fix tests
* Test a new approach
* Refactoring
* Refactoring
* Refactoring
* New rework
* Rework BART
* Rework bert+blenderbot
* Rework CTRL
* Rework Distilbert
* Rework DPR
* Rework Electra
* Rework Flaubert
* Rework Funnel
* Rework GPT2
* Rework Longformer
* Rework Lxmert
* Rework marian+mbart
* Rework mobilebert
* Rework mpnet
* Rework openai
* Rework pegasus
* Rework Roberta
* Rework T5
* Rework xlm+xlnet
* Rework template
* Fix TFT5EncoderOnly + DPRs
* Restore previous methods
* Fix Funnel
* Fix CTRL and TransforXL
* Apply style
* Apply Sylvain's comments
* Restore a test in DPR
* Address the comments
* Fix bug
* Apply style
* remove unused import
* Fix test
* Forgot a method
* missing test
* Trigger CI
* naming update
* Rebase
* Trigger CI
2021-01-11 06:27:28 -05:00
cf416764f4
Fix template ( #9504 )
2021-01-11 05:21:25 -05:00
4f7022d68d
Reformat ( #9482 )
2021-01-10 15:10:15 +01:00
1bdf42409c
Fast imports part 3 ( #9474 )
...
* New intermediate inits
* Update template
* Avoid importing torch/tf/flax in tokenization unless necessary
* Styling
* Shutup flake8
* Better python version check
2021-01-08 07:40:59 -05:00
758ed3332b
Transformers fast import part 2 ( #9446 )
...
* Main init work
* Add version
* Change from absolute to relative imports
* Fix imports
* One more typo
* More typos
* Styling
* Make quality script pass
* Add necessary replace in template
* Fix typos
* Spaces are ignored in replace for some reason
* Forgot one models.
* Fixes for import
Co-authored-by: LysandreJik <lysandre.debut@reseau.eseo.fr>
* Add documentation
* Styling
Co-authored-by: LysandreJik <lysandre.debut@reseau.eseo.fr>
2021-01-07 09:36:14 -05:00
812045adcc
New serving ( #9419 )
...
* Add a serving method
* Add albert
* Add serving for BERT and BART
* Add more models
* Finish the serving addition
* Temp fix
* Restore DPR
* Fix funnel attribute
* Fix attributes GPT2
* Fix OpenAIGPT attribute
* Fix T5 attributes
* Fix Bart attributes
* Fix TransfoXL attributes
* Add versioning
* better test
* Update template
* Fix Flaubert
* Fix T5
* Apply style
* Remove unused imports
* Deactivate extra parameters
* Remove too long test + saved_model default to False
* Ignore the saved model test for some models
* Fix some inputs
* Fix mpnet serving
* Trigger CI
* Address all comments
2021-01-07 11:48:49 +01:00
b8462b5b2a
[GenerationOutputs] Fix GenerationOutputs Tests ( #9443 )
...
* fix generation models
* fix led
* fix docs
* add is_decoder
* fix last docstrings
* make style
* fix t5 cross attentions
* correct t5
2021-01-06 19:37:02 +01:00
453a70d4cb
Allow example to use a revision and work with private models ( #9407 )
...
* Allow example to use a revision and work with private models
* Copy to other examples and template
* Styling
2021-01-06 06:49:23 -05:00
eef66035a2
[PyTorch Bart] Split Bart into different models ( #9343 )
...
* first try
* remove old template
* finish bart
* finish mbart
* delete unnecessary line
* init pegasus
* save intermediate
* correct pegasus
* finish pegasus
* remove cookie cutter leftover
* add marian
* finish blenderbot
* replace in file
* correctly split blenderbot
* delete "old" folder
* correct "add statement"
* adapt config for tf comp
* correct configs for tf
* remove ipdb
* fix more stuff
* fix mbart
* push pegasus fix
* fix mbart
* more fixes
* fix research projects code
* finish docs for bart, mbart, and marian
* delete unnecessary file
* correct attn typo
* correct configs
* remove pegasus for seq class
* correct peg docs
* correct peg docs
* finish configs
* further improve docs
* add copied from statements to mbart
* fix copied from in mbart
* add copy statements to marian
* add copied from to marian
* add pegasus copied from
* finish pegasus
* finish copied from
* Apply suggestions from code review
* make style
* backward comp blenderbot
* apply lysandres and sylvains suggestions
* apply suggestions
* push last fixes
* fix docs
* fix tok tests
* fix imports code style
* fix doc
2021-01-05 22:00:05 +01:00
912f6881d2
add import math ( #9346 )
2020-12-29 19:35:06 +01:00
785e52cd30
improve templates ( #9342 )
2020-12-29 16:48:44 +01:00
Patrick von Platen