1b5ce1e63b
Development on v4.5.0dev0
2021-03-16 11:41:15 -04:00
c988db5af2
Release v4.4.0
2021-03-16 11:33:35 -04:00
5c02b97ca2
Fix URLs from #10744 ( #10748 )
2021-03-16 11:31:29 -04:00
a0a027c2ed
Add DistributedSamplerWithLoop ( #10746 )
...
* Add DistributedSamplerWithLoop
* Fix typo
* Test and small fix
2021-03-16 11:22:39 -04:00
1449222217
Fix DeBERTa + Conversational pipeline slow tests ( #10743 )
...
* Fix DeBERTa-v2 variable assignment
* Fix conversational pipeline test
2021-03-16 11:18:20 -04:00
d3d388b934
fix M2M100 example ( #10745 )
2021-03-16 20:20:00 +05:30
b5492582d0
Remove old links to CDN ( #10744 )
2021-03-16 10:48:53 -04:00
5dcc08f1df
Fix S2T example ( #10741 )
2021-03-16 08:55:07 -04:00
813d730c46
Release utils ( #10735 )
...
* Examples version update
* Refactor a bit
* All version updates
* Fixes
* README cleanup
* Post-release/patch
* Fixes
* More fixes
* Tests
* More fixes
* Moar fixes
* Make commands and update setup
* Replace spaces with weird tabs
* Fix test
* Style
2021-03-16 08:41:47 -04:00
9f8619c6aa
Flax testing should not run the full torch test suite ( #10725 )
...
* make flax tests pytorch independent
* fix typo
* finish
* improve circle ci
* fix return tensors
* correct flax test
* re-add sentencepiece
* last tokenizer fixes
* finish maybe now
2021-03-16 08:05:37 +03:00
87d685b8a9
independent training / eval with local files ( #10710 )
...
* independent training / eval with local files
* remove redundant assert
2021-03-15 19:35:26 -04:00
4c379daf64
Add minimum version check in examples ( #10724 )
...
* Add minimum version check in examples
* Style
* No need for new line maybe?
* Add helpful comment
2021-03-15 19:29:54 -04:00
966ba081c9
zero-shot pipeline multi_class -> multi_label ( #10727 )
2021-03-15 16:02:46 -06:00
58f672e65c
Tests run on Docker ( #10681 )
...
* Tests run on Docker
Co-authored-by: Morgan <funtowiczmo@gmail.com>
* Comments from code review
* Reply to itself
* Dependencies
Co-authored-by: Morgan <funtowiczmo@gmail.com>
2021-03-15 17:28:01 -04:00
d41dd5359b
[Wav2Vec2] Fix documentation inaccuracy ( #10694 )
...
* Update super class reference
* Update default value reference
* Update src/transformers/models/wav2vec2/feature_extraction_wav2vec2.py
* Fix format style
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
2021-03-15 20:11:17 +03:00
f5c097fc4d
Fix backward compatibility with EvaluationStrategy ( #10718 )
2021-03-15 10:20:38 -04:00
d9e693e1d0
make wav2vec2 test deterministic ( #10714 )
2021-03-15 09:50:05 -04:00
6bef764506
Multiple fixes in SageMakerTrainer ( #10687 )
...
* Handle save differently
* Missing imports
* Fix typo
* Adapt to recent changes in save_pretrained
* Forgotten brackets
* Optimizer load
* Fix world size
* Deal wth None
* Remove needless self
2021-03-15 09:28:15 -04:00
3f1714f8a7
Adding required flags to non-default arguments in hf_argparser ( #10688 )
...
* Adding required flags to non-default arguments.
Signed-off-by: Adam Pocock <adam.pocock@oracle.com>
* make style fix.
* Update src/transformers/hf_argparser.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2021-03-15 09:27:55 -04:00
6f840990a7
split seq2seq script into summarization & translation ( #10611 )
...
* split seq2seq script, update docs
* needless diff
* fix readme
* remove test diff
* s/summarization/translation
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* cr
* fix arguments & better mbart/t5 refs
* copyright
Co-authored-by: Suraj Patil <surajp815@gmail.com>
* reword readme
Co-authored-by: Suraj Patil <surajp815@gmail.com>
* s/summarization/translation
* short script names
* fix tests
* fix isort, include mbart doc
* delete old script, update tests
* automate source prefix
* automate source prefix for translation
* s/translation/trans
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* fix script name (short version)
* typos
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* exact parameter
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* remove superfluous source_prefix calls in docs
* rename scripts & warn for source prefix
* black
* flake8
Co-authored-by: theo <theo@matussie.re>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Suraj Patil <surajp815@gmail.com>
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
2021-03-15 09:11:42 -04:00
505494a86f
GPT2DoubleHeadsModel made parallelizable ( #10658 )
...
* GPT2DoubleHeadsModel made parallelizeable
* GPT2DoubleHeadsModel added as parallelizeable onto the GPT2 test suite
2021-03-15 09:10:44 -04:00
e12d6f513e
Distributed barrier before loading model ( #10685 )
2021-03-15 08:28:15 -04:00
339fc51acc
fix styling
2021-03-15 07:59:35 -04:00
4c41c6622c
Wrong link to super class ( #10709 )
...
Documentation was referring to slow tokenizer class while it should be the fast tokenizer.
2021-03-15 07:39:10 -04:00
fcf10214e0
enable loading Mbart50Tokenizer with AutoTokenizer ( #10690 )
...
* enable auto tokenizer for mbart50 tokenizers
* fix imports
2021-03-15 16:20:37 +05:30
bd8f6cafd4
make rag tests smaller ( #10679 )
2021-03-15 10:07:12 +03:00
4c32f9f26e
AdamW is now supported by default ( #9624 )
2021-03-12 13:40:07 -08:00
fa35cda91e
Pass encoder outputs into GenerationMixin ( #10599 )
...
* Pass encoder_outputs into generate()
* Remove an if-statement
* Reformat
* Minimize changes to generate()
* Comment on input_ids
2021-03-12 21:43:11 +05:30
00cad2e5c1
fix : #10628 expanduser path in TrainingArguments ( #10660 )
...
* fix : #10628 expanduser path in TrainingArguments
* docs: explain why we expand paths in TrainingArguments
* Style
Co-authored-by: Sylvain Gugger <sylvain.gugger@gmail.com>
2021-03-12 09:18:19 -05:00
e8246f78f9
Add auto_wrap option in fairscale integration ( #10673 )
...
* Add auto_wrap option in fairscale integration
* Style
2021-03-12 07:50:20 -05:00
184ef8ecd0
TensorFlow tests: having from_pt set to True requires torch to be installed. ( #10664 )
...
* TF model exists for Blenderbot 400M
* Marian
* RAG
2021-03-12 14:16:40 +03:00
543d0549f8
Adding new parameter to `generate`: `max_time`. ( #9846 )
...
* [WIP] Adding new parameter to `generate`: `max_time`.
Generation by tokens number is sometimes a bit clunky because we don't
know how many tokens are good enough or even how many tokens are in
the payload (for pipelines users for instance). This leads to hard
to understand behavior.
This PR proposes a new argument `max_time` which is a float of seconds
for the allowed time for `generate` to run on.
Ideally combinations of `max_tokens=None`, `max_time=2` could be used to
generate as many tokens as possible within time budget.
NB: Another possible approach consists of passing a callback to `generate`
putting the caller in charge of the actual decision of when to stop
generating tokens. It opens the door to 'which args should we pass'
to this callback. It's hard to imagine other use-cases for this
early stopping behavior than time (that are not already covered by
parameters of generate)
* Revamp with StoppingCriteria
* Removing deprecated mentions.
* Forgot arguments to stopping criteria.
* Readding max_length it's not just used as a stopping criteria.
* Default value for `stopping_criteria`.
* Address @patrickvonplaten comments.
- More docstrings
- Actual doc
- Include in global namespace
- Remove TF work.
* Put back `max_length` (deprecation different PR).
* Doc quality.
* Fixing old behavior without `stopping_criteria` but with `max_length`.
Making sure we don't break that in the future.
* Adding more tests for possible inconsistencies between
`max_length` and `stopping_criteria`.
* Fixing the torch imports.
2021-03-12 10:11:50 +01:00
ea46e3fa9c
Adjust loss difference ( #10669 )
2021-03-12 09:09:46 +03:00
c526bde319
fix typing error for HfArgumentParser for Optional[bool] ( #10672 )
...
* fix typing error for TrainingArguments Optional[bool]
* updating equality check for Optional[bool]
2021-03-11 17:42:54 -05:00
fa1a8d102f
Tentative fix for HFArgumentParser in Python 3.8
2021-03-11 14:44:29 -05:00
2f8485199c
Fix broken link ( #10656 )
...
* Fixed broken link
* fixed max length violation
Co-authored-by: WybeKoper <WybeKoper@users.noreply.github.com>
2021-03-11 14:29:02 -05:00
a01ea31b5c
Add DeBERTa to MODEL_FOR_PRETRAINING_MAPPING ( #10668 )
...
* add deberta to pretraining mapping
* add deberta_v2 to PRETRAINING_MAPPING
2021-03-11 13:56:47 -05:00
9fbb4cdc80
Specify minimum version for sacrebleu ( #10662 )
2021-03-11 13:45:06 -05:00
fda703a553
Fix integration slow tests ( #10670 )
...
* PoC
* Fix slow tests for the PT1.8 Embedding problem
2021-03-11 13:43:53 -05:00
3ab6820370
Onnx fix test ( #10663 )
...
* Allow to pass kwargs to model's from_pretrained when using pipeline.
* Disable the use of past_keys_values for GPT2 when exporting to ONNX.
* style
* Remove comment.
* Appease the documentation gods
* Fix style
Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>
2021-03-11 13:38:29 -05:00
a637ae00c4
Fixes Pegasus tokenization tests ( #10671 )
2021-03-11 13:35:50 -05:00
7e4428749c
Conversion to tensors requires padding ( #10661 )
2021-03-11 12:58:15 -05:00
2adc8c926a
W2v2 test require torch ( #10665 )
...
* Adds a @require_torch to a test that requires it
* Tokenizer too
* Style
2021-03-11 12:56:12 -05:00
055ed78f52
[S2T] fix example in docs ( #10667 )
2021-03-11 22:43:37 +05:30
89693e170d
Remove special treatment for custom vocab files ( #10637 )
...
* Remove special path for custom vocab files
* Update src/transformers/tokenization_utils_base.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Expand error message
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
2021-03-11 11:11:56 -05:00
6d9e11a193
S2S + M2M100 should be available in tokenization_auto ( #10657 )
...
* S2S + M2M100 should be available in tokenization_auto
* Requires sentencepiece
* SentencePiece for S2T as well :)
2021-03-11 09:53:36 -05:00
602d63f05c
[XLSR-Wav2Vec2] Add multi-lingual Wav2Vec2 models ( #10648 )
...
* add conversion script
* add wav2vec2 xslr models
* finish
* Update docs/source/model_doc/xlsr_wav2vec2.rst
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2021-03-11 17:44:18 +03:00
63c295ac05
Ensure metric results are JSON-serializable ( #10632 )
2021-03-11 09:00:23 -05:00
27d9e05ce2
Update README.md ( #10647 )
...
correct spell error: 'nether'
2021-03-11 08:58:04 -05:00
053f0197b8
merge_file -> merges_file ( #10653 )
2021-03-11 08:34:08 -05:00
Lysandre