6eb51450fa
* Finished QA example * Dodge a merge conflict * Update text classification and LM examples * Update NER example * New Keras metrics WIP, fix NER example * Update NER example * Update MC, summarization and translation examples * Add XLA warnings when shapes are variable * Make sure batch_size is consistently scaled by num_replicas * Add PushToHubCallback to all models * Add docs links for KerasMetricCallback * Add docs links for prepare_tf_dataset and jit_compile * Correct inferred model names * Don't assume the dataset has 'lang' * Don't assume the dataset has 'lang' * Write metrics in text classification * Add 'framework' to TrainingArguments and TFTrainingArguments * Export metrics in all examples and add tests * Fix training args for Flax * Update command line args for translation test * make fixup * Fix accidentally running other tests in fp16 * Remove do_train/do_eval from run_clm.py * Remove do_train/do_eval from run_mlm.py * Add tensorflow tests to circleci * Fix circleci * Update examples/tensorflow/language-modeling/run_mlm.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update examples/tensorflow/test_tensorflow_examples.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update examples/tensorflow/translation/run_translation.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update examples/tensorflow/token-classification/run_ner.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Fix save path for tests * Fix some model card kwargs * Explain the magical -1000 * Actually enable tests this time * Skip text classification PR until we fix shape inference * make fixup Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> |
||
|---|---|---|
| .. | ||
| benchmarking | ||
| language-modeling | ||
| multiple-choice | ||
| question-answering | ||
| summarization | ||
| text-classification | ||
| token-classification | ||
| translation | ||
| README.md | ||
| _tests_requirements.txt | ||
| test_tensorflow_examples.py | ||
README.md
Examples
This folder contains actively maintained examples of use of 🤗 Transformers organized into different ML tasks. All examples in this folder are TensorFlow examples, and are written using native Keras rather than classes like TFTrainer, which we now consider deprecated. If you've previously only used 🤗 Transformers via TFTrainer, we highly recommend taking a look at the new style - we think it's a big improvement!
In addition, all scripts here now support the 🤗 Datasets library - you can grab entire datasets just by changing one command-line argument!
A note on code folding
Most of these examples have been formatted with #region blocks. In IDEs such as PyCharm and VSCode, these blocks mark named regions of code that can be folded for easier viewing. If you find any of these scripts overwhelming or difficult to follow, we highly recommend beginning with all regions folded and then examining regions one at a time!
The Big Table of Tasks
Here is the list of all our examples:
| Task | Example datasets |
|---|---|
language-modeling |
WikiText-2 |
multiple-choice |
SWAG |
question-answering |
SQuAD |
summarization |
XSum |
text-classification |
GLUE |
token-classification |
CoNLL NER |
translation |
WMT |
Coming soon
- Colab notebooks to easily run through these scripts!