19420fd99e
* move test model folders (TODO: fix imports and others) * fix (potentially partially) imports (in model test modules) * fix (potentially partially) imports (in tokenization test modules) * fix (potentially partially) imports (in feature extraction test modules) * fix import utils.test_modeling_tf_core * fix path ../fixtures/ * fix imports about generation.test_generation_flax_utils * fix more imports * fix fixture path * fix get_test_dir * update module_to_test_file * fix get_tests_dir from wrong transformers.utils * update config.yml (CircleCI) * fix style * remove missing imports * update new model script * update check_repo * update SPECIAL_MODULE_TO_TEST_MAP * fix style * add __init__ * update self-scheduled * fix add_new_model scripts * check one way to get location back * python setup.py build install * fix import in test auto * update self-scheduled.yml * update slack notification script * Add comments about artifact names * fix for yolos Co-authored-by: ydshieh <ydshieh@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| cookiecutter-template-{{cookiecutter.modelname}} | ||
| README.md | ||
| cookiecutter.json | ||
README.md
This folder contains a template to add a tokenization test.
Usage
Using the cookiecutter utility requires to have all the dev dependencies installed.
Let's first fork the transformers repo on github. Once it's done you can clone your fork and install transformers in our environment:
git clone https://github.com/YOUR-USERNAME/transformers
cd transformers
pip install -e ".[dev]"
Once the installation is done, you can generate the template by running the following command. Be careful, the template will be generated inside a new folder in your current working directory.
cookiecutter path-to-the folder/adding_a_missing_tokenization_test/
You will then have to answer some questions about the tokenizer for which you want to add tests. The modelname should be cased according to the plain text casing, i.e., BERT, RoBERTa, DeBERTa.
Once the command has finished, you should have a one new file inside the newly created folder named test_tokenization_Xxx.py. At this point the template is finished and you can move it to the sub-folder of the corresponding model in the test folder.