651408a077
* try to stylify using ruff * might need to remove these changes? * use ruf format andruff check * use isinstance instead of type comparision * use # fmt: skip * use # fmt: skip * nits * soem styling changes * update ci job * nits isinstance * more files update * nits * more nits * small nits * check and format * revert wrong changes * actually use formatter instead of checker * nits * well docbuilder is overwriting this commit * revert notebook changes * try to nuke docbuilder * style * fix feature exrtaction test * remve `indent-width = 4` * fixup * more nits * update the ruff version that we use * style * nuke docbuilder styling * leve the print for detected changes * nits * Remove file I/O Co-authored-by: charliermarsh <charlie.r.marsh@gmail.com> * style * nits * revert notebook changes * Add # fmt skip when possible * Add # fmt skip when possible * Fix * More ` # fmt: skip` usage * More ` # fmt: skip` usage * More ` # fmt: skip` usage * NIts * more fixes * fix tapas * Another way to skip * Recommended way * Fix two more fiels * Remove asynch Remove asynch --------- Co-authored-by: charliermarsh <charlie.r.marsh@gmail.com> |
||
|---|---|---|
| .. | ||
| cookiecutter-template-{{cookiecutter.modelname}} | ||
| README.md | ||
| cookiecutter.json | ||
README.md
This folder contains a template to add a tokenization test.
Usage
Using the cookiecutter utility requires to have all the dev dependencies installed.
Let's first fork the transformers repo on github. Once it's done you can clone your fork and install transformers in our environment:
git clone https://github.com/YOUR-USERNAME/transformers
cd transformers
pip install -e ".[dev]"
Once the installation is done, you can generate the template by running the following command. Be careful, the template will be generated inside a new folder in your current working directory.
cookiecutter path-to-the folder/adding_a_missing_tokenization_test/
You will then have to answer some questions about the tokenizer for which you want to add tests. The modelname should be cased according to the plain text casing, i.e., BERT, RoBERTa, DeBERTa.
Once the command has finished, you should have a one new file inside the newly created folder named test_tokenization_Xxx.py. At this point the template is finished and you can move it to the sub-folder of the corresponding model in the test folder.