transformers/requirements.txt

# progress bars in model download and training scripts
tqdm
# Accessing files from S3 directly.
boto3
# Used for downloading models over HTTP
requests
# For OpenAI GPT
regex
# For XLNet
sentencepiece
# For XLM
sacremoses
preparing for first release 2018-11-16 03:56:10 +08:00			`# progress bars in model download and training scripts`
bump up version minor 2018-11-17 17:43:39 +08:00			`tqdm`
preparing for first release 2018-11-16 03:56:10 +08:00			`# Accessing files from S3 directly.`
			`boto3`
			`# Used for downloading models over HTTP`
forgot to add regex to requirements :( 2019-02-18 18:54:51 +08:00			`requests`
			`# For OpenAI GPT`
add tokenizer and tests 2019-06-21 17:09:51 +08:00			`regex`
			`# For XLNet`
Tokenization behave the same as original XLM proprocessing for most languages except zh, ja and th; Change API to allow specifying language in `tokenize` 2019-08-24 02:40:17 +08:00			`sentencepiece`
			`# For XLM`
Match order of casing in OSS XLM; Improve document; Clean up dependency 2019-08-28 08:03:18 +08:00			`sacremoses`