dnrops
/
transformers
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[s2s] dont document packing because it hurts performance (#6077)

This commit is contained in:
Sam Shleifer 2020-07-27 18:26:00 -04:00 committed by GitHub
parent 9d0d3a6645
commit 1e00ef681d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
examples/seq2seq/README.md
View File

@ -27,17 +27,7 @@ this should make a directory called `cnn_dm/` with files like `test.source`.
```
WMT16 English-Romanian Translation Data:
This dataset comes in two formats. The "packed" version merges short training examples into examples of <200 tokens to increase GPU utilization (and also improves validation performance).
```bash
cd examples/seq2seq
wget https://s3.amazonaws.com/datasets.huggingface.co/translation/wmt_en_ro_packed_train_200.tgz
tar -xzvf wmt_en_ro_packed_200.tgz
export ENRO_DIR=wmt_en_ro_packed_train_200
```
The original data can also be downloaded with this command:
download with this command:
```bash
wget https://s3.amazonaws.com/datasets.huggingface.co/translation/wmt_en_ro.tar.gz
tar -xzvf wmt_en_ro.tar.gz