目录
- notebooks small fix3年前
- tapas[TAPAS] Add tableformer readme2年前
- AUTHORSAdds a stopping criterion to the evaluation.4年前
- CONTRIBUTING.mdAdds a stopping criterion to the evaluation.4年前
- DENSE_TABLE_RETRIEVER.md small fix3年前
- DOT.md small fix3年前
- INTERMEDIATE_PRETRAIN_DATA.mdUpdate readme2年前
- LICENSEAdds a stopping criterion to the evaluation.4年前
- MANIFEST.inAdds a stopping criterion to the evaluation.4年前
- MATE.mdHybridQA training2年前
- PRETRAIN_DATA.mdUpdates citation.4年前
- README.mdAdd MATE news section3年前
- TABLEFORMER.md[TAPAS] Add tableformer readme2年前
- requirements.txtAdd MATE news section3年前
- setup.pyAdds documentation for running and citing TableFormer.2年前
- tox.iniIncrease apache-beam version to 2.28.0 to support py38.3年前
邀请码
TAble PArSing (TAPAS)
Code and checkpoints for training the transformer-based Table QA models introduced in the paper TAPAS: Weakly Supervised Table Parsing via Pre-training.
News
2021/09/15
2021/08/24
2021/08/20
2021/07/23
2021/05/13
2021/03/23
2020/12/17
2020/10/19
2020/10/09
2020/08/26
2020/08/05
reset_position_index_per_cell: New option that allows to train models that instead of using absolute position indices reset the position index when a new cell starts.2020/06/10
2020/06/08
2020/05/07
Installation
The easiest way to try out TAPAS with free GPU/TPU is in our Colab, which shows how to do predictions on SQA.
The repository uses protocol buffers, and requires the
protoccompiler to run. You can download the latest binary for your OS here. On Ubuntu/Debian, it can be installed with:Afterwards, clone and install the git repository:
To run the test suite we use the tox library which can be run by calling:
Models
We provide pre-trained models for different model sizes.
The metrics are computed by our tool and not the official metrics of the respective tasks. We provide them so one can verify whether one’s own runs are in the right ballpark. They are medians over three individual runs.
Models with intermediate pre-training (2020/10/07).
New models based on the ideas discussed in Understanding tables with intermediate pre-training. Learn more about the methods use here.
WTQ
Trained from Mask LM, intermediate data, SQA, WikiSQL.
WIKISQL
Trained from Mask LM, intermediate data, SQA.
TABFACT
Trained from Mask LM, intermediate data.
SQA
Trained from Mask LM, intermediate data.
INTERMEDIATE
Trained from Mask LM.
Small Models & position index reset (2020/08/08)
Based on the pre-trained checkpoints available at the BERT github page. See the page or the paper for detailed information on the model dimensions.
Reset refers to whether the parameter
reset_position_index_per_cellwas set to true or false during training. In general it’s recommended to set it to true.The accuracy depends on the respective task. It’s denotation accuracy for WTQ and WIKISQL, average position accuracy with gold labels for the previous answers for SQA and Mask-LM accuracy for Mask-LM.
The models were trained in a chain as indicated by the model name. For example, sqa_masklm means the model was first trained on the Mask-LM task and then on SQA. No destillation was performed.
WTQ
WIKISQL
SQA
MASKLM
Original Models
The pre-trained TAPAS checkpoints can be downloaded here:
The first two models are pre-trained on the Mask-LM task and the last two on the Mask-LM task first and SQA second.
Fine-Tuning Data
You also need to download the task data for the fine-tuning tasks:
Pre-Training
Note that you can skip pre-training and just use one of the pre-trained checkpoints provided above.
Information about the pre-taining data can be found here.
The TF examples for pre-training can be created using Google Dataflow:
You can also run the pipeline locally but that will take a long time:
This will create two tfrecord files for training and testing. The pre-training can then be started with the command below. The init checkpoint should be a standard BERT checkpoint.
Where compression_type should be set to GZIP if the tfrecords are compressed. You can start a separate eval job by setting
--nodo_train --doeval.Running a fine-tuning task
We need to create the TF examples before starting the training. For example, for SQA that would look like:
Optionally, to handle big tables, we can add a
--prune_columnsflag to apply the HEM method described section 3.3 of our paper to discard some columns based on textual overlap with the sentence.Afterwards, training can be started by running:
This will use the preset hyper-parameters set in
hparam_utils.py.It’s recommended to start a separate eval job to continuously produce predictions for the checkpoints created by the training job. Alternatively, you can run the eval job after training to only get the final results.
Another tool to run experiments is
tapas_classifier_experiment.py. It’s more flexible thanrun_task_main.pybut also requires setting all the hyper-parameters (via the respective command line flags).Evaluation
Here we explain some details about different tasks.
SQA
By default, SQA will evaluate using the reference answers of the previous questions. The number in the paper (Table 5) are computed using the more realistic setup where the previous answer are model predictions.
run_task_main.pywill output additional prediction files for this setup as well if run on GPU.WTQ
For the official evaluation results one should convert the TAPAS predictions to the WTQ format and run the official evaluation script. This can be done using
convert_predictions.py.WikiSQL
As discussed in the paper our code will compute evaluation metrics that deviate from the official evaluation script (Table 3 and 10).
Hardware Requirements
TAPAS is essentialy a BERT model and thus has the same requirements. This means that training the large model with 512 sequence length will require a TPU. You can use the option
max_seq_lengthto create shorter sequences. This will reduce accuracy but also make the model trainable on GPUs. Another option is to reduce the batch size (train_batch_size), but this will likely also affect accuracy. We added an optionsgradient_accumulation_stepsthat allows you to split the gradient over multiple batches. Evaluation with the default test batch size (32) should be possible on GPU.How to cite TAPAS?
You can cite the ACL 2020 paper and the EMNLP 2020 Findings paper for the laters work on pre-training objectives.
Disclaimer
This is not an official Google product.
Contact information
For help or issues, please submit a GitHub issue.