77 lines
4.1 KiB
Markdown
77 lines
4.1 KiB
Markdown
<!--Copyright 2023 The HuggingFace Team. All rights reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
|
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations under the License.
|
|
|
|
⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
|
|
rendered properly in your Markdown viewer.
|
|
|
|
-->
|
|
|
|
# Pix2Struct
|
|
|
|
## Overview
|
|
|
|
The Pix2Struct model was proposed in [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://arxiv.org/abs/2210.03347) by Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, Kristina Toutanova.
|
|
|
|
The abstract from the paper is the following:
|
|
|
|
> Visually-situated language is ubiquitous -- sources range from textbooks with diagrams to web pages with images and tables, to mobile apps with buttons and forms. Perhaps due to this diversity, previous work has typically relied on domain-specific recipes with limited sharing of the underlying data, model architectures, and objectives. We present Pix2Struct, a pretrained image-to-text model for purely visual language understanding, which can be finetuned on tasks containing visually-situated language. Pix2Struct is pretrained by learning to parse masked screenshots of web pages into simplified HTML. The web, with its richness of visual elements cleanly reflected in the HTML structure, provides a large source of pretraining data well suited to the diversity of downstream tasks. Intuitively, this objective subsumes common pretraining signals such as OCR, language modeling, image captioning. In addition to the novel pretraining strategy, we introduce a variable-resolution input representation and a more flexible integration of language and vision inputs, where language prompts such as questions are rendered directly on top of the input image. For the first time, we show that a single pretrained model can achieve state-of-the-art results in six out of nine tasks across four domains: documents, illustrations, user interfaces, and natural images.
|
|
|
|
Tips:
|
|
|
|
Pix2Struct has been fine tuned on a variety of tasks and datasets, ranging from image captioning, visual question answering (VQA) over different inputs (books, charts, science diagrams), captioning UI components etc. The full list can be found in Table 1 of the paper.
|
|
We therefore advise you to use these models for the tasks they have been fine tuned on. For instance, if you want to use Pix2Struct for UI captioning, you should use the model fine tuned on the UI dataset. If you want to use Pix2Struct for image captioning, you should use the model fine tuned on the natural images captioning dataset and so on.
|
|
|
|
If you want to use the model to perform conditional text captioning, make sure to use the processor with `add_special_tokens=False`.
|
|
|
|
This model was contributed by [ybelkada](https://huggingface.co/ybelkada).
|
|
The original code can be found [here](https://github.com/google-research/pix2struct).
|
|
|
|
## Resources
|
|
|
|
- [Fine-tuning Notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynb)
|
|
- [All models](https://huggingface.co/models?search=pix2struct)
|
|
|
|
## Pix2StructConfig
|
|
|
|
[[autodoc]] Pix2StructConfig
|
|
- from_text_vision_configs
|
|
|
|
## Pix2StructTextConfig
|
|
|
|
[[autodoc]] Pix2StructTextConfig
|
|
|
|
## Pix2StructVisionConfig
|
|
|
|
[[autodoc]] Pix2StructVisionConfig
|
|
|
|
## Pix2StructProcessor
|
|
|
|
[[autodoc]] Pix2StructProcessor
|
|
|
|
## Pix2StructImageProcessor
|
|
|
|
[[autodoc]] Pix2StructImageProcessor
|
|
- preprocess
|
|
|
|
## Pix2StructTextModel
|
|
|
|
[[autodoc]] Pix2StructTextModel
|
|
- forward
|
|
|
|
## Pix2StructVisionModel
|
|
|
|
[[autodoc]] Pix2StructVisionModel
|
|
- forward
|
|
|
|
## Pix2StructForConditionalGeneration
|
|
|
|
[[autodoc]] Pix2StructForConditionalGeneration
|
|
- forward |