transformers/docs/source/en/model_doc/openai-gpt.md

OpenAI GPT

Overview

OpenAI GPT model was proposed in Improving Language Understanding by Generative Pre-Training by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. It's a causal (unidirectional) transformer pre-trained using language modeling on a large corpus will long range dependencies, the Toronto Book Corpus.

The abstract from the paper is the following:

Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pretraining of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture. We demonstrate the effectiveness of our approach on a wide range of benchmarks for natural language understanding. Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied.

Write With Transformer is a webapp created and hosted by Hugging Face showcasing the generative capabilities of several models. GPT is one of them.

This model was contributed by thomwolf. The original code can be found here.

Usage tips

• GPT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left.
• GPT was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next token in a sequence. Leveraging this feature allows GPT-2 to generate syntactically coherent text as it can be observed in the run_generation.py example script.

Note:

If you want to reproduce the original tokenization process of the OpenAI GPT paper, you will need to install ftfy and SpaCy:

pip install spacy ftfy==4.4.3
python -m spacy download en

If you don't install ftfy and SpaCy, the [OpenAIGPTTokenizer] will default to tokenize using BERT's BasicTokenizer followed by Byte-Pair Encoding (which should be fine for most usage, don't worry).

Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with OpenAI GPT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

• A blog post on outperforming OpenAI GPT-3 with SetFit for text-classification.
• See also: Text classification task guide

• A blog on how to Finetune a non-English GPT-2 Model with Hugging Face.
• A blog on How to generate text: using different decoding methods for language generation with Transformers with GPT-2.
• A blog on Training CodeParrot 🦜 from Scratch, a large GPT-2 model.
• A blog on Faster Text Generation with TensorFlow and XLA with GPT-2.
• A blog on How to train a Language Model with Megatron-LM with a GPT-2 model.
• A notebook on how to finetune GPT2 to generate lyrics in the style of your favorite artist. 🌎
• A notebook on how to finetune GPT2 to generate tweets in the style of your favorite Twitter user. 🌎
• Causal language modeling chapter of the 🤗 Hugging Face Course.
• [OpenAIGPTLMHeadModel] is supported by this causal language modeling example script, text generation example script and notebook.
• [TFOpenAIGPTLMHeadModel] is supported by this causal language modeling example script and notebook.
• See also: Causal language modeling task guide

• A course material on Byte-Pair Encoding tokenization.

OpenAIGPTConfig

autodoc OpenAIGPTConfig

OpenAIGPTTokenizer

autodoc OpenAIGPTTokenizer - save_vocabulary

OpenAIGPTTokenizerFast

autodoc OpenAIGPTTokenizerFast

OpenAI specific outputs

autodoc models.openai.modeling_openai.OpenAIGPTDoubleHeadsModelOutput

autodoc models.openai.modeling_tf_openai.TFOpenAIGPTDoubleHeadsModelOutput

OpenAIGPTModel

autodoc OpenAIGPTModel - forward

OpenAIGPTLMHeadModel

autodoc OpenAIGPTLMHeadModel - forward

OpenAIGPTDoubleHeadsModel

autodoc OpenAIGPTDoubleHeadsModel - forward

OpenAIGPTForSequenceClassification

autodoc OpenAIGPTForSequenceClassification - forward

TFOpenAIGPTModel

autodoc TFOpenAIGPTModel - call

TFOpenAIGPTLMHeadModel

autodoc TFOpenAIGPTLMHeadModel - call

TFOpenAIGPTDoubleHeadsModel

autodoc TFOpenAIGPTDoubleHeadsModel - call

TFOpenAIGPTForSequenceClassification

autodoc TFOpenAIGPTForSequenceClassification - call