62 lines
2.7 KiB
Markdown
62 lines
2.7 KiB
Markdown
<!--Copyright 2023 The HuggingFace Team. All rights reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
|
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations under the License.
|
|
|
|
⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
|
|
rendered properly in your Markdown viewer.
|
|
|
|
-->
|
|
|
|
# MRA
|
|
|
|
## Overview
|
|
|
|
The MRA model was proposed in [Multi Resolution Analysis (MRA) for Approximate Self-Attention](https://arxiv.org/abs/2207.10284) by Zhanpeng Zeng, Sourav Pal, Jeffery Kline, Glenn M Fung, and Vikas Singh.
|
|
|
|
The abstract from the paper is the following:
|
|
|
|
*Transformers have emerged as a preferred model for many tasks in natural language processing and vision. Recent efforts on training and deploying Transformers more efficiently have identified many strategies to approximate the self-attention matrix, a key module in a Transformer architecture. Effective ideas include various prespecified sparsity patterns, low-rank basis expansions and combinations thereof. In this paper, we revisit classical Multiresolution Analysis (MRA) concepts such as Wavelets, whose potential value in this setting remains underexplored thus far. We show that simple approximations based on empirical feedback and design choices informed by modern hardware and implementation challenges, eventually yield a MRA-based approach for self-attention with an excellent performance profile across most criteria of interest. We undertake an extensive set of experiments and demonstrate that this multi-resolution scheme outperforms most efficient self-attention proposals and is favorable for both short and long sequences. Code is available at https://github.com/mlpen/mra-attention.*
|
|
|
|
This model was contributed by [novice03](https://huggingface.co/novice03).
|
|
The original code can be found [here](https://github.com/mlpen/mra-attention).
|
|
|
|
## MraConfig
|
|
|
|
[[autodoc]] MraConfig
|
|
|
|
## MraModel
|
|
|
|
[[autodoc]] MraModel
|
|
- forward
|
|
|
|
## MraForMaskedLM
|
|
|
|
[[autodoc]] MraForMaskedLM
|
|
- forward
|
|
|
|
## MraForSequenceClassification
|
|
|
|
[[autodoc]] MraForSequenceClassification
|
|
- forward
|
|
|
|
## MraForMultipleChoice
|
|
|
|
[[autodoc]] MraForMultipleChoice
|
|
- forward
|
|
|
|
## MraForTokenClassification
|
|
|
|
[[autodoc]] MraForTokenClassification
|
|
- forward
|
|
|
|
## MraForQuestionAnswering
|
|
|
|
[[autodoc]] MraForQuestionAnswering
|
|
- forward |