* Add `CodeLlamaTokenizer`
* Add `codellama` for testing
* Update default quantization settings
* Refactor `PretrainedModel`
* Remove unnecessary error message
* Update llama-code-tokenizer test
* Add support for `GPTNeoX` models
* Fix `GPTNeoXPreTrainedModel` config
* Add support for `GPTJ` models
* Add support for `WavLM` models
* Update list of supported models
- CodeLlama
- GPT NeoX
- GPT-J
- WavLM
* Add support for XLM models
* Add support for `ResNet` models
* Add support for `BeiT` models
* Fix casing of `BeitModel`
* Remove duplicate code
* Update variable name
* Remove `ts-ignore`
* Remove unnecessary duplication
* Update demo model sizes
* [demo] Update default summarization parameters
* Update default quantization parameters for new models
* Remove duplication in mapping
* Update list of supported marian models
* Add support for `CamemBERT` models
* Add support for `MBart` models
* Add support for `OPT` models
* Add `MBartTokenizer` and `MBart50Tokenizer`
* Add example of multilingual translation with MBart models
* Add `CamembertTokenizer`
* Add support for `HerBERT` models
* Add support for `XLMTokenizer`
* Fix `fuse_unk` config
* Do not remove duplicate keys for `Unigram` models
See https://huggingface.co/camembert-base for an example of a Unigram tokenizer that has two tokens with the same value (`<unk>`)
* Update HerBERT supported model text
* Update generate_tests.py
* Update list of supported models
* Use enum object instead of classes for model types
Fixes https://github.com/xenova/transformers.js/issues/283
* Add link to issue
* Update dependencies for unit tests
* Add `sentencepiece` as a testing requirement
* Add `protobuf` to test dependency
* Remove duplicated models to test
This is necessary while we wait for more models to support ONNX weights. In future, we hope to remove the need for separation.
When testing remotely (e.g., GitHub actions), we will load models from the Hugging Face Hub under the username `Xenova`. On the other hand, when testing locally, we will use the model that is exported using the conversion script.
Joshua Lochner