transformers.js/tests/requirements.txt

transformers[torch]@git+https://github.com/huggingface/transformers
sacremoses==0.0.53
sentencepiece==0.1.99
protobuf==4.24.3
rjieba==0.1.11
New models and refactoring (#276) * Add `CodeLlamaTokenizer` * Add `codellama` for testing * Update default quantization settings * Refactor `PretrainedModel` * Remove unnecessary error message * Update llama-code-tokenizer test * Add support for `GPTNeoX` models * Fix `GPTNeoXPreTrainedModel` config * Add support for `GPTJ` models * Add support for `WavLM` models * Update list of supported models - CodeLlama - GPT NeoX - GPT-J - WavLM * Add support for XLM models * Add support for `ResNet` models * Add support for `BeiT` models * Fix casing of `BeitModel` * Remove duplicate code * Update variable name * Remove `ts-ignore` * Remove unnecessary duplication * Update demo model sizes * [demo] Update default summarization parameters * Update default quantization parameters for new models * Remove duplication in mapping * Update list of supported marian models * Add support for `CamemBERT` models * Add support for `MBart` models * Add support for `OPT` models * Add `MBartTokenizer` and `MBart50Tokenizer` * Add example of multilingual translation with MBart models * Add `CamembertTokenizer` * Add support for `HerBERT` models * Add support for `XLMTokenizer` * Fix `fuse_unk` config * Do not remove duplicate keys for `Unigram` models See https://huggingface.co/camembert-base for an example of a Unigram tokenizer that has two tokens with the same value (`<unk>`) * Update HerBERT supported model text * Update generate_tests.py * Update list of supported models * Use enum object instead of classes for model types Fixes https://github.com/xenova/transformers.js/issues/283 * Add link to issue * Update dependencies for unit tests * Add `sentencepiece` as a testing requirement * Add `protobuf` to test dependency * Remove duplicated models to test 2023-09-08 21:17:05 +08:00			`transformers[torch]@git+https://github.com/huggingface/transformers`
			`sacremoses==0.0.53`
			`sentencepiece==0.1.99`
			`protobuf==4.24.3`
Add support for RoFormer models (#464) * Add `RoFormerTokenizer * Use `clean_text` in bert normalizer config * Add control characters test * Add support for RoFormer models * Use default label if id2label is not specified * Update requirements.txt * Skip roformer tokenizer tests 2023-12-26 21:49:11 +08:00			`rjieba==0.1.11`