* Add custom VITS tokenizer converter
* Do not decode if expected input_ids is empty
* Update vits tokenizer tests
* Implement `VitsTokenizer`
* Add support for VITS model
* Support VITS through pipeline API
* Update JSDoc
* Add TTS unit test
* Add speecht5 unit test
* Fix typo
* Fix typo
* Update speecht5 model id
* Add note about using quantized speecht5 in unit tests
* Monkey-patch `BigInt64Array` and `BigUint64Array`
* Add `RoFormerTokenizer
* Use `clean_text` in bert normalizer config
* Add control characters test
* Add support for RoFormer models
* Use default label if id2label is not specified
* Update requirements.txt
* Skip roformer tokenizer tests
* Add basic support for chat templates
* Cleanup
* JSDoc improvements
* Support conversion of user-defined functions
* Cleanup
* Fix function creation
* Add unit tests for templates
* Cleanup
* Improve JSDoc
* Add missing return types
* Add chat templates docs to table of contents
* Add support for logical negation
* Fix nested logical negation
* Add unit tests for logical operators
* Add loop variables
* Add support for `RuntimeValue` built-in functions
* Add unit tests for string instance methods
* Fix conversion of normal function to `FunctionValue`
* Update object method unit tests
* Save chat template to tokenizer_config.json during conversion
* Fix `raise_exception` error
* Add `!=` operator for booleans
* Remember to increment loop index
* Cleanup for loop evaluator
* Use `is` helper function
* Add support for text nodes
i.e., non Jinja statements/expressions
* Add auto-generated templating tests
* Update unit tests
* Remove unused function
* Add default chat templates
* Use repo with up-to-date tokenizer config
* Temporarily disable zephyr test
* Delete templates.test.js
* Move Jinja functionality to `@huggingface/jinja`
* Fix template cache type
* Update chat template unit tests
* Update `@huggingface/jinja` version
* Fix default llama2 system prompt usage
* Add unit test for llama2 w/o chat template set
* Update jinja version
* Update jinja version
* Add unit test for user-defined chat templates
Example from https://discuss.huggingface.co/t/issue-with-llama-2-chat-template-and-out-of-date-documentation/61645/3
* Add `AddedToken` for improved tokenization
* Add example usage for chat templates
* Add 'first' Metaspace pretokenizer prepend scheme
* Formatting
* Update wav2vec2 converter special tokens whitespace split
* Fix Metaspace pretokenizer split criteria
* Update inputs of `PreTokenizerSequence`
* Improve Metaspace pretokenizer
* Update llama tokenizer tests
* Improve handling of legacy llama tokenizer
* Re-enable SPM tests
* Add static tokenizer test cases
* Add llama2 static tests
* Allow user to override legacy tokenizer behaviour in `.from_pretrained`
* Add legacy tokenizer unit tests
* Bump jinja version to 0.1.0
* Update `VitMatteImageProcessor` test comment
* Add support for ChineseCLIP models
* Add chinese-clip to list of supported models
* Sort zero-shot-image-classification results by score (desc)
* Update expected zero-shot image classification output
* Add support for `VitMatte` models
* Add `VitMatteImageProcessor`
* Add `VitMatteImageProcessor` unit test
* Fix typo
* Add example code for `VitMatteForImageMatting`
* Fix JSDoc
* Fix typo
* Add support for ESM models
* Add ESM tokenizer conversion methods
* Add special test cases for ESM tokenizer
* add special tokens in conversion script
* Do not save decoder
* Add special tokens tokenizer test
* Join tokens with space if decoder is null
* Treat all tokens as added tokens
* Use `WhitespaceSplit` pretokenizer
* `<eos>` and `<bos>` are not special tokens
* Update more supported ESM models
* Add `--tokenizer_id` to conversion script
* Add supported models comments
* Add link to optimum docs for supported architectures
Closes#288
* Refactor `SUPPORTED_MODELS` dict to include task
* Update example model id
* Update list of supported models
* Update generate_tests.py
* Remove requirement of `output_attentions` revision
* Add demo site to examples section (closes#233)
* Fix typo
* Include examples in docs index
* Update github issue templates
* Create config.yml
* Order supported models
* Cleanup
* Update 4_feature-request.yml
* Add FFT unit tests
* Refactor maths.js and audio.js
* Refactor audio processors
* Add support for AST models
* Add another audio-classification example
* Add audio processing unit tests
* Implement `log_mel='dB'` in `spectrogram` function
* Add `ClapFeatureExtractor`
* Implement `ClapFeatureExtractor` unit tests
* Add support for `CLAP`
* Add `ZeroShotAudioClassificationPipeline`
* Add listed support for `zero-shot-audio-classification` pipeline tag
* Cleanup
* `let` -> `const`
* Update `mel_filter_bank` unit test
* Add `'Xenova/tiny-random-ClapModel'`
* Add `ClapAudioModelWithProjection` and `ClapTextModelWithProjection`
* Move audio validation to helper function
* Optimize `mel_filter_bank` computation
-30ms
* Update mel filters unit test
* Cleanup
* Optimizations
* Fix jsdoc
* Optimizations
* Add WIP conversion scripts
Will be updated once https://github.com/huggingface/optimum/pull/1552 is merged
* Add `size` getter to `RawImage`
* Add `DPTFeatureExtractor`
* Add depth-estimation w/ DPT models
* Add GLPN models for depth estimation
* Add missing import in example
* Add `DPTFeatureExtractor` processor test
* Add unit test for GLPN processor
* Add support for `GLPNFeatureExtractor`
Uses `size_divisor` to determine resize width and height
* Add `GLPNForDepthEstimation` example code
* Add DPT to list of supported models
* Add GLPN to list of supported models
* Add `DepthEstimationPipeline`
* Add listed support for depth estimation pipeline
* Add depth estimation pipeline unit tests
* Fix formatting
* Update `pipeline` JSDoc
* Fix typo from merge
* Add `NougatTokenizer`
* Add nougat unit tests
* Add support for `NougatImageProcessor`
* Add `crop` function to `RawImage`
* Fix `RawImage` save function
OffscreenCanvas does not have `toDataURL` function
* Add listed support for nougat models
* Fix `min`/`max` function typing
* Add unknown token to tokenizer class
* Implement `NoBadWordsLogitsProcessor`
* Use `NoBadWordsLogitsProcessor` in `generate`
* Fix regex group substitutions
Python uses \1, \2, etc. for group substitutions, but JavaScript uses $1, $2, etc.
* Create `regexSplit` helper function to split but keep delimiter
* Fix splitting for String pattern types
* Fix docstring
* Set `batch_size=1` for owlvit exports
* Add support for owlvit models
* Update default quantization settings
* Add list of supported models
* Revert update of owlvit quantization settings
* Add `OwlViTProcessor`
* Move `get_bounding_box` to utils
* Add `ZeroShotObjectDetectionPipeline`
* Add unit tests
* Add owlvit processor test
* Add listed support for `zero-shot-object-detection`
* Add OWL-ViT to list of supported models
* Update README.md
* Fix typo from merge
* Add `Swin2SRImageProcessor`
* Add `RawImage.fromTensor` helper function
* Add clamp tensor function
* Add support for `.to` data type conversion
* Add `round` tensor function
* Add support for `mul` tensor function
* Fix image padding
* Only perform padding if it will affect size
* Create basic processors unit test suite
* Add SamProcessor test case
* Move `CONTENT_TYPE_MAP` outside `RawImage` class
* Perform reflective padding for swin2sr models
* Add swin2sr models for image super-resolution
* Add listed support for Swin2SR models
* Add image-to-image pipeline
* Add listed support for image-to-image task
* Add image-to-image unit tests
* Add `add` tensor functions
* Generalize `pad_image` helper function
* Add more unit tests for image processors
* Fix typo
* Add vocoder to export
* Add tokenizer.json export for speecht5 models
* Update speecht5 supported models
* Create `SpeechT5Tokenizer`
* Add `ones` and `ones_like` tensor functions
* Add support for speecht5 text-to-speech
* Disambiguate `SpeechSeq2Seq` and `Seq2SeqLM`
* Create `TextToAudioPipeline`
* Add listed support for `text-to-audio` / `text-to-speech`
* Use unquantized vocoder by default
* Skip speecht5 unit tests for now
Due to bug in transformers: https://github.com/huggingface/transformers/issues/26547
* Update example pipeline output
* Create simple in-browser TTS demo
* Add template README
* Delete package-lock.json
* Update required transformers.js version
* Add link to Transformers.js
* Double -> Single quotes
* Add link to text-to-speech demo
* Update sample speaker embeddings
* Add `add_special_tokens` option to tokenizers
* Improve error messages for loading processors
* Add `DonutFeatureExtractor`
* Add `DonutSwinModel` and `MBartForCausalLM` models
* Fix `addPastKeyValues` for `VisionEncoderDecoder` models
* Add `Donut` to list of supported models
* Make encode parameters optional
* Support batched decoder input ids
* Remove unused import
* Add `do_thumbnail` for donut image processing
* Fix `TypeError: decoder_input_ids[i].map is not a function`
* Only pad if width and height specified in size
* Only pad if `pad_size` is defined
* Only cut `decoder_input_ids` if past model output
* Add donut model
* Add example usage to JSDoc for `DonutSwinModel`
* Add support for `DocumentQuestionAnsweringPipeline`
* Add simple document question answering unit test
* Add listed support for document QA pipeline
* Add support for `Blenderbot` models
Closes#37
References #29
* Add support for `BlenderbotTokenizer`
* Add blenderbot to supported models
* Add support for `BlenderbotSmallTokenizer`
* Add custom tests for blenderbot-small
* Add support for `BlenderbotSmall` models
* Update list of supported models
* Improve `addPastKeyValues` function
* Allow skipping of adding encoder past key values
* Add support for `MinNewTokensLengthLogitsProcessor`
* Add support for `MinLengthLogitsProcessor`
* Fix `generation_config` defaults
* Fix `input_ids_seq_length`
* Add unit tests for generation
* Fix generation parameters test case
* Allow specification of multiple `eos_token_ids`
* Add `CodeLlamaTokenizer`
* Add `codellama` for testing
* Update default quantization settings
* Refactor `PretrainedModel`
* Remove unnecessary error message
* Update llama-code-tokenizer test
* Add support for `GPTNeoX` models
* Fix `GPTNeoXPreTrainedModel` config
* Add support for `GPTJ` models
* Add support for `WavLM` models
* Update list of supported models
- CodeLlama
- GPT NeoX
- GPT-J
- WavLM
* Add support for XLM models
* Add support for `ResNet` models
* Add support for `BeiT` models
* Fix casing of `BeitModel`
* Remove duplicate code
* Update variable name
* Remove `ts-ignore`
* Remove unnecessary duplication
* Update demo model sizes
* [demo] Update default summarization parameters
* Update default quantization parameters for new models
* Remove duplication in mapping
* Update list of supported marian models
* Add support for `CamemBERT` models
* Add support for `MBart` models
* Add support for `OPT` models
* Add `MBartTokenizer` and `MBart50Tokenizer`
* Add example of multilingual translation with MBart models
* Add `CamembertTokenizer`
* Add support for `HerBERT` models
* Add support for `XLMTokenizer`
* Fix `fuse_unk` config
* Do not remove duplicate keys for `Unigram` models
See https://huggingface.co/camembert-base for an example of a Unigram tokenizer that has two tokens with the same value (`<unk>`)
* Update HerBERT supported model text
* Update generate_tests.py
* Update list of supported models
* Use enum object instead of classes for model types
Fixes https://github.com/xenova/transformers.js/issues/283
* Add link to issue
* Update dependencies for unit tests
* Add `sentencepiece` as a testing requirement
* Add `protobuf` to test dependency
* Remove duplicated models to test
* Add example `wav2vec2` models
* Add support for `CTCDecoder` and `Wav2Vec2CTCTokenizer`
* Generate tokenizer.json files for wav2vec2 models
* Fix wav2vec2 custom tokenizer generation
* Implement wav2vec2 audio-speech-recognition
* Add `Wav2Vec2` as a supported architecture
* Update README.md
* Update generate_tests.py
* Ignore invalid tests
* Update supported wav2vec2 models
* Update supported_models.py
* Simplify pipeline construction
* Implement basic audio classification pipeline
* Update default topk value for audio classification pipeline
* Add example usage for the audio classification pipeline
* Move `loadAudio` to utils file
* Add audio classification unit test
* Add wav2vec2 ASR unit test
* Improve generated wav2vec2 tokenizer json
* Update supported_models.py
* Allow `added_tokens_regex` to be null
* Support exporting mms vocabs
* Supported nested vocabularies
* Update supported tasks and models
* Add warnings to ignore language and task for wav2vec2 models
Will add in future
* Mark internal methods as private
* Add typing to audio variable
* Update node-audio-processing.mdx
* Move node-audio-processing to guides
* Update table of contents
* Add example code for performing feature extraction w/ `Wav2Vec2Model`
NOTE: feature extraction of MMS models is currently broken in the python library, but it works correctly here. See
https://github.com/huggingface/transformers/issues/25485 for more info
* Refactor `Pipeline` class params
* Fix `pipeline` function
* Fix typo in `pipeline` JSDoc
* Fix second typo
* Add `M2M100Tokenizer`
* Allow `added_tokens` list to be empty
* Apply hot-fix for issue in HF's `M2M100Tokenizer`
* Skip M2M100 tokenizer tests for now
TODO: Remove when https://github.com/huggingface/transformers/pull/25478 is merged
* Fix `_build_translation_inputs` for `M2M100Tokenizer`
* Add example code in JSDoc for `TranslationPipeline`
* Update supported_models.py
* Create basic tokenizer playground app
* Default to no display when user adding large body of text
* Optimize BPE algorithm
- Use map instead of object for `bpe_ranks`
- Replace reduction in BPE algorithm with for loop
- Avoid conversions between sets and arrays
* Use for loop to avoid stack issues with `.push(...items)`
* Fix `mergeArrays` typing
* Remove unnecessary try-catch block in BPE
* Add Llama, T5, and BERT tokenizers to the playground
* Improve how BERT/T5 tokens are displayed
* Improve how token margins are displayed
* Use `Map` for cache
* Add efficient heap-based priority queue implementation
* Add more unit tests for LlamaTokenizer
Selected from https://github.com/belladoreai/llama-tokenizer-js/blob/master/llama-tokenizer.js#L381-L452
* Implement priority-queue-based BPE algorithm
* Remove old code
* Update `bpe` docstring
* Add `data-structures` page to docs
* Update JSDoc for data-structures.js
* Update data-structures.js
* Move `TokenLattice` and `CharTrie` to data-structures module
* Minor refactoring
* Define custom CLIP ONNX configs
* Update conversion script
* Support specifying custom model file name
* Use int64 for CLIP input ids
* Add support for CLIP text and vision models
* Fix JSDoc
* Add docs for `CLIPTextModelWithProjection`
* Add docs for `CLIPVisionModelWithProjection`
* Add unit test for CLIP text models
* Add unit test for CLIP vision models
* Set resize precision to 3 decimal places
* Fix `RawImage.save()` function
* Throw error when reading image and status != 200
* Create basic semantic image search application
* Separate out components
* Add `update-database` script
* Update transformers.js version
* Add new tokenizer unit test (#199)
* Perform `NFKC` normalization for sentencepiece models w/ precompiled charmap
* Fix JSDoc indentation
* Add problematic string to unit tests
* Use consistent BPE split token
* Add second problematic string
* Support outputting attentions in generate function
* Add unit tests for concatenating tensors
* Implement `cat` for `dim>0`
* Add `cat` unit tests for > 2 tensors
* Allow for negative indexing + bounds checking
* Add test case for `cat` with negative indexing
* Clean up `safeIndex` helper function
* Allow indexing error message to include dimension
* Reuse `safeIndex` helper function for `normalize_`
* Optimize `cat` indexing
* Implement `stack` tensor operation
+ add unit tests
* Add TODOs
* Implement `mean` tensor operation
* Implement `std_mean` tensor ops
* Fix order of `std_mean` returns
* Implement median filter
* Implement dynamic time warping
* Implement `neg` tensor op
* Throw error if audio sent to processor is not a `Float32Array`
* Add `round` helper function
* [WIP] Implement basic version of word-level-timestamps
Known issues:
- timestamps not correct for index > 0
- punctuation not same as python version
* Fix typo
* Fix timestamps
* Round to 2 decimals
* Fix punctuation
* Fix typing
* Remove debug statements
* Cleanup code
* Cleanup
* Remove debug statements
* Update JSDoc for extract token timestamps function
* Add return type for `std_mean` tensor function
* Improve typing of private whisper tokenizer functions
* Indicate method is private
* Allow whisper feature extractor to be called with Float64Array input
* Fix typo
* Throw error if `cross_attentions` are not present in model output when extracting token timestamps
* Throw error during generate function
* Allow whisper models to be exported with `output_attentions=True`
* Add alignment heads to generation config
* Remove print statement
* Update versions
* Override protobufjs version
* Update package-lock.json
* Require onnx==1.13.1 for conversion
Will update once onnxruntime-web supports onnx IR version 9
* Add unit test for word-level timestamps
* Extract add attentions function out of `generate`
* Fix `findLongestCommonSequence` return types
* Downgrade back to onnxruntime 1.14.0
1.15.1 is a little to unstable right now.
* Cleanup
- use `.map`
- rename variables
* Update comments
* Add examples for how to transcribe w/ word-level timestamps
* Add example for transcribing/translating audio longer than 30 seconds
* Make example more compact
* Recursively replace tensors with custom class
* Add mobile vit models
* Add example code for `ImageClassificationPipeline`
* Fix example urls
* Add MobileViT models and processors
* Update optimum requirement in conversion script
Previous name is deprecated
* Update supported models
* Update supported_models.py
* Update supported_models.py
* Update tokenizer test generator script
* Add special test case for falcon tokenizers
* Update tokenizer test script
* Add support for `FalconTokenizer`
* Update `BertPreTokenizer` call parameter types
* Add `GPTNeoXTokenizer` tokenizer (mpt)
* Use transformers from source when testing
* Reuse `prepare_model_inputs` function type
Better than using `@see {@link ... }` since it works with intellisense.
* Only run encoder with required inputs
* Add basic whisper unit tests
* Add newline after heading for docs
* Add unit test for transcribing english with timestamps
* Add multilingual test case
* Override `LOAD_FUNCTION` for decoder-only models
* Use object destructuring in `_call` functions
* Allow decoder-only models to be called
* Fix detection of default call function
* Update default `_call` JSDoc
* Mark helper functions as private
* Remove outdated comments
* Fix JSDoc
* Rename functions
* Specify model types
Reduces major code duplication
* Improve model output classes
* Remove `encoder_input_name` from seq2seq forward method
* Extract `validateInputs` helper function from `sessionRun`
* Move `compare` helper function to separate utility file
* Default `model_type` to null
* Reduce duplication when loading models using `.from_pretrained`
* Add unit tests for loading models using `.from_pretrained()`
* Compute attention mask for decoder if not given
* Improve decoder attention computation
* Implement `flatten` and `view` tensor ops
* Add documentation for new tensor ops
* Fix `flatten` input types
Joshua Lochner