* Add new tokenizer unit test (#199)
* Perform `NFKC` normalization for sentencepiece models w/ precompiled charmap
* Fix JSDoc indentation
* Add problematic string to unit tests
* Use consistent BPE split token
* Add second problematic string
* Support outputting attentions in generate function
* Add unit tests for concatenating tensors
* Implement `cat` for `dim>0`
* Add `cat` unit tests for > 2 tensors
* Allow for negative indexing + bounds checking
* Add test case for `cat` with negative indexing
* Clean up `safeIndex` helper function
* Allow indexing error message to include dimension
* Reuse `safeIndex` helper function for `normalize_`
* Optimize `cat` indexing
* Implement `stack` tensor operation
+ add unit tests
* Add TODOs
* Implement `mean` tensor operation
* Implement `std_mean` tensor ops
* Fix order of `std_mean` returns
* Implement median filter
* Implement dynamic time warping
* Implement `neg` tensor op
* Throw error if audio sent to processor is not a `Float32Array`
* Add `round` helper function
* [WIP] Implement basic version of word-level-timestamps
Known issues:
- timestamps not correct for index > 0
- punctuation not same as python version
* Fix typo
* Fix timestamps
* Round to 2 decimals
* Fix punctuation
* Fix typing
* Remove debug statements
* Cleanup code
* Cleanup
* Remove debug statements
* Update JSDoc for extract token timestamps function
* Add return type for `std_mean` tensor function
* Improve typing of private whisper tokenizer functions
* Indicate method is private
* Allow whisper feature extractor to be called with Float64Array input
* Fix typo
* Throw error if `cross_attentions` are not present in model output when extracting token timestamps
* Throw error during generate function
* Allow whisper models to be exported with `output_attentions=True`
* Add alignment heads to generation config
* Remove print statement
* Update versions
* Override protobufjs version
* Update package-lock.json
* Require onnx==1.13.1 for conversion
Will update once onnxruntime-web supports onnx IR version 9
* Add unit test for word-level timestamps
* Extract add attentions function out of `generate`
* Fix `findLongestCommonSequence` return types
* Downgrade back to onnxruntime 1.14.0
1.15.1 is a little to unstable right now.
* Cleanup
- use `.map`
- rename variables
* Update comments
* Add examples for how to transcribe w/ word-level timestamps
* Add example for transcribing/translating audio longer than 30 seconds
* Make example more compact
* Add example code for running text-generation models
* Fix non-greedy sampling functions
* Update samplers
* Remove duplicate requirement
`onnxruntime` is specified in `optimum[onnxruntime]`
* Align `generate` function output with python library
Include starting tokens in output
* [docs] Add example text-generation code
* Update demo site text streaming for causal language models
* Override default code highlighting for operators
* Fix order of link
* link to the conversion Space for maximum simplicity
* add some types to script (very optional)
* typo
* no need for trailing slash here
* Node is also a valid option
* Document how to find a compatible checkpoint on the hub
* Update README
* Fix typing
* Update docs index
---------
Co-authored-by: Julien Chaumond <julien@huggingface.co>
* Recursively replace tensors with custom class
* Add mobile vit models
* Add example code for `ImageClassificationPipeline`
* Fix example urls
* Add MobileViT models and processors
* Update optimum requirement in conversion script
Previous name is deprecated
* Update supported models
* Update supported_models.py
* Update supported_models.py
* Update tokenizer test generator script
* Add special test case for falcon tokenizers
* Update tokenizer test script
* Add support for `FalconTokenizer`
* Update `BertPreTokenizer` call parameter types
* Add `GPTNeoXTokenizer` tokenizer (mpt)
* Use transformers from source when testing
* Reuse `prepare_model_inputs` function type
Better than using `@see {@link ... }` since it works with intellisense.
* Allow user to set `per_channel` and `reduce_range` quantization parameters (#156)
Also save quantization options
* Get operators of graph and subgraphs
* Only run encoder with required inputs
* Add basic whisper unit tests
* Add newline after heading for docs
* Add unit test for transcribing english with timestamps
* Add multilingual test case
* Update typo in node tutorial
* Create node audio processing tutorial
* Point to tutorial in `read_audio` function
* Rename `.md` to `.mdx`
* Add node audio processing tutorial to table of contents
* Add link to model in tutorial
* Update error message grammar
* Override `LOAD_FUNCTION` for decoder-only models
* Use object destructuring in `_call` functions
* Allow decoder-only models to be called
* Fix detection of default call function
* Update default `_call` JSDoc
* Mark helper functions as private
* Remove outdated comments
* Fix JSDoc
* Rename functions
* Specify model types
Reduces major code duplication
* Improve model output classes
* Remove `encoder_input_name` from seq2seq forward method
* Extract `validateInputs` helper function from `sessionRun`
* Move `compare` helper function to separate utility file
* Default `model_type` to null
* Reduce duplication when loading models using `.from_pretrained`
* Add unit tests for loading models using `.from_pretrained()`
* Compute attention mask for decoder if not given
* Improve decoder attention computation
* Implement `flatten` and `view` tensor ops
* Add documentation for new tensor ops
* Fix `flatten` input types
* Align `.generate()` return type with python library
* Add multilingual transcription + translation for whisper models (#87, #95)
* Include `return_timestamps` in calculation of `forced_decoder_ids`
* Only return non-null `forced_decoder_ids`
* Allow user to specify task in any case
* Only set `forced_decoder_ids` when non-empty
* Implement `SuppressTokensAtBeginLogitsProcessor`
Joshua Lochner