* Add basic support for chat templates
* Cleanup
* JSDoc improvements
* Support conversion of user-defined functions
* Cleanup
* Fix function creation
* Add unit tests for templates
* Cleanup
* Improve JSDoc
* Add missing return types
* Add chat templates docs to table of contents
* Add support for logical negation
* Fix nested logical negation
* Add unit tests for logical operators
* Add loop variables
* Add support for `RuntimeValue` built-in functions
* Add unit tests for string instance methods
* Fix conversion of normal function to `FunctionValue`
* Update object method unit tests
* Save chat template to tokenizer_config.json during conversion
* Fix `raise_exception` error
* Add `!=` operator for booleans
* Remember to increment loop index
* Cleanup for loop evaluator
* Use `is` helper function
* Add support for text nodes
i.e., non Jinja statements/expressions
* Add auto-generated templating tests
* Update unit tests
* Remove unused function
* Add default chat templates
* Use repo with up-to-date tokenizer config
* Temporarily disable zephyr test
* Delete templates.test.js
* Move Jinja functionality to `@huggingface/jinja`
* Fix template cache type
* Update chat template unit tests
* Update `@huggingface/jinja` version
* Fix default llama2 system prompt usage
* Add unit test for llama2 w/o chat template set
* Update jinja version
* Update jinja version
* Add unit test for user-defined chat templates
Example from https://discuss.huggingface.co/t/issue-with-llama-2-chat-template-and-out-of-date-documentation/61645/3
* Add `AddedToken` for improved tokenization
* Add example usage for chat templates
* Add 'first' Metaspace pretokenizer prepend scheme
* Formatting
* Update wav2vec2 converter special tokens whitespace split
* Fix Metaspace pretokenizer split criteria
* Update inputs of `PreTokenizerSequence`
* Improve Metaspace pretokenizer
* Update llama tokenizer tests
* Improve handling of legacy llama tokenizer
* Re-enable SPM tests
* Add static tokenizer test cases
* Add llama2 static tests
* Allow user to override legacy tokenizer behaviour in `.from_pretrained`
* Add legacy tokenizer unit tests
* Bump jinja version to 0.1.0
* Update `VitMatteImageProcessor` test comment
* Add support for ChineseCLIP models
* Add chinese-clip to list of supported models
* Sort zero-shot-image-classification results by score (desc)
* Update expected zero-shot image classification output
* Add support for `VitMatte` models
* Add `VitMatteImageProcessor`
* Add `VitMatteImageProcessor` unit test
* Fix typo
* Add example code for `VitMatteForImageMatting`
* Fix JSDoc
* Fix typo
* Add support for ESM models
* Add ESM tokenizer conversion methods
* Add special test cases for ESM tokenizer
* add special tokens in conversion script
* Do not save decoder
* Add special tokens tokenizer test
* Join tokens with space if decoder is null
* Treat all tokens as added tokens
* Use `WhitespaceSplit` pretokenizer
* `<eos>` and `<bos>` are not special tokens
* Update more supported ESM models
* Add `--tokenizer_id` to conversion script
* Add supported models comments
* Add link to optimum docs for supported architectures
Closes#288
* Refactor `SUPPORTED_MODELS` dict to include task
* Update example model id
* Update list of supported models
* Update generate_tests.py
* Remove requirement of `output_attentions` revision
* Add demo site to examples section (closes#233)
* Fix typo
* Include examples in docs index
* Update github issue templates
* Create config.yml
* Order supported models
* Cleanup
* Update 4_feature-request.yml
* Add FFT unit tests
* Refactor maths.js and audio.js
* Refactor audio processors
* Add support for AST models
* Add another audio-classification example
* Add audio processing unit tests
* Implement `log_mel='dB'` in `spectrogram` function
* Add `ClapFeatureExtractor`
* Implement `ClapFeatureExtractor` unit tests
* Add support for `CLAP`
* Add `ZeroShotAudioClassificationPipeline`
* Add listed support for `zero-shot-audio-classification` pipeline tag
* Cleanup
* `let` -> `const`
* Update `mel_filter_bank` unit test
* Add `'Xenova/tiny-random-ClapModel'`
* Add `ClapAudioModelWithProjection` and `ClapTextModelWithProjection`
* Move audio validation to helper function
* Optimize `mel_filter_bank` computation
-30ms
* Update mel filters unit test
* Cleanup
* Optimizations
* Fix jsdoc
* Optimizations
* Add WIP conversion scripts
Will be updated once https://github.com/huggingface/optimum/pull/1552 is merged
* Add `size` getter to `RawImage`
* Add `DPTFeatureExtractor`
* Add depth-estimation w/ DPT models
* Add GLPN models for depth estimation
* Add missing import in example
* Add `DPTFeatureExtractor` processor test
* Add unit test for GLPN processor
* Add support for `GLPNFeatureExtractor`
Uses `size_divisor` to determine resize width and height
* Add `GLPNForDepthEstimation` example code
* Add DPT to list of supported models
* Add GLPN to list of supported models
* Add `DepthEstimationPipeline`
* Add listed support for depth estimation pipeline
* Add depth estimation pipeline unit tests
* Fix formatting
* Update `pipeline` JSDoc
* Fix typo from merge
* Add `NougatTokenizer`
* Add nougat unit tests
* Add support for `NougatImageProcessor`
* Add `crop` function to `RawImage`
* Fix `RawImage` save function
OffscreenCanvas does not have `toDataURL` function
* Add listed support for nougat models
* Fix `min`/`max` function typing
* Add unknown token to tokenizer class
* Implement `NoBadWordsLogitsProcessor`
* Use `NoBadWordsLogitsProcessor` in `generate`
* Fix regex group substitutions
Python uses \1, \2, etc. for group substitutions, but JavaScript uses $1, $2, etc.
* Create `regexSplit` helper function to split but keep delimiter
* Fix splitting for String pattern types
* Fix docstring
* Set `batch_size=1` for owlvit exports
* Add support for owlvit models
* Update default quantization settings
* Add list of supported models
* Revert update of owlvit quantization settings
* Add `OwlViTProcessor`
* Move `get_bounding_box` to utils
* Add `ZeroShotObjectDetectionPipeline`
* Add unit tests
* Add owlvit processor test
* Add listed support for `zero-shot-object-detection`
* Add OWL-ViT to list of supported models
* Update README.md
* Fix typo from merge
* Move tensor clone for Worker ownership NaN issue
* Update src/models.js - Use conditional operator
Co-authored-by: Joshua Lochner <admin@xenova.com>
* Update src/models.js - Object.create(null)
Co-authored-by: Joshua Lochner <admin@xenova.com>
* tensor.js: remove "Object" type to fix types (since ONNX exports correct type now)
* models.js / validateInputs(): Remove promise/await because it is not needed
Use "tensor instanceof Tensor" check because otherwise validateInputs() thinks it has an input even if it doesn't
* Fix JSDoc
* Update JSDoc
---------
Co-authored-by: Joshua Lochner <admin@xenova.com>
* Add `Swin2SRImageProcessor`
* Add `RawImage.fromTensor` helper function
* Add clamp tensor function
* Add support for `.to` data type conversion
* Add `round` tensor function
* Add support for `mul` tensor function
* Fix image padding
* Only perform padding if it will affect size
* Create basic processors unit test suite
* Add SamProcessor test case
* Move `CONTENT_TYPE_MAP` outside `RawImage` class
* Perform reflective padding for swin2sr models
* Add swin2sr models for image super-resolution
* Add listed support for Swin2SR models
* Add image-to-image pipeline
* Add listed support for image-to-image task
* Add image-to-image unit tests
* Add `add` tensor functions
* Generalize `pad_image` helper function
* Add more unit tests for image processors
* Fix typo
* By default, do not add special tokens in text-generation
See 147e8ce4ae/src/transformers/pipelines/text_generation.py (L106)
* Add support for mistral models
* Add support for Falcon models
* Replace `batch_size` with variable
* Add Falcon to list of supported models
* Fix typing issue with bigint literals
Joshua Lochner