dnrops
/
transformers.js
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,074 Commits 57 Branches 49 Tags 103 MiB
Commit Graph

1074 Commits

Author SHA1 Message Date
Joshua Lochner da2688626d [version] Update to 2.17.2 2024-05-29 16:22:05 +02:00
Joshua Lochner f542c52c94
Update Next.js demos to 14.2.3 (#772) 2024-05-24 00:44:09 +02:00
Nikhil Verma 64b3da608b
Update pipelines.js to allow for `token_embeddings` as well (#770) 
* Update pipelines.js to allow for `token_embeddings` as well

In recent examples of optimum pipeline export the feature extraction pipelines have their output state as `token_embeddings` instead of `last_hidden_state`.

So we should support this as well.

* Keep diff small

* Keep diff small

---------

Co-authored-by: Joshua Lochner <admin@xenova.com>
 2024-05-23 22:45:32 +02:00
Nawar Alsafar 250fcd7ada
Add model file name (#594) 
* add model_file_name

* Allow developers to configure  model_file_name (only works for decoder-only models for now)

* remove prettier formatting

* merge with 2.15.1

* add model_file_name

* Apply suggestions from code review

---------

Co-authored-by: Joshua Lochner <admin@xenova.com>
 2024-05-23 12:13:52 +02:00
Joshua Lochner db9250b3e8
Add sequence post processor (#771) 
* Add `Sequence` PostProcessor

Required by https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/

* Support `return_token_type_ids`

* Add llama3 tokenizer to unit tests

* Add test for allowing user to request for token type ids

* Add JSDoc

* Update generate_tests.py
 2024-05-23 11:42:24 +02:00
Joshua Lochner e50f5686a2
Add support for FastViT (#749) 
* Add support for FastViT

* Comment out list of timm models
 2024-05-23 01:47:16 +02:00
Kendell R da0e9204c4
Auto rotate image (#737) 2024-05-23 01:43:56 +02:00
Joshua Lochner 8d166ca642
Optimize FFT (#766) 
* Optimize FFT for real transforms

* Throw error if power is not specified

https://github.com/huggingface/transformers/issues/27772
 2024-05-23 01:43:05 +02:00
Joshua Lochner 8963720585
Fix CI - update tests.yml (#768) 2024-05-23 01:40:50 +02:00
Joshua Lochner 0af1e2f3c2
Remove old import from `stream/web` for `ReadableStream` (#752) 
Node 16 is no longer supported
 2024-05-20 18:05:48 +02:00
Joshua Lochner bd315529ab
Update tokenizer playground (#717) 
* [tokenizer playground] Add Mistral v3 tokenizer

* Add support for custom tokenizers

* Add llama 3 tokenizer
 2024-05-10 23:56:28 +02:00
Hans 880cd3ea65
Support reading data from blob URI (#645) 
* Make blob as valid URL

* Create function to detect the blob URI

* Change to `isValidUrl`

* Remove comment

Co-authored-by: Joshua Lochner <admin@xenova.com>

* Merge `isValidHttpUrl` into `isValidUrl`

* Correct implement

* Update docs

* Add test

* Remove export for `isValidUrl`

* Test read blob via `getFile`

* Use `res.text()` instead `res.body`

---------

Co-authored-by: Joshua Lochner <admin@xenova.com>
 2024-05-08 13:14:50 +02:00
Ikko Eltociear Ashimine 8bb8c5a33c
docs: update vanilla-js.md (#738) 
minor fix
 2024-05-05 23:59:50 +02:00
Joshua Lochner 7122a0a5a1
Add mobilevitv2 (#721) 
* Add support for MobileViTV2

* Update supported_models.py

* Add support for `do_flip_channel_order`

* Add unit test for `do_flip_channel_order=true`
 2024-05-02 11:42:44 +02:00
Joshua Lochner 1a9964fb09
Use ungated version of mistral tokenizer (#718) 2024-04-20 16:01:17 +02:00
Joshua Lochner 992f643e2a [version] Update to 2.17.1 2024-04-18 17:25:49 +02:00
Joshua Lochner 6d5901e652
Add `ignore_merges` option to BPE tokenizers (#716) 2024-04-18 13:59:52 +02:00
Joshua Lochner 642743136e [version] Update to 2.17.0 2024-04-11 01:20:24 +02:00
Joshua Lochner c2c45cb577
Improve support of conversational models (#658) 
* Add `return_full_text` option for text-generation models

* [wip] Support chat inputs in text-generation pipeline

* Align return type with python version

* Remove conversational task (moved to text-generation)

* Fix typos
 2024-04-11 00:58:50 +02:00
Joshua Lochner aa542cf548
Update dependencies (#704) 
* Update dependencies

* Bump protobufjs from 7.2.4 to 7.2.6 in /examples/semantic-image-search (#705)

Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.2.4 to 7.2.6.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md)
- [Commits](https://github.com/protobufjs/protobuf.js/compare/protobufjs-v7.2.4...protobufjs-v7.2.6)

---
updated-dependencies:
- dependency-name: protobufjs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump protobufjs from 7.2.4 to 7.2.6 in /examples/next-client (#706)

Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.2.4 to 7.2.6.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md)
- [Commits](https://github.com/protobufjs/protobuf.js/compare/protobufjs-v7.2.4...protobufjs-v7.2.6)

---
updated-dependencies:
- dependency-name: protobufjs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump protobufjs from 7.2.4 to 7.2.6 in /examples/next-server (#707)

Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.2.4 to 7.2.6.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md)
- [Commits](https://github.com/protobufjs/protobuf.js/compare/protobufjs-v7.2.4...protobufjs-v7.2.6)

---
updated-dependencies:
- dependency-name: protobufjs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump protobufjs in /examples/semantic-image-search-client (#708)

Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.2.4 to 7.2.6.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md)
- [Commits](https://github.com/protobufjs/protobuf.js/compare/protobufjs-v7.2.4...protobufjs-v7.2.6)

---
updated-dependencies:
- dependency-name: protobufjs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-04-11 00:24:51 +02:00
Varun Patil 15b90b7523
maths: switch dot to faster for loop (#667) 
* maths: switch dot to faster for loop

* Update src/utils/maths.js

---------

Co-authored-by: Joshua Lochner <admin@xenova.com>
 2024-04-10 17:01:03 +02:00
Joshua Lochner cdbc532cf6
Update dependencies (#702) 
* Bump vite from 4.5.2 to 4.5.3 in /examples/tokenizer-playground (#693)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 4.5.2 to 4.5.3.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v4.5.3/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v4.5.3/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump express from 4.18.2 to 4.19.2 (#666)

Bumps [express](https://github.com/expressjs/express) from 4.18.2 to 4.19.2.
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/master/History.md)
- [Commits](https://github.com/expressjs/express/compare/4.18.2...4.19.2)

---
updated-dependencies:
- dependency-name: express
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 5.0.12 to 5.0.13 in /examples/remove-background-client (#694)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 5.0.12 to 5.0.13.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v5.0.13/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v5.0.13/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 5.0.12 to 5.0.13 in /examples/zero-shot-classification (#695)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 5.0.12 to 5.0.13.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v5.0.13/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v5.0.13/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 5.0.12 to 5.0.13 in /examples/depth-anything-client (#696)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 5.0.12 to 5.0.13.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v5.0.13/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v5.0.13/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 5.0.12 to 5.0.13 in /examples/semantic-audio-search (#697)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 5.0.12 to 5.0.13.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v5.0.13/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v5.0.13/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 4.5.2 to 4.5.3 in /examples/code-completion (#698)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 4.5.2 to 4.5.3.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v4.5.3/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v4.5.3/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 5.1.4 to 5.1.7 in /examples/video-object-detection (#699)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 5.1.4 to 5.1.7.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v5.1.7/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v5.1.7/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 5.1.1 to 5.1.7 in /examples/cross-encoder (#700)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 5.1.1 to 5.1.7.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v5.1.7/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v5.1.7/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump vite from 5.1.1 to 5.1.7 in /examples/adaptive-retrieval (#701)

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 5.1.1 to 5.1.7.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v5.1.7/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v5.1.7/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-04-10 16:07:58 +02:00
Jonathan Padilla f71ee32dd9
Add binary embedding quantization support to FeatureExtraction pipeline (#691) 
* Add binary embedding quantization support to FeatureExtraction pipeline

* Add JSDoc return type

* Apply suggestions from code review

* Update src/utils/tensor.js

* Update example code snippet

* Update src/pipelines.js

---------

Co-authored-by: Joshua Lochner <admin@xenova.com>
 2024-04-10 15:13:36 +02:00
dependabot[bot] 8804c36591
Bump ip from 2.0.0 to 2.0.1 in /examples/electron (#665) 
Bumps [ip](https://github.com/indutny/node-ip) from 2.0.0 to 2.0.1.
- [Commits](https://github.com/indutny/node-ip/compare/v2.0.0...v2.0.1)

---
updated-dependencies:
- dependency-name: ip
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-26 22:45:55 +02:00
dependabot[bot] 3509fc405b
Bump webpack-dev-middleware from 5.3.3 to 5.3.4 (#661) 
Bumps [webpack-dev-middleware](https://github.com/webpack/webpack-dev-middleware) from 5.3.3 to 5.3.4.
- [Release notes](https://github.com/webpack/webpack-dev-middleware/releases)
- [Changelog](https://github.com/webpack/webpack-dev-middleware/blob/v5.3.4/CHANGELOG.md)
- [Commits](https://github.com/webpack/webpack-dev-middleware/compare/v5.3.3...v5.3.4)

---
updated-dependencies:
- dependency-name: webpack-dev-middleware
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-26 15:49:41 +02:00
Joshua Lochner d50b3193fb [version] Update to 2.16.1 2024-03-20 16:44:39 +02:00
Joshua Lochner 4cab8eceb6
Add `image-feature-extractor` pipeline (#650) 
* Add `image-feature-extractor` pipeline

* Update "Image Feature Extraction" heading
 2024-03-20 15:30:27 +02:00
Joshua Lochner f0ef2e8eee
Update tokenizer `apply_chat_template` functionality (#647) 
* Allow custom kwargs in `tokenizer.apply_chat_template`

* Update jinja dependency version

* Add `tokenizer_kwargs` options

* Add support for dictionaries of chat templates in the tokenizer config

* Add `CohereTokenizer`

* `apply_chat_template` is no longer async

* Add unit test for multiple chat templates

* Update tokenizers.js

* Also update when `chat_template` is undefined

* Support setting tokenizer and text from URL

* Update Claude tokenizer display name

* Add Cohere Command-R tokenizer to playground

* Add `Grok1Tokenizer`

* Throw error if chat template object is malformed

* Improved error checking

* Remove redundant error check

* `template_dict` can be a null-prototype object
 2024-03-20 15:22:01 +02:00
Joshua Lochner 40cdd36597
Fix channel format when padding non-square images for certain models (#655) 
* Add tensor permute unit tests

* Rename transpose -> permute

* Fix padding for non-square images

* Add vitmatte padding unit test

* Create `RawImage.toTensor` helper function

* Add bankers rounding test case

* `.toBe()` -> `.toBeCloseTo()` for floating point numbers
 2024-03-20 14:35:30 +02:00
dependabot[bot] 8c731fa54b
Bump follow-redirects from 1.15.4 to 1.15.6 (#652) 
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.4 to 1.15.6.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.15.4...v1.15.6)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-19 00:17:47 +02:00
Joshua Lochner a51a61be22
Add support for EfficientNet (#639) 
* Add support for efficientnet

* Add `EfficientNetImageProcessor`
 2024-03-16 14:49:32 +02:00
Joshua Lochner 314b7f0dc4 [version] Update to 2.16.0 2024-03-07 15:45:22 +02:00
Dave 8eef154b1e
Add WavLM- & Wav2Vec2ForAudioFrameClassification support (#611) 
* Add WavLMForXVector support

* fix model docs

* Add WavLMForAudioFrameClassification

* Add missing wWav2Vec2ForAudioFrameCl.

* Add doc comment

* Add doc string wav2vec2

* update comment

* make example like python

* Update src/models.js

---------

Co-authored-by: Joshua Lochner <admin@xenova.com>
 2024-03-07 15:27:49 +02:00
Dave 5bb8d25337
Add StableLM support (#616) 
* add stablelm model impl.

* add stablelm mapping

* Update `StableLMPreTrainedModel` class config

* Update stablelm conversion script quantization settings

* Add StableLm to list of supported models

* Remove StableLMForSequenceClassification

will be added in seperate PR

Co-authored-by: Joshua Lochner <admin@xenova.com>

* Remove SequenceClassifierOutputWithPast

will be added in seperate PR

Co-authored-by: Joshua Lochner <admin@xenova.com>

* Remove StableLMForSequenceClassification mapping

will be added in seperate pr

Co-authored-by: Joshua Lochner <admin@xenova.com>

* Update src/models.js

* Naming nit

* Naming nits

* Update scripts/convert.py

* Update src/models.js

---------

Co-authored-by: Joshua Lochner <admin@xenova.com>
 2024-03-07 03:05:27 +02:00
Joshua Lochner dab1b8139f
Remove GH-specific redirect logic (#630) 2024-03-07 02:47:32 +02:00
Joshua Lochner 8694a856ee
Bump `@huggingface/jinja` (#629) 2024-03-07 01:59:06 +02:00
Joshua Lochner 382ff3af35
Create cross-encoder demo (#617) 2024-03-06 17:31:57 +02:00
Joshua Lochner e072ee6b70
Add support for starcoder2 models (#622) 2024-03-06 13:09:04 +02:00
Joshua Lochner 5f572241b4
Add support for `UniSpeech` and `UniSpeechSat` models (#624) 
* Add support for `UniSpeech` and `UniSpeechSat` models

* Remove listed support for unispeech STT models

* Align unispeech w/ existing wav2vec logic
 2024-03-06 13:08:28 +02:00
Joshua Lochner 2e53f51ce7
Update `@huggingface/jinja` -> 0.2.0 (#627) 2024-03-05 17:04:08 +02:00
Joshua Lochner 55a4a69491
Check for existence of `onnx_env.wasm` before updating wasmPaths (#621) 2024-03-05 17:03:25 +02:00
Joshua Lochner d525c8bfd4
Add Claude 3 + Mistral to tokenizer playground 2024-03-05 00:52:12 +02:00
Joshua Lochner ce4fd62202
Update tokenizer playground dependencies (#599) 
* Update transformers.js version

* Update Token.jsx
 2024-03-05 00:42:37 +02:00
Dave b5a548fc65
Add support for `WavLMForXVector` (#603) 
* Add WavLMForXVector support

* fix model docs

* fix bad naming

* Apply suggestions from code review

* Update default `wavlm` quantization settings

* Update list of supported `wavlm` models

* Update JSDoc

* Fix typo

---------

Co-authored-by: Joshua Lochner <admin@xenova.com>
 2024-02-28 21:24:41 +02:00
Joshua Lochner 271c6f1744
Create video object detection demo (#607) 2024-02-27 19:35:12 +02:00
Joshua Lochner 7772d1db0a [version] Update to 2.15.1 2024-02-21 16:26:05 +02:00
Joshua Lochner 68ed7f6cbb
Add Gemma Tokenizer (#598) 
* Fix styling for whitespace tokens

* Add `GemmaTokenizer`

* Update minimum `@huggingface/jinja` version

* Add Gemma to tokenizer playground

* Add Gemma tokenizer unit test

* Update tokenizer names in playground

* Update Gemma tokenizer test
 2024-02-21 16:22:22 +02:00
Joshua Lochner 6d2808b571
Create adaptive retrieval demo (#587) 
* Create adaptive retrieval demo

* Remove waiting label
 2024-02-21 15:16:51 +02:00
Joshua Lochner d1eabaeb0b
Update tokenizer playground (#597) 
* Update transformers.js version

* Remove unused images

* Remove link to placeholder image
 2024-02-21 14:04:11 +02:00
Joshua Lochner 5ac17bda83
Add support for OWLv2 models (#579) 2024-02-19 14:57:06 +02:00