Joshua Lochner 1165f04a9f Fix BPE tokenization for weird whitespace characters (Closes #199) (#208) ... * Add new tokenizer unit test (#199) * Perform `NFKC` normalization for sentencepiece models w/ precompiled charmap * Fix JSDoc indentation * Add problematic string to unit tests * Use consistent BPE split token * Add second problematic string		2023-07-22 04:51:11 +02:00
..
data	Generate tests before running unit tests	2023-05-02 20:01:19 +02:00
configs.test.js	Remove unnecessary imports	2023-05-31 02:06:34 +02:00
generate_tests.py	Fix BPE tokenization for weird whitespace characters (Closes #199) (#208)	2023-07-22 04:51:11 +02:00
hub.test.js	Fix failing tests	2023-05-05 00:05:03 +02:00
init.js	Implement `fetch` workaround	2023-06-01 01:09:14 +02:00
maths.test.js	Whisper word-level timestamps (#184)	2023-07-09 23:21:43 +02:00
models.test.js	Support calling of decoder-only models (Fixes #137) (#149)	2023-06-20 15:24:35 +02:00
pipelines.test.js	Object-detection pipeline improvements + better documentation (#189)	2023-07-11 02:09:03 +02:00
tensor.test.js	Whisper word-level timestamps (#184)	2023-07-09 23:21:43 +02:00
test_utils.js	Support calling of decoder-only models (Fixes #137) (#149)	2023-06-20 15:24:35 +02:00
tokenizers.test.js	[WIP] New model/tokenizer types (#165)	2023-06-28 15:14:44 +02:00
utils.test.js	Create test suite for utility functions	2023-05-16 22:35:57 +02:00