scisharp
/
LLamaSharp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
582 Commits 8 Branches 17 Tags 317 MiB
Commit Graph

582 Commits

Author SHA1 Message Date
Martin Evans b4e57f7966 Added extra diagnostics to test runner 2023-10-25 16:22:40 +01:00
Martin Evans 666455d144
Merge pull request #219 from martindevans/fix/broken_text_decoding 
Fixed Broken Text Decoding
 2023-10-25 14:52:40 +01:00
Martin Evans 36c71abcfb Fixed `LLama.StreamingTokenDecoderLLamaLLama.StreamingTokenDecoderLLamaLLama.StreamingTokenDecoderLLama` spam in all executors except Stateless. 2023-10-25 13:57:00 +01:00
Martin Evans 5b6408b072
Merge pull request #205 from martindevans/roundtrip_tokenization_investigation 
RoundTrip Tokenization Errors
 2023-10-24 20:46:48 +01:00
Martin Evans 4a6319742d
Update README.md 2023-10-24 18:49:09 +01:00
Martin Evans 4e315423a1
Update README.md 2023-10-24 18:48:44 +01:00
Rinne f0552dc813
Merge pull request #217 from AsakusaRinne/master 
docs: update the readme.
 2023-10-25 01:47:00 +08:00
Yaohui Liu 4c80b40870 docs: update the readme. 2023-10-25 01:45:54 +08:00
Rinne 1299de3e5b
Merge pull request #216 from AsakusaRinne/auto_release_ci 
ci: try to fix error of pushing package.
 2023-10-25 01:23:51 +08:00
Yaohui Liu 4b86bc8d9d ci: try to fix error of pushing package. 2023-10-25 01:07:55 +08:00
Yaohui Liu 27b21cf136 ci: try to fix error of pushing package. 2023-10-25 01:03:04 +08:00
Yaohui Liu 0f6408f60a ci: try to fix error of pushing package. 2023-10-25 00:58:20 +08:00
Yaohui Liu c6c478f52b ci: try to fix error of pushing package. 2023-10-25 00:52:38 +08:00
Rinne f62afad1e8
Merge pull request #215 from AsakusaRinne/auto_release_ci 
ci: try to fix error of pushing package.
 2023-10-25 00:46:40 +08:00
Yaohui Liu a5e66b31dc ci: fix typo. 2023-10-25 00:45:49 +08:00
Yaohui Liu 20d8cb1e84 ci: try to fix error of pushing package. 2023-10-25 00:44:44 +08:00
Rinne 3fd1063a87
Merge pull request #214 from AsakusaRinne/auto_release_ci 
ci: try to fix error of pushing package.
 2023-10-25 00:28:35 +08:00
Yaohui Liu e04c2c51ab ci: try to fix error of pushing package. 2023-10-25 00:27:51 +08:00
Rinne 31f933ab7f
Merge pull request #213 from AsakusaRinne/auto_release_ci 
ci: try to fix error of pushing package.
 2023-10-25 00:11:39 +08:00
Yaohui Liu 99ecfd3e5b ci: try to fix error of pushing package. 2023-10-25 00:11:06 +08:00
Rinne d0e798b623
Merge pull request #212 from AsakusaRinne/auto_release_ci 
ci: try to fix error of pushing package.
 2023-10-25 00:08:35 +08:00
Yaohui Liu 0fbca6a38f ci: try to fix error of pushing package. 2023-10-25 00:08:06 +08:00
Yaohui Liu 74e16966fe ci: try to fix error of pushing package. 2023-10-25 00:06:41 +08:00
Rinne 0906b4bf56
Merge pull request #210 from AsakusaRinne/auto_release_ci 
ci: try to fix error of pushing package.
 2023-10-24 09:29:40 +08:00
Yaohui Liu 7551b4a3be ci: try to fix error of pushing package. 2023-10-24 09:28:35 +08:00
Rinne 32662a9be8
Merge pull request #209 from AsakusaRinne/auto_release_ci 
ci: try to fix error of pushing package.
 2023-10-24 09:20:13 +08:00
Yaohui Liu 792215fd99 ci: try to fix error of pushing package. 2023-10-24 09:19:18 +08:00
Martin Evans a03fe003de Fixed decoding of text "accumulating" over time (never properly clearing buffer) 2023-10-23 16:42:38 +01:00
Rinne 376e445760
Merge pull request #207 from AsakusaRinne/auto_release_ci 
ci: try to fix error of pushing package.
 2023-10-23 20:48:42 +08:00
Yaohui Liu b0af0cf4ea ci: try to fix error of pushing package. 2023-10-23 20:47:59 +08:00
Rinne 335544ae49
Merge pull request #206 from AsakusaRinne/auto_release_ci 
ci: fix error of pushing package.
 2023-10-23 20:34:58 +08:00
Yaohui Liu 114b3aa722 Merge branch 'auto_release_ci' of github.com:AsakusaRinne/LLamaSharp into auto_release_ci 2023-10-23 20:32:44 +08:00
Yaohui Liu d0cb185b1d ci: fix error of pushing package. 2023-10-23 20:32:30 +08:00
Rinne 37a306d0bc
Merge pull request #204 from AsakusaRinne/auto_release_ci 2023-10-23 12:12:43 +08:00
Martin Evans 51d4411a58 Added two new classes for detokenization tasks: 
- `AntipromptProcessor` accepts chunks of text and returns a value indicating if any antiprompt has been detected.
 - `StreamingTokenDecoder` decodes tokens into text, maintaining some internal state to handle single characters which are encoded as multiple tokens.

Added tests for these classes and updated StatelessExecutor to use them.

Removed most DeTokenize methods, marked the rest as obsolete (should always use a `StreamingTokenDecoder`).
 2023-10-23 00:33:50 +01:00
Martin Evans efdf3d630c - Removed all `TokenToString` methods (it's never correct to use them, because sometimes one single character may be represented by multiple tokens). 
- Built a new (hacky) `Detokenize` method which handles this
 2023-10-22 21:43:36 +01:00
Rinne 231efe06f2
Update LLama/runtimes/build/LLamaSharp.Backend.Cpu.nuspec 
Co-authored-by: Martin Evans <martindevans@gmail.com>
 2023-10-23 00:01:09 +08:00
Rinne ecf852c4e2
Update LLama/runtimes/build/LLamaSharp.Backend.MacMetal.nuspec 
Co-authored-by: Martin Evans <martindevans@gmail.com>
 2023-10-23 00:00:46 +08:00
Rinne 95669c2ea3
Update LLama/runtimes/build/LLamaSharp.Backend.Cuda12.nuspec 
Co-authored-by: Martin Evans <martindevans@gmail.com>
 2023-10-23 00:00:40 +08:00
Rinne 5eaebd68ba
Update LLama/runtimes/build/LLamaSharp.Backend.Cuda11.nuspec 
Co-authored-by: Martin Evans <martindevans@gmail.com>
 2023-10-23 00:00:34 +08:00
Rinne 6724b39713
Update LLama/runtimes/build/LLamaSharp.Backend.Cpu.nuspec 
Co-authored-by: Martin Evans <martindevans@gmail.com>
 2023-10-23 00:00:27 +08:00
Martin Evans 1d0620e634 Created a test that "roundtrips" strings through tokenization. This reveals some flaws with certain characters 2023-10-22 15:28:36 +01:00
Yaohui Liu 7855e07509 fix: typo in prepare_release.sh. 2023-10-22 18:40:09 +08:00
Rinne 0215ecfd20
Merge pull request #201 from xbotter/sk-examples 
chore: Update LLama.Examples and LLama.SemanticKernel
 2023-10-22 17:54:42 +08:00
Yaohui Liu b7a7dc00b6 ci: fix typos. 2023-10-22 17:50:37 +08:00
Yaohui Liu 252992ec6e ci: fix icon and typos. 2023-10-22 17:45:07 +08:00
Yaohui Liu ae1fe1f995 ci: fix error. 2023-10-22 17:18:23 +08:00
Yaohui Liu b48867839a ci: fix error. 2023-10-22 17:17:29 +08:00
Yaohui Liu 1b86f9ed70 ci: fix error. 2023-10-22 17:12:27 +08:00
Yaohui Liu 96d6edd794 ci: fix error. 2023-10-22 17:04:56 +08:00