This is the final result of my experimentation with the simplest solution I have found. One more step could be done by adding a clever script to pre-configure llama.cpp with the required settings (GPU, AVX2, platform...) and even full automatic build.
* Add llava_binaries, update all binaries to make the test
* Llava API + LlavaTest
Preliminary
* First prototype of Load + Unit Test
* Temporary run test con branch LlavaAPI
* Disable Embed test to review the rest of the test
* Restore Embedding test
* Use BatchThread to eval image embeddings
Test Threads default value to ensure it doesn´t produce problems.
* Rename test file
* Update action versions
* Test only one method, no release embeddings
* Revert "Test only one method, no release embeddings"
This reverts commit 264e176dccc9cd0be318b800ae5e102a4635d01c.
* Correct API call
* Only test llava related functionality
* Cuda and Cblast binaries
* Restore build policy
* Changes related with code review
* Add SafeHandles
* Set overwrite to upload-artifact@v4
* Revert to upload-artifact@v3
* revert to upload-artifact@v3
* Added a lock object into `SafeLlamaModelHandle` which all calls to `llama_decode` (in the `SafeLLamaContextHandle`) lock first. This prevents two contexts from running inference on the same model at the same time, which seems to be unsafe in llama.cpp.
* Modified the lock to be global over _all_ inferences. This seems to be necessary (at least with the CUDA backend).
Modified LLamaBatch to not share tokens with other sequences if logits is true. This ensures that the logit span at the end in used by exactly one sequence - therefore it's safe to mutate. This removes the need for copying _very_ large arrays (vocab size) and simplifies sampling pipelines.
* Added a `Guidance` method to `LLamaTokenDataArray` which applies classifier free guidance
* Factored out a safer `llama_sample_apply_guidance` method based on spans
* Created a guided sampling demo using the batched executor
* fixed comment, "classifier free" not "context free"
* Rebased onto master and fixed breakage due to changes in `BaseSamplingPipeline`
* Asking user for guidance weight
* Progress bar in batched fork demo
* Improved fork example (using tree display)
* Added proper disposal of resources in batched examples
* Added some more comments in BatchedExecutorGuidance
* - Modified ISamplingPipeline to accept `ReadOnlySpan<float>` of logits directly. This moves responsibility to copy the logits into the pipeline.
- Added a flag to `BaseSamplingPipeline` indicating if a logit copy is necessary. Skipping it in most cases.
* Fixed `RestoreProtectedTokens` not working if logit processing is skipped
* - Implemented a new greedy sampling pipeline (always sample most likely token)
- Moved `Grammar` into `BaseSamplingPipeline`
- Removed "protected tokens" concept from `BaseSamplingPipeline`. Was introducing a lot of incidental complexity.
- Implemented newline logit save/restore in `DefaultSamplingPipeline` (only place protected tokens was used)
* Implemented pipelines for mirostat v1 and v2
- UserSettings, simplifying the validation/re-ask loop down to one call
- Program, adding colour to figlet title
- Batched examples, showing default prompt
- ExampleRunner, resetting state after running an example
* LLama.Examples: disable console logging
* LLama.Examples: rename titles to signal grouped topics
* LLama.Examples: add additional PDF for Q&A
* LLama.Examples: improve kernel memory demo
multi-document ingestion
* LLama.Examples: improve message before resetting to main menu
* LLama.Examples: document Q&A with local memory
* LLama.Examples: RepoUtils.cs → ConsoleLogger.cs
* LLama.Examples: Examples/Runner.cs → ExampleRunner.cs
* LLama.Examples: delete unused console logger
* LLama.Examples: improve splash screen appearance
the llama_empty_call() no longer shows configuration information on startup, but it will display it automatically the first time a model is engaged
* LLama.Examples: Runner → ExampleRunner
* LLama.Examples: improve model path prompt
The last used model is stored in a config file and is re-used when a blank path is provided
* LLama.Examples: NativeApi.llama_empty_call() at startup
* LLama.Examples: reduce console noise when saving model path
* Embeddings example: set EmbeddingMode true
prevents an exception from being thrown when GetEmbeddings() is called
* Embeddings example: improve documentation and styling
* docs: improve GetEmbeddings page
If EmbeddingMode is not set to true, GetEmbeddings() throws an exception
* docs: improve GetEmbeddings page
The previous commit 6c9ff3158c was inaccurate
* Embeddings example: improve styling
displays the example description after the model is loaded to ensure the text is on the screen at the time the prompt is first requested
Rinne
Zoli Somogyi