The figure below shows the core framework structure, which is separated to four levels.
+LLamaModel, LLamaEmbedder and LLamaQuantizer.InteractiveExecutor, InstructuExecutor and StatelessExecutor.InteractiveExecutor and LLamaModel, which supports interactive tasks and saving/re-loading sessions. It also provides a flexible way to customize the text process by IHistoryTransform, ITextTransform and ITextStreamTransform.
Since LLamaModel interact with native library, it's not recommended to use the methods of it directly unless you know what you are doing. So does the NativeApi, which is not included in the arcitecher figure above.
ChatSession is recommended to be used when you want to build an application similar to ChatGPT, or the ChatBot, because it works best with InteractiveExecutor. Though other executors are also allowed to passed as a parameter to initialize a ChatSession, it's not encouraged if you are new to LLamaSharp and LLM.
High-level applications, such as BotSharp, are supposed to be used when you concentrate on the part not related with LLM. For example, if you want to deploy a chat bot to help you remember your schedules, using BotSharp may be a good choice.
+Note that the APIs of the high-level applications may not be stable now. Please take it into account when using them.
+ + + + + + +ChatSession is a higher-level absatrction than the executors. In the context of a chat application like ChatGPT, a "chat session" refers to an interactive conversation or exchange of messages between the user and the chatbot. It represents a continuous flow of communication where the user enters input or asks questions, and the chatbot responds accordingly. A chat session typically starts when the user initiates a conversation with the chatbot and continues until the interaction comes to a natural end or is explicitly terminated by either the user or the system. During a chat session, the chatbot maintains the context of the conversation, remembers previous messages, and generates appropriate responses based on the user's inputs and the ongoing dialogue.
Currently, the only parameter that is accepted is an ILLamaExecutor, because this is the only parameter that we're sure to exist in all the future versions. Since it's the high-level absatrction, we're conservative to the API designs. In the future, there may be more kinds of constructors added.
InteractiveExecutor ex = new(new LLamaModel(new ModelParams(modelPath)));
+ChatSession session = new ChatSession(ex);
+There'll be two kinds of input accepted by the Chat API, which are ChatHistory and String. The API with string is quite similar to that of the executors. Meanwhile, the API with ChatHistory is aimed to provide more flexible usages. For example, you have had a chat with the bot in session A before you open the session B. Now session B has no memory for what you said before. Therefore, you can feed the history of A to B.
string prompt = "What is C#?";
+
+foreach (var text in session.Chat(prompt, new InferenceParams() { Temperature = 0.6f, AntiPrompts = new List<string> { "User:" } })) // the inference params should be changed depending on your statement
+{
+ Console.Write(text);
+}
+Currently History is a property of ChatSession.
foreach(var rec in session.History.Messages)
+{
+ Console.WriteLine($"{rec.AuthorRole}: {rec.Content}");
+}
+Generally, the chat session could be switched, which requires the ability of loading and saving session.
+When building a chat bot app, it's NOT encouraged to initialize many chat sessions and keep them in memory to wait for being switched, because the memory comsumption of both CPU and GPU is expensive. It's recommended to save the current session before switching to a new session, and load the file when switching back to the session.
+The API is also quite simple, the files will be saved into a directory you specified. If the path does not exist, a new directory will be created.
+string savePath = "<save dir>";
+session.SaveSession(savePath);
+
+session.LoadSession(savePath);
+There's three important elements in ChatSession, which are input, output and history. Besides, there're some conversions between them. Since the process of them under different conditions varies, LLamaSharp hands over this part of the power to the users.
Currently, there're three kinds of process that could be customized, as introduced below.
+In general, the input of the chat API is a text (without stream), therefore ChatSession processes it in a pipeline. If you want to use your customized transform, you need to define a transform that implements ITextTransform and add it to the pipeline of ChatSession.
public interface ITextTransform
+{
+ string Transform(string text);
+}
+public class MyInputTransform1 : ITextTransform
+{
+ public string Transform(string text)
+ {
+ return $"Question: {text}\n";
+ }
+}
+
+public class MyInputTransform2 : ITextTransform
+{
+ public string Transform(string text)
+ {
+ return text + "Answer: ";
+ }
+}
+
+session.AddInputTransform(new MyInputTransform1()).AddInputTransform(new MyInputTransform2());
+Different from the input, the output of chat API is a text stream. Therefore you need to process it word by word, instead of getting the full text at once.
+The interface of it has an IEnumerable<string> as input, which is actually a yield sequence.
public interface ITextStreamTransform
+{
+ IEnumerable<string> Transform(IEnumerable<string> tokens);
+ IAsyncEnumerable<string> TransformAsync(IAsyncEnumerable<string> tokens);
+}
+When implementing it, you could throw a not-implemented exception in one of them if you only need to use the chat API in synchronously or asynchronously.
+Different from the input transform pipeline, the output transform only supports one transform.
+session.WithOutputTransform(new MyOutputTransform());
+Here's an example of how to implement the interface. In this example, the transform detects wether there's some keywords in the response and removes them.
+/// <summary>
+/// A text output transform that removes the keywords from the response.
+/// </summary>
+public class KeywordTextOutputStreamTransform : ITextStreamTransform
+{
+ HashSet<string> _keywords;
+ int _maxKeywordLength;
+ bool _removeAllMatchedTokens;
+
+ /// <summary>
+ ///
+ /// </summary>
+ /// <param name="keywords">Keywords that you want to remove from the response.</param>
+ /// <param name="redundancyLength">The extra length when searching for the keyword. For example, if your only keyword is "highlight",
+ /// maybe the token you get is "\r\nhighligt". In this condition, if redundancyLength=0, the token cannot be successfully matched because the length of "\r\nhighligt" (10)
+ /// has already exceeded the maximum length of the keywords (8). On the contrary, setting redundancyLengyh >= 2 leads to successful match.
+ /// The larger the redundancyLength is, the lower the processing speed. But as an experience, it won't introduce too much performance impact when redundancyLength <= 5 </param>
+ /// <param name="removeAllMatchedTokens">If set to true, when getting a matched keyword, all the related tokens will be removed. Otherwise only the part of keyword will be removed.</param>
+ public KeywordTextOutputStreamTransform(IEnumerable<string> keywords, int redundancyLength = 3, bool removeAllMatchedTokens = false)
+ {
+ _keywords = new(keywords);
+ _maxKeywordLength = keywords.Select(x => x.Length).Max() + redundancyLength;
+ _removeAllMatchedTokens = removeAllMatchedTokens;
+ }
+ /// <inheritdoc />
+ public IEnumerable<string> Transform(IEnumerable<string> tokens)
+ {
+ var window = new Queue<string>();
+
+ foreach (var s in tokens)
+ {
+ window.Enqueue(s);
+ var current = string.Join("", window);
+ if (_keywords.Any(x => current.Contains(x)))
+ {
+ var matchedKeyword = _keywords.First(x => current.Contains(x));
+ int total = window.Count;
+ for (int i = 0; i < total; i++)
+ {
+ window.Dequeue();
+ }
+ if (!_removeAllMatchedTokens)
+ {
+ yield return current.Replace(matchedKeyword, "");
+ }
+ }
+ if (current.Length >= _maxKeywordLength)
+ {
+ if (_keywords.Any(x => current.Contains(x)))
+ {
+ var matchedKeyword = _keywords.First(x => current.Contains(x));
+ int total = window.Count;
+ for (int i = 0; i < total; i++)
+ {
+ window.Dequeue();
+ }
+ if (!_removeAllMatchedTokens)
+ {
+ yield return current.Replace(matchedKeyword, "");
+ }
+ }
+ else
+ {
+ int total = window.Count;
+ for (int i = 0; i < total; i++)
+ {
+ yield return window.Dequeue();
+ }
+ }
+ }
+ }
+ int totalCount = window.Count;
+ for (int i = 0; i < totalCount; i++)
+ {
+ yield return window.Dequeue();
+ }
+ }
+ /// <inheritdoc />
+ public async IAsyncEnumerable<string> TransformAsync(IAsyncEnumerable<string> tokens)
+ {
+ throw new NotImplementedException(); // This is implemented in `LLamaTransforms` but we ignore it here.
+ }
+}
+The chat history could be converted to or from a text, which is exactly what the interface of it.
+public interface IHistoryTransform
+{
+ string HistoryToText(ChatHistory history);
+ ChatHistory TextToHistory(AuthorRole role, string text);
+}
+Similar to the output transform, the history transform is added in the following way:
+session.WithHistoryTransform(new MyHistoryTransform());
+The implementation is quite flexible, depending on what you want the history message to be like. Here's an example, which is the default history transform in LLamaSharp.
+/// <summary>
+/// The default history transform.
+/// Uses plain text with the following format:
+/// [Author]: [Message]
+/// </summary>
+public class DefaultHistoryTransform : IHistoryTransform
+{
+ private readonly string defaultUserName = "User";
+ private readonly string defaultAssistantName = "Assistant";
+ private readonly string defaultSystemName = "System";
+ private readonly string defaultUnknownName = "??";
+
+ string _userName;
+ string _assistantName;
+ string _systemName;
+ string _unknownName;
+ bool _isInstructMode;
+ public DefaultHistoryTransform(string? userName = null, string? assistantName = null,
+ string? systemName = null, string? unknownName = null, bool isInstructMode = false)
+ {
+ _userName = userName ?? defaultUserName;
+ _assistantName = assistantName ?? defaultAssistantName;
+ _systemName = systemName ?? defaultSystemName;
+ _unknownName = unknownName ?? defaultUnknownName;
+ _isInstructMode = isInstructMode;
+ }
+
+ public virtual string HistoryToText(ChatHistory history)
+ {
+ StringBuilder sb = new();
+ foreach (var message in history.Messages)
+ {
+ if (message.AuthorRole == AuthorRole.User)
+ {
+ sb.AppendLine($"{_userName}: {message.Content}");
+ }
+ else if (message.AuthorRole == AuthorRole.System)
+ {
+ sb.AppendLine($"{_systemName}: {message.Content}");
+ }
+ else if (message.AuthorRole == AuthorRole.Unknown)
+ {
+ sb.AppendLine($"{_unknownName}: {message.Content}");
+ }
+ else if (message.AuthorRole == AuthorRole.Assistant)
+ {
+ sb.AppendLine($"{_assistantName}: {message.Content}");
+ }
+ }
+ return sb.ToString();
+ }
+
+ public virtual ChatHistory TextToHistory(AuthorRole role, string text)
+ {
+ ChatHistory history = new ChatHistory();
+ history.AddMessage(role, TrimNamesFromText(text, role));
+ return history;
+ }
+
+ public virtual string TrimNamesFromText(string text, AuthorRole role)
+ {
+ if (role == AuthorRole.User && text.StartsWith($"{_userName}:"))
+ {
+ text = text.Substring($"{_userName}:".Length).TrimStart();
+ }
+ else if (role == AuthorRole.Assistant && text.EndsWith($"{_assistantName}:"))
+ {
+ text = text.Substring(0, text.Length - $"{_assistantName}:".Length).TrimEnd();
+ }
+ if (_isInstructMode && role == AuthorRole.Assistant && text.EndsWith("\n> "))
+ {
+ text = text.Substring(0, text.Length - "\n> ".Length).TrimEnd();
+ }
+ return text;
+ }
+}
+Hi, welcome to develop LLamaSharp with us together! We are always open for every contributor and any format of contributions! If you want to maintain this library actively together, please contact us to get the write access after some PRs. (Email: AsakusaRinne@gmail.com)
+In this page, we'd like to introduce how to make contributions here easily. 😊
+Firstly, please clone the llama.cpp repository and following the instructions in llama.cpp readme to configure your local environment.
+If you want to support cublas in the compilation, please make sure that you've installed the cuda.
+When building from source, please add -DBUILD_SHARED_LIBS=ON to the cmake instruction. For example, when building with cublas but without openblas, use the following instruction:
cmake .. -DLLAMA_CUBLAS=ON -DBUILD_SHARED_LIBS=ON
+After running cmake --build . --config Release, you could find the llama.dll, llama.so or llama.dylib in your build directory. After pasting it to LLamaSharp/LLama/runtimes and renaming it to libllama.dll, libllama.so or libllama.dylib, you can use it as the native library in LLamaSharp.
After refactoring the framework in v0.4.0, LLamaSharp will try to maintain the backward compatibility. However, in the following cases, break change is okay:
If a new feature could be added without introducing any break change, please open a PR rather than open an issue first. We will never refuse the PR but help to improve it, unless it's malicious.
+When adding the feature, please take care of the namespace and the naming convention. For example, if you are adding an integration for WPF, please put the code under namespace LLama.WPF or LLama.Integration.WPF instead of putting it under the root namespace. The naming convention of LLamaSharp follows the pascal naming convention, but in some parts that are invisible to users, you can do whatever you want.
If the issue is related to the LLM internal behaviors, such as endless generating the response, the best way to find the problem is to do comparison test between llama.cpp and LLamaSharp.
+You could use exactly the same prompt, the same model and the same parameters to run the inference in llama.cpp and LLamaSharp respectively to see if it's really a problem caused by the implementation in LLamaSharp.
+If the experiment showed that it worked well in llama.cpp but didn't in LLamaSharp, a the search for the problem could be started. While the reason of the problem could be various, the best way I think is to add log-print in the code of llama.cpp and use it in LLamaSharp after compilation. Thus, when running LLamaSharp, you could see what happened in the native library.
+After finding out the reason, a painful but happy process comes. When working on the BUG fix, there's only one rule to follow, that is keeping the examples working well. If the modification fixed the BUG but impact on other functions, it would not be a good fix.
+During the BUG fix process, please don't hesitate to discuss together when you stuck on something.
+All kinds of integration are welcomed here! Currently the following integrations are under work or on our schedule:
+Besides, for some other integrations, like ASP.NET core, SQL, Blazor and so on, we'll appreciate it if you could help with that. If the time is limited for you, providing an example for it also means a lot!
There're mainly two ways to add an example:
+LLama.Examples of the repository.LLamaSharp uses mkdocs to build the documantation, please follow the tutorial of mkdocs to add or modify documents in LLamaSharp.
+ + + + + + +Firstly, search LLamaSharp in nuget package manager and install it.
PM> Install-Package LLamaSharp
+Then, search and install one of the following backends:
+LLamaSharp.Backend.Cpu
+LLamaSharp.Backend.Cuda11
+LLamaSharp.Backend.Cuda12
+Here's the mapping of them and corresponding model samples provided by LLamaSharp. If you're not sure which model is available for a version, please try our sample model.
| LLamaSharp.Backend | +LLamaSharp | +Verified Model Resources | +llama.cpp commit id | +
|---|---|---|---|
| - | +v0.2.0 | +This version is not recommended to use. | +- | +
| - | +v0.2.1 | +WizardLM, Vicuna (filenames with "old") | +- | +
| v0.2.2 | +v0.2.2, v0.2.3 | +WizardLM, Vicuna (filenames without "old") | +63d2046 | +
| v0.3.0 | +v0.3.0 | +LLamaSharpSamples v0.3.0, WizardLM | +7e4ea5b | +
One of the following models could be okay:
+Note that because llama.cpp is under fast development now and often introduce break changes, some model weights on huggingface which works under a version may be invalid with another version. If it's your first time to configure LLamaSharp, we'd like to suggest for using verified model weights in the table above.
Please create a console program with dotnet runtime >= netstandard 2.0 (>= net6.0 is more recommended). Then, paste the following code to program.cs;
using LLama.Common;
+using LLama;
+
+string modelPath = "<Your model path>" // change it to your own model path
+var prompt = "Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.\r\n\r\nUser: Hello, Bob.\r\nBob: Hello. How may I help you today?\r\nUser: Please tell me the largest city in Europe.\r\nBob: Sure. The largest city in Europe is Moscow, the capital of Russia.\r\nUser:"; // use the "chat-with-bob" prompt here.
+
+// Initialize a chat session
+var ex = new InteractiveExecutor(new LLamaModel(new ModelParams(modelPath, contextSize: 1024, seed: 1337, gpuLayerCount: 5)));
+ChatSession session = new ChatSession(ex);
+
+// show the prompt
+Console.WriteLine();
+Console.Write(prompt);
+
+// run the inference in a loop to chat with LLM
+while (true)
+{
+ foreach (var text in session.Chat(prompt, new InferenceParams() { Temperature = 0.6f, AntiPrompts = new List<string> { "User:" } }))
+ {
+ Console.Write(text);
+ }
+
+ Console.ForegroundColor = ConsoleColor.Green;
+ prompt = Console.ReadLine();
+ Console.ForegroundColor = ConsoleColor.White;
+}
+After starting it, you'll see the following outputs.
+Please input your model path: D:\development\llama\weights\wizard-vicuna-13B.ggmlv3.q4_1.bin
+llama.cpp: loading model from D:\development\llama\weights\wizard-vicuna-13B.ggmlv3.q4_1.bin
+llama_model_load_internal: format = ggjt v3 (latest)
+llama_model_load_internal: n_vocab = 32000
+llama_model_load_internal: n_ctx = 1024
+llama_model_load_internal: n_embd = 5120
+llama_model_load_internal: n_mult = 256
+llama_model_load_internal: n_head = 40
+llama_model_load_internal: n_layer = 40
+llama_model_load_internal: n_rot = 128
+llama_model_load_internal: ftype = 3 (mostly Q4_1)
+llama_model_load_internal: n_ff = 13824
+llama_model_load_internal: n_parts = 1
+llama_model_load_internal: model size = 13B
+llama_model_load_internal: ggml ctx size = 7759.48 MB
+llama_model_load_internal: mem required = 9807.48 MB (+ 1608.00 MB per state)
+....................................................................................................
+llama_init_from_file: kv self size = 800.00 MB
+
+Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.
+
+User: Hello, Bob.
+Bob: Hello. How may I help you today?
+User: Please tell me the largest city in Europe.
+Bob: Sure. The largest city in Europe is Moscow, the capital of Russia.
+User:
+Now, enjoy chatting with LLM!
+ + + + + + +The document is under work, please have a wait. Thank you for your support! :)
+ + + + + + +There're currently three kinds of executors provided, which are InteractiveExecutor, InstructExecutor and StatelessExecutor.
In a word, InteractiveExecutor is suitable for getting answer of your questions from LLM continuously. InstructExecutor let LLM execute your instructions, such as "continue writing". StatelessExecutor is best for one-time job because the previous inference has no impact on the current inference.
Both of them are taking "completing the prompt" as the goal to generate the response. For example, if you input Long long ago, there was a fox who wanted to make friend with humen. One day, then the LLM will continue to write the story.
Under interactive mode, you serve a role of user and the LLM serves the role of assistant. Then it will help you with your question or request.
+Under instruct mode, you give LLM some instructions and it follows.
+Though the behaviors of them sounds similar, it could introduce many differences depending on your prompt. For example, "chat-with-bob" has good performance under interactive mode and alpaca does well with instruct mode.
// chat-with-bob
+
+Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.
+
+User: Hello, Bob.
+Bob: Hello. How may I help you today?
+User: Please tell me the largest city in Europe.
+Bob: Sure. The largest city in Europe is Moscow, the capital of Russia.
+User:
+// alpaca
+
+Below is an instruction that describes a task. Write a response that appropriately completes the request.
+Therefore, please modify the prompt correspondingly when switching from one mode to the other.
+Despite the differences between interactive mode and instruct mode, both of them are stateful mode. That is, your previous question/instruction will impact on the current response from LLM. On the contrary, the steteless executor does not have such a "memory". No matter how many times you talk to it, it will only concentrate on what you say in this time.
+Since the stateless executor has no memory of conversations before, you need to input your question with the whole prompt into it to get the better answer.
+For example, if you feed Q: Who is Trump? A: to the steteless executor, it may give the following answer with the antiprompt Q:.
Donald J. Trump, born June 14, 1946, is an American businessman, television personality, politician and the 45th President of the United States (2017-2021). # Anexo:Torneo de Hamburgo 2022 (individual masculino)
+
+## Presentación previa
+
+* Defensor del título: Daniil Medvédev
+It seems that things went well at first. However, after answering the question itself, LLM began to talk about some other things until the answer reached the token count limit. The reason of this strange behavior is the anti-prompt cannot be match. With the input, LLM cannot decide whether to append a string "A: " at the end of the response.
+As an improvement, let's take the following text as the input:
+Q: What is the capital of the USA? A: Washingtong. Q: What is the sum of 1 and 2? A: 3. Q: Who is Trump? A:
+Then, I got the following answer with the anti-prompt Q:.
45th president of the United States.
+At this time, by repeating the same mode of Q: xxx? A: xxx., LLM outputs the anti-prompt we want to help to decide where to dtop the generation.
Different from LLamaModel, when using an exeuctor, InferenceParams is passed to the Infer method instead of constructor. This is because executors only define the ways to run the model, therefore in each run, you can change the settings for this time inference.
Namespace: LLama.Common
+public class InferenceParams
+Inheritance Object → InferenceParams
+number of tokens to keep from initial prompt
+public int TokensKeep { get; set; }
+how many new tokens to predict (n_predict), set to -1 to inifinitely generate response + until it complete.
+public int MaxTokens { get; set; }
+logit bias for specific tokens
+public Dictionary<int, float> LogitBias { get; set; }
+Sequences where the model will stop generating further tokens.
+public IEnumerable<string> AntiPrompts { get; set; }
+path to file for saving/loading model eval state
+public string PathSession { get; set; }
+string to suffix user inputs with
+public string InputSuffix { get; set; }
+string to prefix user inputs with
+public string InputPrefix { get; set; }
+0 or lower to use vocab size
+public int TopK { get; set; }
+1.0 = disabled
+public float TopP { get; set; }
+1.0 = disabled
+public float TfsZ { get; set; }
+1.0 = disabled
+public float TypicalP { get; set; }
+1.0 = disabled
+public float Temperature { get; set; }
+1.0 = disabled
+public float RepeatPenalty { get; set; }
+last n tokens to penalize (0 = disable penalty, -1 = context size) (repeat_last_n)
+public int RepeatLastTokensCount { get; set; }
+frequency penalty coefficient + 0.0 = disabled
+public float FrequencyPenalty { get; set; }
+presence penalty coefficient + 0.0 = disabled
+public float PresencePenalty { get; set; }
+Mirostat uses tokens instead of words. + algorithm described in the paper https://arxiv.org/abs/2007.14966. + 0 = disabled, 1 = mirostat, 2 = mirostat 2.0
+public MiroStateType Mirostat { get; set; }
+target entropy
+public float MirostatTau { get; set; }
+learning rate
+public float MirostatEta { get; set; }
+consider newlines as a repeatable token (penalize_nl)
+public bool PenalizeNL { get; set; }
+Similar to LLamaModel, an executor also has its state, which can be saved and loaded. Note that in most of cases, the state of executor and the state of the model should be loaded and saved at the same time.
To decouple the model and executor, we provide APIs to save/load state for model and executor respectively. However, during the inference, the processed information will leave footprint in LLamaModel's native context. Therefore, if you just load a state from another executor but keep the model unmodified, some strange things may happen. So will loading model state only.
Is there a condition that requires to load one of them only? The answer is YES. For example, after resetting the model state, if you don't want the inference starting from the new position, leaving the executor unmodified is okay. But, anyway, this flexible usage may cause some unexpected behaviors, therefore please ensure you know what you're doing before using it in this way.
+In the future version, we'll open the access for some variables inside the executor to support more flexible usages.
+The APIs to load/save state of the executors is similar to that of LLamaModel. However, note that StatelessExecutor doesn't have such APIs because it's stateless itself. Besides, the output of GetStateData is an object of type ExecutorBaseState.
LLamaModel model = new LLamaModel(new ModelParams("<modelPath>"));
+InteractiveExecutor executor = new InteractiveExecutor(model);
+// do some things...
+executor.SaveState("executor.st");
+var stateData = model.GetStateData();
+
+InteractiveExecutor executor2 = new InteractiveExecutor(model);
+executor2.LoadState(stateData);
+// do some things...
+
+InteractiveExecutor executor3 = new InteractiveExecutor(model);
+executor3.LoadState("executor.st");
+// do some things...
+All the executors implements the interface ILLamaExecutor, which provides two APIs to execute text-to-text tasks.
public interface ILLamaExecutor
+{
+ public LLamaModel Model { get; }
+
+ IEnumerable<string> Infer(string text, InferenceParams? inferenceParams = null, CancellationToken token = default);
+
+ IAsyncEnumerable<string> InferAsync(string text, InferenceParams? inferenceParams = null, CancellationToken token = default);
+}
+Just pass the text to the executor with the inference parameters. For the inference parameters, please refer to executor inference parameters doc.
+The output of both two APIs are yield enumerable. Therefore, when receiving the output, you can directly use foreach to take actions on each word you get by order, instead of waiting for the whole process completed.
Getting the embeddings of a text in LLM is sometimes useful, for example, to train other MLP models.
+To get the embeddings, please initialize a LLamaEmbedder and then call GetEmbeddings.
var embedder = new LLamaEmbedder(new ModelParams("<modelPath>"));
+string text = "hello, LLM.";
+float[] embeddings = embedder.GetEmbeddings(text);
+The output is a float array. Note that the length of the array is related with the model you load. If you just want to get a smaller size embedding, please consider changing a model.
+ + + + + + +When initializing a LLamaModel object, there're three parameters, ModelParams Params, string encoding = "UTF-8", ILLamaLogger? logger = null.
The usage of logger will be further introduced in logger doc. The encoding is the encoding you want to use when dealing with text via this model.
The most improtant of all, is the ModelParams, which is defined as below. We'll explain the parameters step by step in this document.
public class ModelParams
+{
+ public int ContextSize { get; set; } = 512;
+ public int GpuLayerCount { get; set; } = 20;
+ public int Seed { get; set; } = 1686349486;
+ public bool UseFp16Memory { get; set; } = true;
+ public bool UseMemorymap { get; set; } = true;
+ public bool UseMemoryLock { get; set; } = false;
+ public bool Perplexity { get; set; } = false;
+ public string ModelPath { get; set; }
+ public string LoraAdapter { get; set; } = string.Empty;
+ public string LoraBase { get; set; } = string.Empty;
+ public int Threads { get; set; } = Math.Max(Environment.ProcessorCount / 2, 1);
+ public int BatchSize { get; set; } = 512;
+ public bool ConvertEosToNewLine { get; set; } = false;
+}
+Namespace: LLama.Common
+public class ModelParams
+Inheritance Object → ModelParams
+Model context size (n_ctx)
+public int ContextSize { get; set; }
+Number of layers to run in VRAM / GPU memory (n_gpu_layers)
+public int GpuLayerCount { get; set; }
+Seed for the random number generator (seed)
+public int Seed { get; set; }
+Use f16 instead of f32 for memory kv (memory_f16)
+public bool UseFp16Memory { get; set; }
+Use mmap for faster loads (use_mmap)
+public bool UseMemorymap { get; set; }
+Use mlock to keep model in memory (use_mlock)
+public bool UseMemoryLock { get; set; }
+Compute perplexity over the prompt (perplexity)
+public bool Perplexity { get; set; }
+Model path (model)
+public string ModelPath { get; set; }
+lora adapter path (lora_adapter)
+public string LoraAdapter { get; set; }
+base model path for the lora adapter (lora_base)
+public string LoraBase { get; set; }
+Number of threads (-1 = autodetect) (n_threads)
+public int Threads { get; set; }
+batch size for prompt processing (must be >=32 to use BLAS) (n_batch)
+public int BatchSize { get; set; }
+Whether to convert eos to newline during the inference.
+public bool ConvertEosToNewLine { get; set; }
+Whether to use embedding mode. (embedding) Note that if this is set to true, + The LLamaModel won't produce text response anymore.
+public bool EmbeddingMode { get; set; }
+Quantization is significant to accelerate the model inference. Since there's little accuracy (performance) reduction when quantizing the model, get it easy to quantize it!
+To quantize the model, please call Quantize from LLamaQuantizer, which is a static method.
string srcPath = "<model.bin>";
+string dstPath = "<model_q4_0.bin>";
+LLamaQuantizer.Quantize(srcPath, dstPath, "q4_0");
+// The following overload is also okay.
+// LLamaQuantizer.Quantize(srcPath, dstPath, LLamaFtype.LLAMA_FTYPE_MOSTLY_Q4_0);
+After calling it, a quantized model file will be saved.
+There're currently 5 types of quantization supported:
+There're two ways to load state: loading from path and loading from bite array. Therefore, correspondingly, state data can be extracted as byte array or saved to a file.
+LLamaModel model = new LLamaModel(new ModelParams("<modelPath>"));
+// do some things...
+model.SaveState("model.st");
+var stateData = model.GetStateData();
+model.Dispose();
+
+LLamaModel model2 = new LLamaModel(new ModelParams("<modelPath>"));
+model2.LoadState(stateData);
+// do some things...
+
+LLamaModel model3 = new LLamaModel(new ModelParams("<modelPath>"));
+model3.LoadState("model.st");
+// do some things...
+A pair of APIs to make conversion between text and tokens.
+The basic usage is to call Tokenize after initializing the model.
LLamaModel model = new LLamaModel(new ModelParams("<modelPath>"));
+string text = "hello";
+int[] tokens = model.Tokenize(text).ToArray();
+Depending on different model (or vocab), the output will be various.
+Similar to tokenization, just pass an IEnumerable<int> to Detokenize method.
LLamaModel model = new LLamaModel(new ModelParams("<modelPath>"));
+int[] tokens = new int[] {125, 2568, 13245};
+string text = model.Detokenize(tokens);
+It's supported now but the document is under work. Please wait for some time. Thank you for your support! :)
+ + + + + + +Sometimes, your application with LLM and LLamaSharp may have strange behaviors. Before opening an issue to report the BUG, the following tricks may worth a try.
+Anti-prompt can also be called as "Stop-keyword", which decides when to stop the response generation. Under interactive mode, the maximum tokens count is always not set, which makes the LLM generates responses infinitively. Therefore, setting anti-prompt correctly helps a lot to avoid the strange behaviors. For example, the prompt file chat-with-bob.txt has the following content:
Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.
+
+User: Hello, Bob.
+Bob: Hello. How may I help you today?
+User: Please tell me the largest city in Europe.
+Bob: Sure. The largest city in Europe is Moscow, the capital of Russia.
+User:
+Therefore, the anti-prompt should be set as "User:". If the last line of the prompt is removed, LLM will automatically generate a question (user) and a response (bob) for one time when running the chat session. Therefore, the antiprompt is suggested to be appended to the prompt when starting a chat session.
+What if an extra line is appended? The string "User:" in the prompt will be followed with a char "\n". Thus when running the model, the automatic generation of a pair of question and response may appear because the anti-prompt is "User:" but the last token is "User:\n". As for whether it will appear, it's an undefined behavior, which depends on the implementation inside the LLamaExecutor. Anyway, since it may leads to unexpected behaviors, it's recommended to trim your prompt or carefully keep consistent with your anti-prompt.
Sometimes we want to input a long prompt to execute a task. However, the context size may limit the inference of LLama model. Please ensure the inequality below holds.
+$$ len(prompt) + len(response) < len(context) $$
+In this inequality, len(response) refers to the expected tokens for LLM to generate.
Some prompt works well under interactive mode, such as chat-with-bob, some others may work well with instruct mode, such as alpaca. Besides, if your input is quite simple and one-time job, such as "Q: what is the satellite of the earth? A: ", stateless mode will be a good choice.
If your chat bot has bad performance, trying different executor will possibly make it work well.
+The differences between modes may lead to much different behaviors under the same task. For example, if you're building a chat bot with non-English, a fine-tuned model specially for the language you want to use will have huge effect on the performance.
+Currently, the GpuLayerCount param, which decides the number of layer loaded into GPU, is set to 20 by default. However, if you have some efficient GPUs, setting it as a larger number will attain faster inference.
LLamaSharp is the C#/.NET binding of llama.cpp. It provides APIs to inference the LLaMa Models and deploy it on native environment or Web. It could help C# developers to deploy the LLM (Large Language Model) locally and integrate with C# apps.
+If you are new to LLM, here're some tips for you to help you to get start with LLamaSharp. If you are experienced in this field, we'd still recommend you to take a few minutes to read it because somethings performs differently compared to cpp/python.
LLamaSharp and LLama.Backend. After installing LLamaSharp, please install one of LLama.Backend.Cpu, LLama.Backend.Cuda11 and LLama.Backend.Cuda12. If you use the source code, dynamic libraries could be found in LLama/Runtimes. Then rename the one you want to use to libllama.dll.LLaMa originally refers to the weights released by Meta (Facebook Research). After that, many models are fine-tuned based on it, such as Vicuna, GPT4All, and Pyglion. Though all of these models are supported by LLamaSharp, some steps are necessary with different file formats. There're mainly three kinds of files, which are .pth, .bin (ggml), .bin (quantized). If you have the .bin (quantized) file, it could be used directly by LLamaSharp. If you have the .bin (ggml) file, you could use it directly but get higher inference speed after the quantization. If you have the .pth file, you need to follow the instructions in llama.cpp to convert it to .bin (ggml) file at first.Community effort is always one of the most important things in open-source projects. Any contribution in any way is welcomed here. For example, the following things mean a lot for LLamaSharp:
+If you'd like to get deeply involved in development, please touch us in discord channel or send email to AsakusaRinne@gmail.com. :)
LLamaSharp is the C#/.NET binding of llama.cpp. It provides APIs to inference the LLaMa Models and deploy it on native environment or Web. It could help C# developers to deploy the LLM (Large Language Model) locally and integrate with C# apps.
"},{"location":"#main-features","title":"Main features","text":"If you are new to LLM, here're some tips for you to help you to get start with LLamaSharp. If you are experienced in this field, we'd still recommend you to take a few minutes to read it because somethings performs differently compared to cpp/python.
LLamaSharp and LLama.Backend. After installing LLamaSharp, please install one of LLama.Backend.Cpu, LLama.Backend.Cuda11 and LLama.Backend.Cuda12. If you use the source code, dynamic libraries could be found in LLama/Runtimes. Then rename the one you want to use to libllama.dll.LLaMa originally refers to the weights released by Meta (Facebook Research). After that, many models are fine-tuned based on it, such as Vicuna, GPT4All, and Pyglion. Though all of these models are supported by LLamaSharp, some steps are necessary with different file formats. There're mainly three kinds of files, which are .pth, .bin (ggml), .bin (quantized). If you have the .bin (quantized) file, it could be used directly by LLamaSharp. If you have the .bin (ggml) file, you could use it directly but get higher inference speed after the quantization. If you have the .pth file, you need to follow the instructions in llama.cpp to convert it to .bin (ggml) file at first.Community effort is always one of the most important things in open-source projects. Any contribution in any way is welcomed here. For example, the following things mean a lot for LLamaSharp:
If you'd like to get deeply involved in development, please touch us in discord channel or send email to AsakusaRinne@gmail.com. :)
The figure below shows the core framework structure, which is separated to four levels.
LLamaModel, LLamaEmbedder and LLamaQuantizer.InteractiveExecutor, InstructuExecutor and StatelessExecutor.InteractiveExecutor and LLamaModel, which supports interactive tasks and saving/re-loading sessions. It also provides a flexible way to customize the text process by IHistoryTransform, ITextTransform and ITextStreamTransform.Since LLamaModel interact with native library, it's not recommended to use the methods of it directly unless you know what you are doing. So does the NativeApi, which is not included in the arcitecher figure above.
ChatSession is recommended to be used when you want to build an application similar to ChatGPT, or the ChatBot, because it works best with InteractiveExecutor. Though other executors are also allowed to passed as a parameter to initialize a ChatSession, it's not encouraged if you are new to LLamaSharp and LLM.
High-level applications, such as BotSharp, are supposed to be used when you concentrate on the part not related with LLM. For example, if you want to deploy a chat bot to help you remember your schedules, using BotSharp may be a good choice.
Note that the APIs of the high-level applications may not be stable now. Please take it into account when using them.
"},{"location":"ContributingGuide/","title":"LLamaSharp Contributing Guide","text":"Hi, welcome to develop LLamaSharp with us together! We are always open for every contributor and any format of contributions! If you want to maintain this library actively together, please contact us to get the write access after some PRs. (Email: AsakusaRinne@gmail.com)
In this page, we'd like to introduce how to make contributions here easily. \ud83d\ude0a
"},{"location":"ContributingGuide/#compile-the-native-library-from-source","title":"Compile the native library from source","text":"Firstly, please clone the llama.cpp repository and following the instructions in llama.cpp readme to configure your local environment.
If you want to support cublas in the compilation, please make sure that you've installed the cuda.
When building from source, please add -DBUILD_SHARED_LIBS=ON to the cmake instruction. For example, when building with cublas but without openblas, use the following instruction:
cmake .. -DLLAMA_CUBLAS=ON -DBUILD_SHARED_LIBS=ON\nAfter running cmake --build . --config Release, you could find the llama.dll, llama.so or llama.dylib in your build directory. After pasting it to LLamaSharp/LLama/runtimes and renaming it to libllama.dll, libllama.so or libllama.dylib, you can use it as the native library in LLamaSharp.
After refactoring the framework in v0.4.0, LLamaSharp will try to maintain the backward compatibility. However, in the following cases, break change is okay:
If a new feature could be added without introducing any break change, please open a PR rather than open an issue first. We will never refuse the PR but help to improve it, unless it's malicious.
When adding the feature, please take care of the namespace and the naming convention. For example, if you are adding an integration for WPF, please put the code under namespace LLama.WPF or LLama.Integration.WPF instead of putting it under the root namespace. The naming convention of LLamaSharp follows the pascal naming convention, but in some parts that are invisible to users, you can do whatever you want.
If the issue is related to the LLM internal behaviors, such as endless generating the response, the best way to find the problem is to do comparison test between llama.cpp and LLamaSharp.
You could use exactly the same prompt, the same model and the same parameters to run the inference in llama.cpp and LLamaSharp respectively to see if it's really a problem caused by the implementation in LLamaSharp.
If the experiment showed that it worked well in llama.cpp but didn't in LLamaSharp, a the search for the problem could be started. While the reason of the problem could be various, the best way I think is to add log-print in the code of llama.cpp and use it in LLamaSharp after compilation. Thus, when running LLamaSharp, you could see what happened in the native library.
After finding out the reason, a painful but happy process comes. When working on the BUG fix, there's only one rule to follow, that is keeping the examples working well. If the modification fixed the BUG but impact on other functions, it would not be a good fix.
During the BUG fix process, please don't hesitate to discuss together when you stuck on something.
"},{"location":"ContributingGuide/#add-integrations","title":"Add integrations","text":"All kinds of integration are welcomed here! Currently the following integrations are under work or on our schedule:
Besides, for some other integrations, like ASP.NET core, SQL, Blazor and so on, we'll appreciate it if you could help with that. If the time is limited for you, providing an example for it also means a lot!
There're mainly two ways to add an example:
LLama.Examples of the repository.LLamaSharp uses mkdocs to build the documantation, please follow the tutorial of mkdocs to add or modify documents in LLamaSharp.
"},{"location":"GetStarted/","title":"Get Started","text":""},{"location":"GetStarted/#install-packages","title":"Install packages","text":"Firstly, search LLamaSharp in nuget package manager and install it.
PM> Install-Package LLamaSharp\nThen, search and install one of the following backends:
LLamaSharp.Backend.Cpu\nLLamaSharp.Backend.Cuda11\nLLamaSharp.Backend.Cuda12\nHere's the mapping of them and corresponding model samples provided by LLamaSharp. If you're not sure which model is available for a version, please try our sample model.
One of the following models could be okay:
Note that because llama.cpp is under fast development now and often introduce break changes, some model weights on huggingface which works under a version may be invalid with another version. If it's your first time to configure LLamaSharp, we'd like to suggest for using verified model weights in the table above.
Please create a console program with dotnet runtime >= netstandard 2.0 (>= net6.0 is more recommended). Then, paste the following code to program.cs;
using LLama.Common;\nusing LLama;\n\nstring modelPath = \"<Your model path>\" // change it to your own model path\nvar prompt = \"Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.\\r\\n\\r\\nUser: Hello, Bob.\\r\\nBob: Hello. How may I help you today?\\r\\nUser: Please tell me the largest city in Europe.\\r\\nBob: Sure. The largest city in Europe is Moscow, the capital of Russia.\\r\\nUser:\"; // use the \"chat-with-bob\" prompt here.\n\n// Initialize a chat session\nvar ex = new InteractiveExecutor(new LLamaModel(new ModelParams(modelPath, contextSize: 1024, seed: 1337, gpuLayerCount: 5)));\nChatSession session = new ChatSession(ex);\n\n// show the prompt\nConsole.WriteLine();\nConsole.Write(prompt);\n\n// run the inference in a loop to chat with LLM\nwhile (true)\n{\n foreach (var text in session.Chat(prompt, new InferenceParams() { Temperature = 0.6f, AntiPrompts = new List<string> { \"User:\" } }))\n {\n Console.Write(text);\n }\n\n Console.ForegroundColor = ConsoleColor.Green;\n prompt = Console.ReadLine();\n Console.ForegroundColor = ConsoleColor.White;\n}\nAfter starting it, you'll see the following outputs.
Please input your model path: D:\\development\\llama\\weights\\wizard-vicuna-13B.ggmlv3.q4_1.bin\nllama.cpp: loading model from D:\\development\\llama\\weights\\wizard-vicuna-13B.ggmlv3.q4_1.bin\nllama_model_load_internal: format = ggjt v3 (latest)\nllama_model_load_internal: n_vocab = 32000\nllama_model_load_internal: n_ctx = 1024\nllama_model_load_internal: n_embd = 5120\nllama_model_load_internal: n_mult = 256\nllama_model_load_internal: n_head = 40\nllama_model_load_internal: n_layer = 40\nllama_model_load_internal: n_rot = 128\nllama_model_load_internal: ftype = 3 (mostly Q4_1)\nllama_model_load_internal: n_ff = 13824\nllama_model_load_internal: n_parts = 1\nllama_model_load_internal: model size = 13B\nllama_model_load_internal: ggml ctx size = 7759.48 MB\nllama_model_load_internal: mem required = 9807.48 MB (+ 1608.00 MB per state)\n....................................................................................................\nllama_init_from_file: kv self size = 800.00 MB\n\nTranscript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.\n\nUser: Hello, Bob.\nBob: Hello. How may I help you today?\nUser: Please tell me the largest city in Europe.\nBob: Sure. The largest city in Europe is Moscow, the capital of Russia.\nUser:\nNow, enjoy chatting with LLM!
"},{"location":"Tricks/","title":"Tricks for FAQ","text":"Sometimes, your application with LLM and LLamaSharp may have strange behaviors. Before opening an issue to report the BUG, the following tricks may worth a try.
"},{"location":"Tricks/#carefully-set-the-anti-prompts","title":"Carefully set the anti-prompts","text":"Anti-prompt can also be called as \"Stop-keyword\", which decides when to stop the response generation. Under interactive mode, the maximum tokens count is always not set, which makes the LLM generates responses infinitively. Therefore, setting anti-prompt correctly helps a lot to avoid the strange behaviors. For example, the prompt file chat-with-bob.txt has the following content:
Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.\n\nUser: Hello, Bob.\nBob: Hello. How may I help you today?\nUser: Please tell me the largest city in Europe.\nBob: Sure. The largest city in Europe is Moscow, the capital of Russia.\nUser:\nTherefore, the anti-prompt should be set as \"User:\". If the last line of the prompt is removed, LLM will automatically generate a question (user) and a response (bob) for one time when running the chat session. Therefore, the antiprompt is suggested to be appended to the prompt when starting a chat session.
What if an extra line is appended? The string \"User:\" in the prompt will be followed with a char \"\\n\". Thus when running the model, the automatic generation of a pair of question and response may appear because the anti-prompt is \"User:\" but the last token is \"User:\\n\". As for whether it will appear, it's an undefined behavior, which depends on the implementation inside the LLamaExecutor. Anyway, since it may leads to unexpected behaviors, it's recommended to trim your prompt or carefully keep consistent with your anti-prompt.
Sometimes we want to input a long prompt to execute a task. However, the context size may limit the inference of LLama model. Please ensure the inequality below holds.
$$ len(prompt) + len(response) < len(context) $$
In this inequality, len(response) refers to the expected tokens for LLM to generate.
Some prompt works well under interactive mode, such as chat-with-bob, some others may work well with instruct mode, such as alpaca. Besides, if your input is quite simple and one-time job, such as \"Q: what is the satellite of the earth? A: \", stateless mode will be a good choice.
If your chat bot has bad performance, trying different executor will possibly make it work well.
"},{"location":"Tricks/#choose-models-weight-depending-on-you-task","title":"Choose models weight depending on you task","text":"The differences between modes may lead to much different behaviors under the same task. For example, if you're building a chat bot with non-English, a fine-tuned model specially for the language you want to use will have huge effect on the performance.
"},{"location":"Tricks/#set-the-layer-count-you-want-to-offload-to-gpu","title":"Set the layer count you want to offload to GPU","text":"Currently, the GpuLayerCount param, which decides the number of layer loaded into GPU, is set to 20 by default. However, if you have some efficient GPUs, setting it as a larger number will attain faster inference.
ChatSession is a higher-level absatrction than the executors. In the context of a chat application like ChatGPT, a \"chat session\" refers to an interactive conversation or exchange of messages between the user and the chatbot. It represents a continuous flow of communication where the user enters input or asks questions, and the chatbot responds accordingly. A chat session typically starts when the user initiates a conversation with the chatbot and continues until the interaction comes to a natural end or is explicitly terminated by either the user or the system. During a chat session, the chatbot maintains the context of the conversation, remembers previous messages, and generates appropriate responses based on the user's inputs and the ongoing dialogue.
Currently, the only parameter that is accepted is an ILLamaExecutor, because this is the only parameter that we're sure to exist in all the future versions. Since it's the high-level absatrction, we're conservative to the API designs. In the future, there may be more kinds of constructors added.
InteractiveExecutor ex = new(new LLamaModel(new ModelParams(modelPath)));\nChatSession session = new ChatSession(ex);\nThere'll be two kinds of input accepted by the Chat API, which are ChatHistory and String. The API with string is quite similar to that of the executors. Meanwhile, the API with ChatHistory is aimed to provide more flexible usages. For example, you have had a chat with the bot in session A before you open the session B. Now session B has no memory for what you said before. Therefore, you can feed the history of A to B.
string prompt = \"What is C#?\";\n\nforeach (var text in session.Chat(prompt, new InferenceParams() { Temperature = 0.6f, AntiPrompts = new List<string> { \"User:\" } })) // the inference params should be changed depending on your statement\n{\n Console.Write(text);\n}\nCurrently History is a property of ChatSession.
foreach(var rec in session.History.Messages)\n{\n Console.WriteLine($\"{rec.AuthorRole}: {rec.Content}\");\n}\nGenerally, the chat session could be switched, which requires the ability of loading and saving session.
When building a chat bot app, it's NOT encouraged to initialize many chat sessions and keep them in memory to wait for being switched, because the memory comsumption of both CPU and GPU is expensive. It's recommended to save the current session before switching to a new session, and load the file when switching back to the session.
The API is also quite simple, the files will be saved into a directory you specified. If the path does not exist, a new directory will be created.
string savePath = \"<save dir>\";\nsession.SaveSession(savePath);\n\nsession.LoadSession(savePath);\nThere's three important elements in ChatSession, which are input, output and history. Besides, there're some conversions between them. Since the process of them under different conditions varies, LLamaSharp hands over this part of the power to the users.
Currently, there're three kinds of process that could be customized, as introduced below.
"},{"location":"ChatSession/transforms/#input-transform","title":"Input transform","text":"In general, the input of the chat API is a text (without stream), therefore ChatSession processes it in a pipeline. If you want to use your customized transform, you need to define a transform that implements ITextTransform and add it to the pipeline of ChatSession.
public interface ITextTransform\n{\n string Transform(string text);\n}\npublic class MyInputTransform1 : ITextTransform\n{\n public string Transform(string text)\n {\n return $\"Question: {text}\\n\";\n }\n}\n\npublic class MyInputTransform2 : ITextTransform\n{\n public string Transform(string text)\n {\n return text + \"Answer: \";\n }\n}\n\nsession.AddInputTransform(new MyInputTransform1()).AddInputTransform(new MyInputTransform2());\nDifferent from the input, the output of chat API is a text stream. Therefore you need to process it word by word, instead of getting the full text at once.
The interface of it has an IEnumerable<string> as input, which is actually a yield sequence.
public interface ITextStreamTransform\n{\n IEnumerable<string> Transform(IEnumerable<string> tokens);\n IAsyncEnumerable<string> TransformAsync(IAsyncEnumerable<string> tokens);\n}\nWhen implementing it, you could throw a not-implemented exception in one of them if you only need to use the chat API in synchronously or asynchronously.
Different from the input transform pipeline, the output transform only supports one transform.
session.WithOutputTransform(new MyOutputTransform());\nHere's an example of how to implement the interface. In this example, the transform detects wether there's some keywords in the response and removes them.
/// <summary>\n/// A text output transform that removes the keywords from the response.\n/// </summary>\npublic class KeywordTextOutputStreamTransform : ITextStreamTransform\n{\n HashSet<string> _keywords;\n int _maxKeywordLength;\n bool _removeAllMatchedTokens;\n\n /// <summary>\n /// \n /// </summary>\n /// <param name=\"keywords\">Keywords that you want to remove from the response.</param>\n /// <param name=\"redundancyLength\">The extra length when searching for the keyword. For example, if your only keyword is \"highlight\", \n /// maybe the token you get is \"\\r\\nhighligt\". In this condition, if redundancyLength=0, the token cannot be successfully matched because the length of \"\\r\\nhighligt\" (10)\n /// has already exceeded the maximum length of the keywords (8). On the contrary, setting redundancyLengyh >= 2 leads to successful match.\n /// The larger the redundancyLength is, the lower the processing speed. But as an experience, it won't introduce too much performance impact when redundancyLength <= 5 </param>\n /// <param name=\"removeAllMatchedTokens\">If set to true, when getting a matched keyword, all the related tokens will be removed. Otherwise only the part of keyword will be removed.</param>\n public KeywordTextOutputStreamTransform(IEnumerable<string> keywords, int redundancyLength = 3, bool removeAllMatchedTokens = false)\n {\n _keywords = new(keywords);\n _maxKeywordLength = keywords.Select(x => x.Length).Max() + redundancyLength;\n _removeAllMatchedTokens = removeAllMatchedTokens;\n }\n /// <inheritdoc />\n public IEnumerable<string> Transform(IEnumerable<string> tokens)\n {\n var window = new Queue<string>();\n\n foreach (var s in tokens)\n {\n window.Enqueue(s);\n var current = string.Join(\"\", window);\n if (_keywords.Any(x => current.Contains(x)))\n {\n var matchedKeyword = _keywords.First(x => current.Contains(x));\n int total = window.Count;\n for (int i = 0; i < total; i++)\n {\n window.Dequeue();\n }\n if (!_removeAllMatchedTokens)\n {\n yield return current.Replace(matchedKeyword, \"\");\n }\n }\n if (current.Length >= _maxKeywordLength)\n {\n if (_keywords.Any(x => current.Contains(x)))\n {\n var matchedKeyword = _keywords.First(x => current.Contains(x));\n int total = window.Count;\n for (int i = 0; i < total; i++)\n {\n window.Dequeue();\n }\n if (!_removeAllMatchedTokens)\n {\n yield return current.Replace(matchedKeyword, \"\");\n }\n }\n else\n {\n int total = window.Count;\n for (int i = 0; i < total; i++)\n {\n yield return window.Dequeue();\n }\n }\n }\n }\n int totalCount = window.Count;\n for (int i = 0; i < totalCount; i++)\n {\n yield return window.Dequeue();\n }\n }\n /// <inheritdoc />\n public async IAsyncEnumerable<string> TransformAsync(IAsyncEnumerable<string> tokens)\n {\n throw new NotImplementedException(); // This is implemented in `LLamaTransforms` but we ignore it here.\n }\n}\nThe chat history could be converted to or from a text, which is exactly what the interface of it.
public interface IHistoryTransform\n{\n string HistoryToText(ChatHistory history);\n ChatHistory TextToHistory(AuthorRole role, string text);\n}\nSimilar to the output transform, the history transform is added in the following way:
session.WithHistoryTransform(new MyHistoryTransform());\nThe implementation is quite flexible, depending on what you want the history message to be like. Here's an example, which is the default history transform in LLamaSharp.
/// <summary>\n/// The default history transform.\n/// Uses plain text with the following format:\n/// [Author]: [Message]\n/// </summary>\npublic class DefaultHistoryTransform : IHistoryTransform\n{\n private readonly string defaultUserName = \"User\";\n private readonly string defaultAssistantName = \"Assistant\";\n private readonly string defaultSystemName = \"System\";\n private readonly string defaultUnknownName = \"??\";\n\n string _userName;\n string _assistantName;\n string _systemName;\n string _unknownName;\n bool _isInstructMode;\n public DefaultHistoryTransform(string? userName = null, string? assistantName = null, \n string? systemName = null, string? unknownName = null, bool isInstructMode = false)\n {\n _userName = userName ?? defaultUserName;\n _assistantName = assistantName ?? defaultAssistantName;\n _systemName = systemName ?? defaultSystemName;\n _unknownName = unknownName ?? defaultUnknownName;\n _isInstructMode = isInstructMode;\n }\n\n public virtual string HistoryToText(ChatHistory history)\n {\n StringBuilder sb = new();\n foreach (var message in history.Messages)\n {\n if (message.AuthorRole == AuthorRole.User)\n {\n sb.AppendLine($\"{_userName}: {message.Content}\");\n }\n else if (message.AuthorRole == AuthorRole.System)\n {\n sb.AppendLine($\"{_systemName}: {message.Content}\");\n }\n else if (message.AuthorRole == AuthorRole.Unknown)\n {\n sb.AppendLine($\"{_unknownName}: {message.Content}\");\n }\n else if (message.AuthorRole == AuthorRole.Assistant)\n {\n sb.AppendLine($\"{_assistantName}: {message.Content}\");\n }\n }\n return sb.ToString();\n }\n\n public virtual ChatHistory TextToHistory(AuthorRole role, string text)\n {\n ChatHistory history = new ChatHistory();\n history.AddMessage(role, TrimNamesFromText(text, role));\n return history;\n }\n\n public virtual string TrimNamesFromText(string text, AuthorRole role)\n {\n if (role == AuthorRole.User && text.StartsWith($\"{_userName}:\"))\n {\n text = text.Substring($\"{_userName}:\".Length).TrimStart();\n }\n else if (role == AuthorRole.Assistant && text.EndsWith($\"{_assistantName}:\"))\n {\n text = text.Substring(0, text.Length - $\"{_assistantName}:\".Length).TrimEnd();\n }\n if (_isInstructMode && role == AuthorRole.Assistant && text.EndsWith(\"\\n> \"))\n {\n text = text.Substring(0, text.Length - \"\\n> \".Length).TrimEnd();\n }\n return text;\n }\n}\nThe document is under work, please have a wait. Thank you for your support! :)
"},{"location":"LLamaExecutors/differences/","title":"Differences of Executors","text":""},{"location":"LLamaExecutors/differences/#differences-between-the-executors","title":"Differences between the executors","text":"There're currently three kinds of executors provided, which are InteractiveExecutor, InstructExecutor and StatelessExecutor.
In a word, InteractiveExecutor is suitable for getting answer of your questions from LLM continuously. InstructExecutor let LLM execute your instructions, such as \"continue writing\". StatelessExecutor is best for one-time job because the previous inference has no impact on the current inference.
Both of them are taking \"completing the prompt\" as the goal to generate the response. For example, if you input Long long ago, there was a fox who wanted to make friend with humen. One day, then the LLM will continue to write the story.
Under interactive mode, you serve a role of user and the LLM serves the role of assistant. Then it will help you with your question or request.
Under instruct mode, you give LLM some instructions and it follows.
Though the behaviors of them sounds similar, it could introduce many differences depending on your prompt. For example, \"chat-with-bob\" has good performance under interactive mode and alpaca does well with instruct mode.
// chat-with-bob\n\nTranscript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.\n\nUser: Hello, Bob.\nBob: Hello. How may I help you today?\nUser: Please tell me the largest city in Europe.\nBob: Sure. The largest city in Europe is Moscow, the capital of Russia.\nUser:\n// alpaca\n\nBelow is an instruction that describes a task. Write a response that appropriately completes the request.\nTherefore, please modify the prompt correspondingly when switching from one mode to the other.
"},{"location":"LLamaExecutors/differences/#stateful-mode-and-stateless-mode","title":"Stateful mode and Stateless mode.","text":"Despite the differences between interactive mode and instruct mode, both of them are stateful mode. That is, your previous question/instruction will impact on the current response from LLM. On the contrary, the steteless executor does not have such a \"memory\". No matter how many times you talk to it, it will only concentrate on what you say in this time.
Since the stateless executor has no memory of conversations before, you need to input your question with the whole prompt into it to get the better answer.
For example, if you feed Q: Who is Trump? A: to the steteless executor, it may give the following answer with the antiprompt Q:.
Donald J. Trump, born June 14, 1946, is an American businessman, television personality, politician and the 45th President of the United States (2017-2021). # Anexo:Torneo de Hamburgo 2022 (individual masculino)\n\n## Presentaci\u00f3n previa\n\n* Defensor del t\u00edtulo: Daniil Medv\u00e9dev\nIt seems that things went well at first. However, after answering the question itself, LLM began to talk about some other things until the answer reached the token count limit. The reason of this strange behavior is the anti-prompt cannot be match. With the input, LLM cannot decide whether to append a string \"A: \" at the end of the response.
As an improvement, let's take the following text as the input:
Q: What is the capital of the USA? A: Washingtong. Q: What is the sum of 1 and 2? A: 3. Q: Who is Trump? A: \nThen, I got the following answer with the anti-prompt Q:.
45th president of the United States.\nAt this time, by repeating the same mode of Q: xxx? A: xxx., LLM outputs the anti-prompt we want to help to decide where to dtop the generation.
Different from LLamaModel, when using an exeuctor, InferenceParams is passed to the Infer method instead of constructor. This is because executors only define the ways to run the model, therefore in each run, you can change the settings for this time inference.
Namespace: LLama.Common
public class InferenceParams\nInheritance Object \u2192 InferenceParams
"},{"location":"LLamaExecutors/parameters/#properties","title":"Properties","text":""},{"location":"LLamaExecutors/parameters/#tokenskeep","title":"TokensKeep","text":"number of tokens to keep from initial prompt
public int TokensKeep { get; set; }\nInt32
"},{"location":"LLamaExecutors/parameters/#maxtokens","title":"MaxTokens","text":"how many new tokens to predict (n_predict), set to -1 to inifinitely generate response until it complete.
public int MaxTokens { get; set; }\nInt32
"},{"location":"LLamaExecutors/parameters/#logitbias","title":"LogitBias","text":"logit bias for specific tokens
public Dictionary<int, float> LogitBias { get; set; }\nDictionary<Int32, Single>
"},{"location":"LLamaExecutors/parameters/#antiprompts","title":"AntiPrompts","text":"Sequences where the model will stop generating further tokens.
public IEnumerable<string> AntiPrompts { get; set; }\nIEnumerable<String>
"},{"location":"LLamaExecutors/parameters/#pathsession","title":"PathSession","text":"path to file for saving/loading model eval state
public string PathSession { get; set; }\nString
"},{"location":"LLamaExecutors/parameters/#inputsuffix","title":"InputSuffix","text":"string to suffix user inputs with
public string InputSuffix { get; set; }\nString
"},{"location":"LLamaExecutors/parameters/#inputprefix","title":"InputPrefix","text":"string to prefix user inputs with
public string InputPrefix { get; set; }\nString
"},{"location":"LLamaExecutors/parameters/#topk","title":"TopK","text":"0 or lower to use vocab size
public int TopK { get; set; }\nInt32
"},{"location":"LLamaExecutors/parameters/#topp","title":"TopP","text":"1.0 = disabled
public float TopP { get; set; }\nSingle
"},{"location":"LLamaExecutors/parameters/#tfsz","title":"TfsZ","text":"1.0 = disabled
public float TfsZ { get; set; }\nSingle
"},{"location":"LLamaExecutors/parameters/#typicalp","title":"TypicalP","text":"1.0 = disabled
public float TypicalP { get; set; }\nSingle
"},{"location":"LLamaExecutors/parameters/#temperature","title":"Temperature","text":"1.0 = disabled
public float Temperature { get; set; }\nSingle
"},{"location":"LLamaExecutors/parameters/#repeatpenalty","title":"RepeatPenalty","text":"1.0 = disabled
public float RepeatPenalty { get; set; }\nSingle
"},{"location":"LLamaExecutors/parameters/#repeatlasttokenscount","title":"RepeatLastTokensCount","text":"last n tokens to penalize (0 = disable penalty, -1 = context size) (repeat_last_n)
public int RepeatLastTokensCount { get; set; }\nInt32
"},{"location":"LLamaExecutors/parameters/#frequencypenalty","title":"FrequencyPenalty","text":"frequency penalty coefficient 0.0 = disabled
public float FrequencyPenalty { get; set; }\nSingle
"},{"location":"LLamaExecutors/parameters/#presencepenalty","title":"PresencePenalty","text":"presence penalty coefficient 0.0 = disabled
public float PresencePenalty { get; set; }\nSingle
"},{"location":"LLamaExecutors/parameters/#mirostat","title":"Mirostat","text":"Mirostat uses tokens instead of words. algorithm described in the paper https://arxiv.org/abs/2007.14966. 0 = disabled, 1 = mirostat, 2 = mirostat 2.0
public MiroStateType Mirostat { get; set; }\nMiroStateType
"},{"location":"LLamaExecutors/parameters/#mirostattau","title":"MirostatTau","text":"target entropy
public float MirostatTau { get; set; }\nSingle
"},{"location":"LLamaExecutors/parameters/#mirostateta","title":"MirostatEta","text":"learning rate
public float MirostatEta { get; set; }\nSingle
"},{"location":"LLamaExecutors/parameters/#penalizenl","title":"PenalizeNL","text":"consider newlines as a repeatable token (penalize_nl)
public bool PenalizeNL { get; set; }\nBoolean
"},{"location":"LLamaExecutors/save-load-state/","title":"Save/Load State of Executor","text":"Similar to LLamaModel, an executor also has its state, which can be saved and loaded. Note that in most of cases, the state of executor and the state of the model should be loaded and saved at the same time.
To decouple the model and executor, we provide APIs to save/load state for model and executor respectively. However, during the inference, the processed information will leave footprint in LLamaModel's native context. Therefore, if you just load a state from another executor but keep the model unmodified, some strange things may happen. So will loading model state only.
Is there a condition that requires to load one of them only? The answer is YES. For example, after resetting the model state, if you don't want the inference starting from the new position, leaving the executor unmodified is okay. But, anyway, this flexible usage may cause some unexpected behaviors, therefore please ensure you know what you're doing before using it in this way.
In the future version, we'll open the access for some variables inside the executor to support more flexible usages.
The APIs to load/save state of the executors is similar to that of LLamaModel. However, note that StatelessExecutor doesn't have such APIs because it's stateless itself. Besides, the output of GetStateData is an object of type ExecutorBaseState.
LLamaModel model = new LLamaModel(new ModelParams(\"<modelPath>\"));\nInteractiveExecutor executor = new InteractiveExecutor(model);\n// do some things...\nexecutor.SaveState(\"executor.st\");\nvar stateData = model.GetStateData();\n\nInteractiveExecutor executor2 = new InteractiveExecutor(model);\nexecutor2.LoadState(stateData);\n// do some things...\n\nInteractiveExecutor executor3 = new InteractiveExecutor(model);\nexecutor3.LoadState(\"executor.st\");\n// do some things...\nAll the executors implements the interface ILLamaExecutor, which provides two APIs to execute text-to-text tasks.
public interface ILLamaExecutor\n{\n public LLamaModel Model { get; }\n\n IEnumerable<string> Infer(string text, InferenceParams? inferenceParams = null, CancellationToken token = default);\n\n IAsyncEnumerable<string> InferAsync(string text, InferenceParams? inferenceParams = null, CancellationToken token = default);\n}\nJust pass the text to the executor with the inference parameters. For the inference parameters, please refer to executor inference parameters doc.
The output of both two APIs are yield enumerable. Therefore, when receiving the output, you can directly use foreach to take actions on each word you get by order, instead of waiting for the whole process completed.
Getting the embeddings of a text in LLM is sometimes useful, for example, to train other MLP models.
To get the embeddings, please initialize a LLamaEmbedder and then call GetEmbeddings.
var embedder = new LLamaEmbedder(new ModelParams(\"<modelPath>\"));\nstring text = \"hello, LLM.\";\nfloat[] embeddings = embedder.GetEmbeddings(text);\nThe output is a float array. Note that the length of the array is related with the model you load. If you just want to get a smaller size embedding, please consider changing a model.
"},{"location":"LLamaModel/parameters/","title":"LLamaModel Parameters","text":"When initializing a LLamaModel object, there're three parameters, ModelParams Params, string encoding = \"UTF-8\", ILLamaLogger? logger = null.
The usage of logger will be further introduced in logger doc. The encoding is the encoding you want to use when dealing with text via this model.
The most improtant of all, is the ModelParams, which is defined as below. We'll explain the parameters step by step in this document.
public class ModelParams\n{\n public int ContextSize { get; set; } = 512;\n public int GpuLayerCount { get; set; } = 20;\n public int Seed { get; set; } = 1686349486;\n public bool UseFp16Memory { get; set; } = true;\n public bool UseMemorymap { get; set; } = true;\n public bool UseMemoryLock { get; set; } = false;\n public bool Perplexity { get; set; } = false;\n public string ModelPath { get; set; }\n public string LoraAdapter { get; set; } = string.Empty;\n public string LoraBase { get; set; } = string.Empty;\n public int Threads { get; set; } = Math.Max(Environment.ProcessorCount / 2, 1);\n public int BatchSize { get; set; } = 512;\n public bool ConvertEosToNewLine { get; set; } = false;\n}\nNamespace: LLama.Common
public class ModelParams\nInheritance Object \u2192 ModelParams
"},{"location":"LLamaModel/parameters/#properties","title":"Properties","text":""},{"location":"LLamaModel/parameters/#contextsize","title":"ContextSize","text":"Model context size (n_ctx)
public int ContextSize { get; set; }\nInt32
"},{"location":"LLamaModel/parameters/#gpulayercount","title":"GpuLayerCount","text":"Number of layers to run in VRAM / GPU memory (n_gpu_layers)
public int GpuLayerCount { get; set; }\nInt32
"},{"location":"LLamaModel/parameters/#seed","title":"Seed","text":"Seed for the random number generator (seed)
public int Seed { get; set; }\nInt32
"},{"location":"LLamaModel/parameters/#usefp16memory","title":"UseFp16Memory","text":"Use f16 instead of f32 for memory kv (memory_f16)
public bool UseFp16Memory { get; set; }\nBoolean
"},{"location":"LLamaModel/parameters/#usememorymap","title":"UseMemorymap","text":"Use mmap for faster loads (use_mmap)
public bool UseMemorymap { get; set; }\nBoolean
"},{"location":"LLamaModel/parameters/#usememorylock","title":"UseMemoryLock","text":"Use mlock to keep model in memory (use_mlock)
public bool UseMemoryLock { get; set; }\nBoolean
"},{"location":"LLamaModel/parameters/#perplexity","title":"Perplexity","text":"Compute perplexity over the prompt (perplexity)
public bool Perplexity { get; set; }\nBoolean
"},{"location":"LLamaModel/parameters/#modelpath","title":"ModelPath","text":"Model path (model)
public string ModelPath { get; set; }\nString
"},{"location":"LLamaModel/parameters/#loraadapter","title":"LoraAdapter","text":"lora adapter path (lora_adapter)
public string LoraAdapter { get; set; }\nString
"},{"location":"LLamaModel/parameters/#lorabase","title":"LoraBase","text":"base model path for the lora adapter (lora_base)
public string LoraBase { get; set; }\nString
"},{"location":"LLamaModel/parameters/#threads","title":"Threads","text":"Number of threads (-1 = autodetect) (n_threads)
public int Threads { get; set; }\nInt32
"},{"location":"LLamaModel/parameters/#batchsize","title":"BatchSize","text":"batch size for prompt processing (must be >=32 to use BLAS) (n_batch)
public int BatchSize { get; set; }\nInt32
"},{"location":"LLamaModel/parameters/#converteostonewline","title":"ConvertEosToNewLine","text":"Whether to convert eos to newline during the inference.
public bool ConvertEosToNewLine { get; set; }\nBoolean
"},{"location":"LLamaModel/parameters/#embeddingmode","title":"EmbeddingMode","text":"Whether to use embedding mode. (embedding) Note that if this is set to true, The LLamaModel won't produce text response anymore.
public bool EmbeddingMode { get; set; }\nBoolean
"},{"location":"LLamaModel/quantization/","title":"Quantization","text":"Quantization is significant to accelerate the model inference. Since there's little accuracy (performance) reduction when quantizing the model, get it easy to quantize it!
To quantize the model, please call Quantize from LLamaQuantizer, which is a static method.
string srcPath = \"<model.bin>\";\nstring dstPath = \"<model_q4_0.bin>\";\nLLamaQuantizer.Quantize(srcPath, dstPath, \"q4_0\");\n// The following overload is also okay.\n// LLamaQuantizer.Quantize(srcPath, dstPath, LLamaFtype.LLAMA_FTYPE_MOSTLY_Q4_0);\nAfter calling it, a quantized model file will be saved.
There're currently 5 types of quantization supported:
There're two ways to load state: loading from path and loading from bite array. Therefore, correspondingly, state data can be extracted as byte array or saved to a file.
LLamaModel model = new LLamaModel(new ModelParams(\"<modelPath>\"));\n// do some things...\nmodel.SaveState(\"model.st\");\nvar stateData = model.GetStateData();\nmodel.Dispose();\n\nLLamaModel model2 = new LLamaModel(new ModelParams(\"<modelPath>\"));\nmodel2.LoadState(stateData);\n// do some things...\n\nLLamaModel model3 = new LLamaModel(new ModelParams(\"<modelPath>\"));\nmodel3.LoadState(\"model.st\");\n// do some things...\nA pair of APIs to make conversion between text and tokens.
"},{"location":"LLamaModel/tokenization/#tokenization","title":"Tokenization","text":"The basic usage is to call Tokenize after initializing the model.
LLamaModel model = new LLamaModel(new ModelParams(\"<modelPath>\"));\nstring text = \"hello\";\nint[] tokens = model.Tokenize(text).ToArray();\nDepending on different model (or vocab), the output will be various.
"},{"location":"LLamaModel/tokenization/#detokenization","title":"Detokenization","text":"Similar to tokenization, just pass an IEnumerable<int> to Detokenize method.
LLamaModel model = new LLamaModel(new ModelParams(\"<modelPath>\"));\nint[] tokens = new int[] {125, 2568, 13245};\nstring text = model.Detokenize(tokens);\nLLamaSharp supports customized logger because it could be used in many kinds of applications, like Winform/WPF, WebAPI and Blazor, so that the preference of logger varies.
"},{"location":"More/log/#define-customized-logger","title":"Define customized logger","text":"What you need to do is to implement the ILogger interface.
public interface ILLamaLogger\n{\n public enum LogLevel\n {\n Info,\n Debug,\n Warning,\n Error\n }\n void Log(string source, string message, LogLevel level);\n}\nThe source specifies where the log message is from, which could be a function, a class, etc..
The message is the log message itself.
The level is the level of the information in the log. As shown above, there're four levels, which are info, debug, warning and error respectively.
The following is a simple example of theb logger implementation:
public sealed class LLamaDefaultLogger : ILLamaLogger\n{\n private static readonly Lazy<LLamaDefaultLogger> _instance = new Lazy<LLamaDefaultLogger>(() => new LLamaDefaultLogger());\n\n private bool _toConsole = true;\n private bool _toFile = false;\n\n private FileStream? _fileStream = null;\n private StreamWriter _fileWriter = null;\n\n public static LLamaDefaultLogger Default => _instance.Value;\n\n private LLamaDefaultLogger()\n {\n\n }\n\n public LLamaDefaultLogger EnableConsole()\n {\n _toConsole = true;\n return this;\n }\n\n public LLamaDefaultLogger DisableConsole()\n {\n _toConsole = false;\n return this;\n }\n\n public LLamaDefaultLogger EnableFile(string filename, FileMode mode = FileMode.Append)\n {\n _fileStream = new FileStream(filename, mode, FileAccess.Write);\n _fileWriter = new StreamWriter(_fileStream);\n _toFile = true;\n return this;\n }\n\n public LLamaDefaultLogger DisableFile(string filename)\n {\n if (_fileWriter is not null)\n {\n _fileWriter.Close();\n _fileWriter = null;\n }\n if (_fileStream is not null)\n {\n _fileStream.Close();\n _fileStream = null;\n }\n _toFile = false;\n return this;\n }\n\n public void Log(string source, string message, LogLevel level)\n {\n if (level == LogLevel.Info)\n {\n Info(message);\n }\n else if (level == LogLevel.Debug)\n {\n\n }\n else if (level == LogLevel.Warning)\n {\n Warn(message);\n }\n else if (level == LogLevel.Error)\n {\n Error(message);\n }\n }\n\n public void Info(string message)\n {\n message = MessageFormat(\"info\", message);\n if (_toConsole)\n {\n Console.ForegroundColor = ConsoleColor.White;\n Console.WriteLine(message);\n Console.ResetColor();\n }\n if (_toFile)\n {\n Debug.Assert(_fileStream is not null);\n Debug.Assert(_fileWriter is not null);\n _fileWriter.WriteLine(message);\n }\n }\n\n public void Warn(string message)\n {\n message = MessageFormat(\"warn\", message);\n if (_toConsole)\n {\n Console.ForegroundColor = ConsoleColor.Yellow;\n Console.WriteLine(message);\n Console.ResetColor();\n }\n if (_toFile)\n {\n Debug.Assert(_fileStream is not null);\n Debug.Assert(_fileWriter is not null);\n _fileWriter.WriteLine(message);\n }\n }\n\n public void Error(string message)\n {\n message = MessageFormat(\"error\", message);\n if (_toConsole)\n {\n Console.ForegroundColor = ConsoleColor.Red;\n Console.WriteLine(message);\n Console.ResetColor();\n }\n if (_toFile)\n {\n Debug.Assert(_fileStream is not null);\n Debug.Assert(_fileWriter is not null);\n _fileWriter.WriteLine(message);\n }\n }\n\n private string MessageFormat(string level, string message)\n {\n DateTime now = DateTime.Now;\n string formattedDate = now.ToString(\"yyyy.MM.dd HH:mm:ss\");\n return $\"[{formattedDate}][{level}]: {message}\";\n }\n}\nIt's supported now but the document is under work. Please wait for some time. Thank you for your support! :)
"},{"location":"xmldocs/","title":"LLamaSharp","text":""},{"location":"xmldocs/#llama","title":"LLama","text":"ChatSession
InstructExecutor
InteractiveExecutor
LLamaEmbedder
LLamaModel
LLamaQuantizer
LLamaTransforms
ResettableLLamaModel
StatefulExecutorBase
StatelessExecutor
"},{"location":"xmldocs/#llamaabstractions","title":"LLama.Abstractions","text":"IHistoryTransform
ILLamaExecutor
ITextStreamTransform
ITextTransform
"},{"location":"xmldocs/#llamacommon","title":"LLama.Common","text":"AuthorRole
ChatHistory
FixedSizeQueue<T>
ILLamaLogger
InferenceParams
LLamaDefaultLogger
MiroStateType
ModelParams
"},{"location":"xmldocs/#llamaexceptions","title":"LLama.Exceptions","text":"RuntimeError
"},{"location":"xmldocs/#llamaextensions","title":"LLama.Extensions","text":"DictionaryExtension
"},{"location":"xmldocs/#llamanative","title":"LLama.Native","text":"LLamaContextParams
LLamaFtype
LLamaTokenData
LLamaTokenDataArray
LLamaTokenDataArrayNative
NativeApi
SafeLLamaContextHandle
SafeLLamaHandleBase
"},{"location":"xmldocs/#llamaoldversion","title":"LLama.OldVersion","text":"ChatCompletion
ChatCompletionChoice
ChatCompletionChunk
ChatCompletionChunkChoice
ChatCompletionChunkDelta
ChatCompletionMessage
ChatMessageRecord
ChatRole
ChatSession<T>
Completion
CompletionChoice
CompletionChunk
CompletionLogprobs
CompletionUsage
Embedding
EmbeddingData
EmbeddingUsage
IChatModel
LLamaEmbedder
LLamaModel
LLamaParams
"},{"location":"xmldocs/llama.abstractions.ihistorytransform/","title":"IHistoryTransform","text":"Namespace: LLama.Abstractions
Transform history to plain text and vice versa.
public interface IHistoryTransform\nConvert a ChatHistory instance to plain text.
string HistoryToText(ChatHistory history)\nhistory ChatHistory The ChatHistory instance
String
"},{"location":"xmldocs/llama.abstractions.ihistorytransform/#texttohistoryauthorrole-string","title":"TextToHistory(AuthorRole, String)","text":"Converts plain text to a ChatHistory instance.
ChatHistory TextToHistory(AuthorRole role, string text)\nrole AuthorRole The role for the author.
text String The chat history as plain text.
ChatHistory The updated history.
"},{"location":"xmldocs/llama.abstractions.illamaexecutor/","title":"ILLamaExecutor","text":"Namespace: LLama.Abstractions
A high level interface for LLama models.
public interface ILLamaExecutor\nThe loaded model for this executor.
public abstract LLamaModel Model { get; }\nLLamaModel
"},{"location":"xmldocs/llama.abstractions.illamaexecutor/#methods","title":"Methods","text":""},{"location":"xmldocs/llama.abstractions.illamaexecutor/#inferstring-inferenceparams-cancellationtoken","title":"Infer(String, InferenceParams, CancellationToken)","text":"Infers a response from the model.
IEnumerable<string> Infer(string text, InferenceParams inferenceParams, CancellationToken token)\ntext String Your prompt
inferenceParams InferenceParams Any additional parameters
token CancellationToken A cancellation token.
IEnumerable<String>
"},{"location":"xmldocs/llama.abstractions.illamaexecutor/#inferasyncstring-inferenceparams-cancellationtoken","title":"InferAsync(String, InferenceParams, CancellationToken)","text":"IAsyncEnumerable<string> InferAsync(string text, InferenceParams inferenceParams, CancellationToken token)\ntext String
inferenceParams InferenceParams
token CancellationToken
IAsyncEnumerable<String>
"},{"location":"xmldocs/llama.abstractions.itextstreamtransform/","title":"ITextStreamTransform","text":"Namespace: LLama.Abstractions
Takes a stream of tokens and transforms them.
public interface ITextStreamTransform\nTakes a stream of tokens and transforms them, returning a new stream of tokens.
IEnumerable<string> Transform(IEnumerable<string> tokens)\ntokens IEnumerable<String>
IEnumerable<String>
"},{"location":"xmldocs/llama.abstractions.itextstreamtransform/#transformasynciasyncenumerablestring","title":"TransformAsync(IAsyncEnumerable<String>)","text":"Takes a stream of tokens and transforms them, returning a new stream of tokens asynchronously.
IAsyncEnumerable<string> TransformAsync(IAsyncEnumerable<string> tokens)\ntokens IAsyncEnumerable<String>
IAsyncEnumerable<String>
"},{"location":"xmldocs/llama.abstractions.itexttransform/","title":"ITextTransform","text":"Namespace: LLama.Abstractions
An interface for text transformations. These can be used to compose a pipeline of text transformations, such as: - Tokenization - Lowercasing - Punctuation removal - Trimming - etc.
public interface ITextTransform\nTakes a string and transforms it.
string Transform(string text)\ntext String
String
"},{"location":"xmldocs/llama.chatsession/","title":"ChatSession","text":"Namespace: LLama
The main chat session class.
public class ChatSession\nInheritance Object \u2192 ChatSession
"},{"location":"xmldocs/llama.chatsession/#fields","title":"Fields","text":""},{"location":"xmldocs/llama.chatsession/#outputtransform","title":"OutputTransform","text":"The output transform used in this session.
public ITextStreamTransform OutputTransform;\nThe executor for this session.
public ILLamaExecutor Executor { get; }\nILLamaExecutor
"},{"location":"xmldocs/llama.chatsession/#history","title":"History","text":"The chat history for this session.
public ChatHistory History { get; }\nChatHistory
"},{"location":"xmldocs/llama.chatsession/#historytransform","title":"HistoryTransform","text":"The history transform used in this session.
public IHistoryTransform HistoryTransform { get; set; }\nIHistoryTransform
"},{"location":"xmldocs/llama.chatsession/#inputtransformpipeline","title":"InputTransformPipeline","text":"The input transform pipeline used in this session.
public List<ITextTransform> InputTransformPipeline { get; set; }\nList<ITextTransform>
"},{"location":"xmldocs/llama.chatsession/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.chatsession/#chatsessionillamaexecutor","title":"ChatSession(ILLamaExecutor)","text":"public ChatSession(ILLamaExecutor executor)\nexecutor ILLamaExecutor The executor for this session
Use a custom history transform.
public ChatSession WithHistoryTransform(IHistoryTransform transform)\ntransform IHistoryTransform
ChatSession
"},{"location":"xmldocs/llama.chatsession/#addinputtransformitexttransform","title":"AddInputTransform(ITextTransform)","text":"Add a text transform to the input transform pipeline.
public ChatSession AddInputTransform(ITextTransform transform)\ntransform ITextTransform
ChatSession
"},{"location":"xmldocs/llama.chatsession/#withoutputtransformitextstreamtransform","title":"WithOutputTransform(ITextStreamTransform)","text":"Use a custom output transform.
public ChatSession WithOutputTransform(ITextStreamTransform transform)\ntransform ITextStreamTransform
ChatSession
"},{"location":"xmldocs/llama.chatsession/#savesessionstring","title":"SaveSession(String)","text":"public void SaveSession(string path)\npath String The directory name to save the session. If the directory does not exist, a new directory will be created.
public void LoadSession(string path)\npath String The directory name to load the session.
Get the response from the LLama model with chat histories.
public IEnumerable<string> Chat(ChatHistory history, InferenceParams inferenceParams, CancellationToken cancellationToken)\nhistory ChatHistory
inferenceParams InferenceParams
cancellationToken CancellationToken
IEnumerable<String>
"},{"location":"xmldocs/llama.chatsession/#chatstring-inferenceparams-cancellationtoken","title":"Chat(String, InferenceParams, CancellationToken)","text":"Get the response from the LLama model. Note that prompt could not only be the preset words, but also the question you want to ask.
public IEnumerable<string> Chat(string prompt, InferenceParams inferenceParams, CancellationToken cancellationToken)\nprompt String
inferenceParams InferenceParams
cancellationToken CancellationToken
IEnumerable<String>
"},{"location":"xmldocs/llama.chatsession/#chatasyncchathistory-inferenceparams-cancellationtoken","title":"ChatAsync(ChatHistory, InferenceParams, CancellationToken)","text":"Get the response from the LLama model with chat histories.
public IAsyncEnumerable<string> ChatAsync(ChatHistory history, InferenceParams inferenceParams, CancellationToken cancellationToken)\nhistory ChatHistory
inferenceParams InferenceParams
cancellationToken CancellationToken
IAsyncEnumerable<String>
"},{"location":"xmldocs/llama.chatsession/#chatasyncstring-inferenceparams-cancellationtoken","title":"ChatAsync(String, InferenceParams, CancellationToken)","text":"Get the response from the LLama model with chat histories asynchronously.
public IAsyncEnumerable<string> ChatAsync(string prompt, InferenceParams inferenceParams, CancellationToken cancellationToken)\nprompt String
inferenceParams InferenceParams
cancellationToken CancellationToken
IAsyncEnumerable<String>
"},{"location":"xmldocs/llama.common.authorrole/","title":"AuthorRole","text":"Namespace: LLama.Common
public enum AuthorRole\nInheritance Object \u2192 ValueType \u2192 Enum \u2192 AuthorRole Implements IComparable, IFormattable, IConvertible
"},{"location":"xmldocs/llama.common.authorrole/#fields","title":"Fields","text":"Name Value Description"},{"location":"xmldocs/llama.common.chathistory/","title":"ChatHistory","text":"Namespace: LLama.Common
The chat history class
public class ChatHistory\nInheritance Object \u2192 ChatHistory
"},{"location":"xmldocs/llama.common.chathistory/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.common.chathistory/#messages","title":"Messages","text":"List of messages in the chat
public List<Message> Messages { get; }\nList<Message>
"},{"location":"xmldocs/llama.common.chathistory/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.common.chathistory/#chathistory_1","title":"ChatHistory()","text":"Create a new instance of the chat content class
public ChatHistory()\nAdd a message to the chat history
public void AddMessage(AuthorRole authorRole, string content)\nauthorRole AuthorRole Role of the message author
content String Message content
Namespace: LLama.Common
A queue with fixed storage size. Currently it's only a naive implementation and needs to be further optimized in the future.
public class FixedSizeQueue<T> : , System.Collections.IEnumerable\nT
Inheritance Object \u2192 FixedSizeQueue<T> Implements IEnumerable<T>, IEnumerable
"},{"location":"xmldocs/llama.common.fixedsizequeue-1/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.common.fixedsizequeue-1/#count","title":"Count","text":"public int Count { get; }\nInt32
"},{"location":"xmldocs/llama.common.fixedsizequeue-1/#capacity","title":"Capacity","text":"public int Capacity { get; }\nInt32
"},{"location":"xmldocs/llama.common.fixedsizequeue-1/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.common.fixedsizequeue-1/#fixedsizequeueint32","title":"FixedSizeQueue(Int32)","text":"public FixedSizeQueue(int size)\nsize Int32
public FixedSizeQueue(int size, IEnumerable<T> data)\nsize Int32
data IEnumerable<T>
public FixedSizeQueue<T> FillWith(T value)\nvalue T
FixedSizeQueue<T>
"},{"location":"xmldocs/llama.common.fixedsizequeue-1/#enqueuet","title":"Enqueue(T)","text":"Enquene an element.
public void Enqueue(T item)\nitem T
public T[] ToArray()\nT[]
"},{"location":"xmldocs/llama.common.fixedsizequeue-1/#getenumerator","title":"GetEnumerator()","text":"public IEnumerator<T> GetEnumerator()\nIEnumerator<T>
"},{"location":"xmldocs/llama.common.illamalogger/","title":"ILLamaLogger","text":"Namespace: LLama.Common
public interface ILLamaLogger\nWrite the log in cosutomized way
void Log(string source, string message, LogLevel level)\nsource String The source of the log. It may be a method name or class name.
message String The message.
level LogLevel The log level.
Namespace: LLama.Common
public class InferenceParams\nInheritance Object \u2192 InferenceParams
"},{"location":"xmldocs/llama.common.inferenceparams/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.common.inferenceparams/#tokenskeep","title":"TokensKeep","text":"number of tokens to keep from initial prompt
public int TokensKeep { get; set; }\nInt32
"},{"location":"xmldocs/llama.common.inferenceparams/#maxtokens","title":"MaxTokens","text":"how many new tokens to predict (n_predict), set to -1 to inifinitely generate response until it complete.
public int MaxTokens { get; set; }\nInt32
"},{"location":"xmldocs/llama.common.inferenceparams/#logitbias","title":"LogitBias","text":"logit bias for specific tokens
public Dictionary<int, float> LogitBias { get; set; }\nDictionary<Int32, Single>
"},{"location":"xmldocs/llama.common.inferenceparams/#antiprompts","title":"AntiPrompts","text":"Sequences where the model will stop generating further tokens.
public IEnumerable<string> AntiPrompts { get; set; }\nIEnumerable<String>
"},{"location":"xmldocs/llama.common.inferenceparams/#pathsession","title":"PathSession","text":"path to file for saving/loading model eval state
public string PathSession { get; set; }\nString
"},{"location":"xmldocs/llama.common.inferenceparams/#inputsuffix","title":"InputSuffix","text":"string to suffix user inputs with
public string InputSuffix { get; set; }\nString
"},{"location":"xmldocs/llama.common.inferenceparams/#inputprefix","title":"InputPrefix","text":"string to prefix user inputs with
public string InputPrefix { get; set; }\nString
"},{"location":"xmldocs/llama.common.inferenceparams/#topk","title":"TopK","text":"0 or lower to use vocab size
public int TopK { get; set; }\nInt32
"},{"location":"xmldocs/llama.common.inferenceparams/#topp","title":"TopP","text":"1.0 = disabled
public float TopP { get; set; }\nSingle
"},{"location":"xmldocs/llama.common.inferenceparams/#tfsz","title":"TfsZ","text":"1.0 = disabled
public float TfsZ { get; set; }\nSingle
"},{"location":"xmldocs/llama.common.inferenceparams/#typicalp","title":"TypicalP","text":"1.0 = disabled
public float TypicalP { get; set; }\nSingle
"},{"location":"xmldocs/llama.common.inferenceparams/#temperature","title":"Temperature","text":"1.0 = disabled
public float Temperature { get; set; }\nSingle
"},{"location":"xmldocs/llama.common.inferenceparams/#repeatpenalty","title":"RepeatPenalty","text":"1.0 = disabled
public float RepeatPenalty { get; set; }\nSingle
"},{"location":"xmldocs/llama.common.inferenceparams/#repeatlasttokenscount","title":"RepeatLastTokensCount","text":"last n tokens to penalize (0 = disable penalty, -1 = context size) (repeat_last_n)
public int RepeatLastTokensCount { get; set; }\nInt32
"},{"location":"xmldocs/llama.common.inferenceparams/#frequencypenalty","title":"FrequencyPenalty","text":"frequency penalty coefficient 0.0 = disabled
public float FrequencyPenalty { get; set; }\nSingle
"},{"location":"xmldocs/llama.common.inferenceparams/#presencepenalty","title":"PresencePenalty","text":"presence penalty coefficient 0.0 = disabled
public float PresencePenalty { get; set; }\nSingle
"},{"location":"xmldocs/llama.common.inferenceparams/#mirostat","title":"Mirostat","text":"Mirostat uses tokens instead of words. algorithm described in the paper https://arxiv.org/abs/2007.14966. 0 = disabled, 1 = mirostat, 2 = mirostat 2.0
public MiroStateType Mirostat { get; set; }\nMiroStateType
"},{"location":"xmldocs/llama.common.inferenceparams/#mirostattau","title":"MirostatTau","text":"target entropy
public float MirostatTau { get; set; }\nSingle
"},{"location":"xmldocs/llama.common.inferenceparams/#mirostateta","title":"MirostatEta","text":"learning rate
public float MirostatEta { get; set; }\nSingle
"},{"location":"xmldocs/llama.common.inferenceparams/#penalizenl","title":"PenalizeNL","text":"consider newlines as a repeatable token (penalize_nl)
public bool PenalizeNL { get; set; }\nBoolean
"},{"location":"xmldocs/llama.common.inferenceparams/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.common.inferenceparams/#inferenceparams_1","title":"InferenceParams()","text":"public InferenceParams()\nNamespace: LLama.Common
The default logger of LLamaSharp. On default it write to console. User methods of LLamaLogger.Default to change the behavior. It's more recommended to inherit ILLamaLogger to cosutomize the behavior.
public sealed class LLamaDefaultLogger : ILLamaLogger\nInheritance Object \u2192 LLamaDefaultLogger Implements ILLamaLogger
"},{"location":"xmldocs/llama.common.llamadefaultlogger/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.common.llamadefaultlogger/#default","title":"Default","text":"public static LLamaDefaultLogger Default { get; }\nLLamaDefaultLogger
"},{"location":"xmldocs/llama.common.llamadefaultlogger/#methods","title":"Methods","text":""},{"location":"xmldocs/llama.common.llamadefaultlogger/#enableconsole","title":"EnableConsole()","text":"public LLamaDefaultLogger EnableConsole()\nLLamaDefaultLogger
"},{"location":"xmldocs/llama.common.llamadefaultlogger/#disableconsole","title":"DisableConsole()","text":"public LLamaDefaultLogger DisableConsole()\nLLamaDefaultLogger
"},{"location":"xmldocs/llama.common.llamadefaultlogger/#enablefilestring-filemode","title":"EnableFile(String, FileMode)","text":"public LLamaDefaultLogger EnableFile(string filename, FileMode mode)\nfilename String
mode FileMode
LLamaDefaultLogger
"},{"location":"xmldocs/llama.common.llamadefaultlogger/#disablefilestring","title":"DisableFile(String)","text":"public LLamaDefaultLogger DisableFile(string filename)\nfilename String
LLamaDefaultLogger
"},{"location":"xmldocs/llama.common.llamadefaultlogger/#logstring-string-loglevel","title":"Log(String, String, LogLevel)","text":"public void Log(string source, string message, LogLevel level)\nsource String
message String
level LogLevel
public void Info(string message)\nmessage String
public void Warn(string message)\nmessage String
public void Error(string message)\nmessage String
Namespace: LLama.Common
public enum MiroStateType\nInheritance Object \u2192 ValueType \u2192 Enum \u2192 MiroStateType Implements IComparable, IFormattable, IConvertible
"},{"location":"xmldocs/llama.common.mirostatetype/#fields","title":"Fields","text":"Name Value Description"},{"location":"xmldocs/llama.common.modelparams/","title":"ModelParams","text":"Namespace: LLama.Common
public class ModelParams\nInheritance Object \u2192 ModelParams
"},{"location":"xmldocs/llama.common.modelparams/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.common.modelparams/#contextsize","title":"ContextSize","text":"Model context size (n_ctx)
public int ContextSize { get; set; }\nInt32
"},{"location":"xmldocs/llama.common.modelparams/#gpulayercount","title":"GpuLayerCount","text":"Number of layers to run in VRAM / GPU memory (n_gpu_layers)
public int GpuLayerCount { get; set; }\nInt32
"},{"location":"xmldocs/llama.common.modelparams/#seed","title":"Seed","text":"Seed for the random number generator (seed)
public int Seed { get; set; }\nInt32
"},{"location":"xmldocs/llama.common.modelparams/#usefp16memory","title":"UseFp16Memory","text":"Use f16 instead of f32 for memory kv (memory_f16)
public bool UseFp16Memory { get; set; }\nBoolean
"},{"location":"xmldocs/llama.common.modelparams/#usememorymap","title":"UseMemorymap","text":"Use mmap for faster loads (use_mmap)
public bool UseMemorymap { get; set; }\nBoolean
"},{"location":"xmldocs/llama.common.modelparams/#usememorylock","title":"UseMemoryLock","text":"Use mlock to keep model in memory (use_mlock)
public bool UseMemoryLock { get; set; }\nBoolean
"},{"location":"xmldocs/llama.common.modelparams/#perplexity","title":"Perplexity","text":"Compute perplexity over the prompt (perplexity)
public bool Perplexity { get; set; }\nBoolean
"},{"location":"xmldocs/llama.common.modelparams/#modelpath","title":"ModelPath","text":"Model path (model)
public string ModelPath { get; set; }\nString
"},{"location":"xmldocs/llama.common.modelparams/#loraadapter","title":"LoraAdapter","text":"lora adapter path (lora_adapter)
public string LoraAdapter { get; set; }\nString
"},{"location":"xmldocs/llama.common.modelparams/#lorabase","title":"LoraBase","text":"base model path for the lora adapter (lora_base)
public string LoraBase { get; set; }\nString
"},{"location":"xmldocs/llama.common.modelparams/#threads","title":"Threads","text":"Number of threads (-1 = autodetect) (n_threads)
public int Threads { get; set; }\nInt32
"},{"location":"xmldocs/llama.common.modelparams/#batchsize","title":"BatchSize","text":"batch size for prompt processing (must be >=32 to use BLAS) (n_batch)
public int BatchSize { get; set; }\nInt32
"},{"location":"xmldocs/llama.common.modelparams/#converteostonewline","title":"ConvertEosToNewLine","text":"Whether to convert eos to newline during the inference.
public bool ConvertEosToNewLine { get; set; }\nBoolean
"},{"location":"xmldocs/llama.common.modelparams/#embeddingmode","title":"EmbeddingMode","text":"Whether to use embedding mode. (embedding) Note that if this is set to true, The LLamaModel won't produce text response anymore.
public bool EmbeddingMode { get; set; }\nBoolean
"},{"location":"xmldocs/llama.common.modelparams/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.common.modelparams/#modelparamsstring-int32-int32-int32-boolean-boolean-boolean-boolean-string-string-int32-int32-boolean-boolean","title":"ModelParams(String, Int32, Int32, Int32, Boolean, Boolean, Boolean, Boolean, String, String, Int32, Int32, Boolean, Boolean)","text":"public ModelParams(string modelPath, int contextSize, int gpuLayerCount, int seed, bool useFp16Memory, bool useMemorymap, bool useMemoryLock, bool perplexity, string loraAdapter, string loraBase, int threads, int batchSize, bool convertEosToNewLine, bool embeddingMode)\nmodelPath String The model path.
contextSize Int32 Model context size (n_ctx)
gpuLayerCount Int32 Number of layers to run in VRAM / GPU memory (n_gpu_layers)
seed Int32 Seed for the random number generator (seed)
useFp16Memory Boolean Whether to use f16 instead of f32 for memory kv (memory_f16)
useMemorymap Boolean Whether to use mmap for faster loads (use_mmap)
useMemoryLock Boolean Whether to use mlock to keep model in memory (use_mlock)
perplexity Boolean Thether to compute perplexity over the prompt (perplexity)
loraAdapter String Lora adapter path (lora_adapter)
loraBase String Base model path for the lora adapter (lora_base)
threads Int32 Number of threads (-1 = autodetect) (n_threads)
batchSize Int32 Batch size for prompt processing (must be >=32 to use BLAS) (n_batch)
convertEosToNewLine Boolean Whether to convert eos to newline during the inference.
embeddingMode Boolean Whether to use embedding mode. (embedding) Note that if this is set to true, The LLamaModel won't produce text response anymore.
Namespace: LLama.Exceptions
public class RuntimeError : System.Exception, System.Runtime.Serialization.ISerializable\nInheritance Object \u2192 Exception \u2192 RuntimeError Implements ISerializable
"},{"location":"xmldocs/llama.exceptions.runtimeerror/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.exceptions.runtimeerror/#targetsite","title":"TargetSite","text":"public MethodBase TargetSite { get; }\nMethodBase
"},{"location":"xmldocs/llama.exceptions.runtimeerror/#message","title":"Message","text":"public string Message { get; }\nString
"},{"location":"xmldocs/llama.exceptions.runtimeerror/#data","title":"Data","text":"public IDictionary Data { get; }\nIDictionary
"},{"location":"xmldocs/llama.exceptions.runtimeerror/#innerexception","title":"InnerException","text":"public Exception InnerException { get; }\nException
"},{"location":"xmldocs/llama.exceptions.runtimeerror/#helplink","title":"HelpLink","text":"public string HelpLink { get; set; }\nString
"},{"location":"xmldocs/llama.exceptions.runtimeerror/#source","title":"Source","text":"public string Source { get; set; }\nString
"},{"location":"xmldocs/llama.exceptions.runtimeerror/#hresult","title":"HResult","text":"public int HResult { get; set; }\nInt32
"},{"location":"xmldocs/llama.exceptions.runtimeerror/#stacktrace","title":"StackTrace","text":"public string StackTrace { get; }\nString
"},{"location":"xmldocs/llama.exceptions.runtimeerror/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.exceptions.runtimeerror/#runtimeerror_1","title":"RuntimeError()","text":"public RuntimeError()\npublic RuntimeError(string message)\nmessage String
Namespace: LLama.Extensions
public static class DictionaryExtension\nInheritance Object \u2192 DictionaryExtension
"},{"location":"xmldocs/llama.extensions.dictionaryextension/#methods","title":"Methods","text":""},{"location":"xmldocs/llama.extensions.dictionaryextension/#deconstructt1-t2keyvaluepairt1-t2-t1-t2","title":"Deconstruct<T1, T2>(KeyValuePair<T1, T2>, T1&, T2&)","text":"public static void Deconstruct<T1, T2>(KeyValuePair<T1, T2> pair, T1& first, T2& second)\nT1
T2
pair KeyValuePair<T1, T2>
first T1&
second T2&
public static void Update<T1, T2>(Dictionary<T1, T2> dic, IDictionary<T1, T2> other)\nT1
T2
dic Dictionary<T1, T2>
other IDictionary<T1, T2>
public static T2 GetOrDefault<T1, T2>(Dictionary<T1, T2> dic, T1 key, T2 defaultValue)\nT1
T2
dic Dictionary<T1, T2>
key T1
defaultValue T2
T2
"},{"location":"xmldocs/llama.instructexecutor/","title":"InstructExecutor","text":"Namespace: LLama
The LLama executor for instruct mode.
public class InstructExecutor : StatefulExecutorBase, LLama.Abstractions.ILLamaExecutor\nInheritance Object \u2192 StatefulExecutorBase \u2192 InstructExecutor Implements ILLamaExecutor
"},{"location":"xmldocs/llama.instructexecutor/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.instructexecutor/#model","title":"Model","text":"The mode used by the executor.
public LLamaModel Model { get; }\nLLamaModel
"},{"location":"xmldocs/llama.instructexecutor/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.instructexecutor/#instructexecutorllamamodel-string-string","title":"InstructExecutor(LLamaModel, String, String)","text":"public InstructExecutor(LLamaModel model, string instructionPrefix, string instructionSuffix)\nmodel LLamaModel
instructionPrefix String
instructionSuffix String
public ExecutorBaseState GetStateData()\nExecutorBaseState
"},{"location":"xmldocs/llama.instructexecutor/#loadstateexecutorbasestate","title":"LoadState(ExecutorBaseState)","text":"public void LoadState(ExecutorBaseState data)\ndata ExecutorBaseState
public void SaveState(string filename)\nfilename String
public void LoadState(string filename)\nfilename String
protected bool GetLoopCondition(InferStateArgs args)\nargs InferStateArgs
Boolean
"},{"location":"xmldocs/llama.instructexecutor/#preprocessinputsstring-inferstateargs","title":"PreprocessInputs(String, InferStateArgs)","text":"protected void PreprocessInputs(string text, InferStateArgs args)\ntext String
args InferStateArgs
protected bool PostProcess(InferenceParams inferenceParams, InferStateArgs args, IEnumerable`1& extraOutputs)\ninferenceParams InferenceParams
args InferStateArgs
extraOutputs IEnumerable`1&
Boolean
"},{"location":"xmldocs/llama.instructexecutor/#inferinternalinferenceparams-inferstateargs","title":"InferInternal(InferenceParams, InferStateArgs)","text":"protected void InferInternal(InferenceParams inferenceParams, InferStateArgs args)\ninferenceParams InferenceParams
args InferStateArgs
Namespace: LLama
The LLama executor for interactive mode.
public class InteractiveExecutor : StatefulExecutorBase, LLama.Abstractions.ILLamaExecutor\nInheritance Object \u2192 StatefulExecutorBase \u2192 InteractiveExecutor Implements ILLamaExecutor
"},{"location":"xmldocs/llama.interactiveexecutor/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.interactiveexecutor/#model","title":"Model","text":"The mode used by the executor.
public LLamaModel Model { get; }\nLLamaModel
"},{"location":"xmldocs/llama.interactiveexecutor/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.interactiveexecutor/#interactiveexecutorllamamodel","title":"InteractiveExecutor(LLamaModel)","text":"public InteractiveExecutor(LLamaModel model)\nmodel LLamaModel
public ExecutorBaseState GetStateData()\nExecutorBaseState
"},{"location":"xmldocs/llama.interactiveexecutor/#loadstateexecutorbasestate","title":"LoadState(ExecutorBaseState)","text":"public void LoadState(ExecutorBaseState data)\ndata ExecutorBaseState
public void SaveState(string filename)\nfilename String
public void LoadState(string filename)\nfilename String
Define whether to continue the loop to generate responses.
protected bool GetLoopCondition(InferStateArgs args)\nargs InferStateArgs
Boolean
"},{"location":"xmldocs/llama.interactiveexecutor/#preprocessinputsstring-inferstateargs","title":"PreprocessInputs(String, InferStateArgs)","text":"protected void PreprocessInputs(string text, InferStateArgs args)\ntext String
args InferStateArgs
Return whether to break the generation.
protected bool PostProcess(InferenceParams inferenceParams, InferStateArgs args, IEnumerable`1& extraOutputs)\ninferenceParams InferenceParams
args InferStateArgs
extraOutputs IEnumerable`1&
Boolean
"},{"location":"xmldocs/llama.interactiveexecutor/#inferinternalinferenceparams-inferstateargs","title":"InferInternal(InferenceParams, InferStateArgs)","text":"protected void InferInternal(InferenceParams inferenceParams, InferStateArgs args)\ninferenceParams InferenceParams
args InferStateArgs
Namespace: LLama
The embedder for LLama, which supports getting embeddings from text.
public class LLamaEmbedder : System.IDisposable\nInheritance Object \u2192 LLamaEmbedder Implements IDisposable
"},{"location":"xmldocs/llama.llamaembedder/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.llamaembedder/#llamaembeddermodelparams","title":"LLamaEmbedder(ModelParams)","text":"public LLamaEmbedder(ModelParams params)\nparams ModelParams
Get the embeddings of the text.
public Single[] GetEmbeddings(string text, int threads, bool addBos, string encoding)\ntext String
threads Int32 Threads used for inference.
addBos Boolean Add bos to the text.
encoding String
Single[]
"},{"location":"xmldocs/llama.llamaembedder/#exceptions","title":"Exceptions","text":"RuntimeError
"},{"location":"xmldocs/llama.llamaembedder/#dispose","title":"Dispose()","text":"public void Dispose()\nNamespace: LLama
The abstraction of a LLama model, which holds the context in the native library.
public class LLamaModel : System.IDisposable\nInheritance Object \u2192 LLamaModel Implements IDisposable
"},{"location":"xmldocs/llama.llamamodel/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.llamamodel/#contextsize","title":"ContextSize","text":"The context size.
public int ContextSize { get; }\nInt32
"},{"location":"xmldocs/llama.llamamodel/#params","title":"Params","text":"The model params set for this model.
public ModelParams Params { get; set; }\nModelParams
"},{"location":"xmldocs/llama.llamamodel/#nativehandle","title":"NativeHandle","text":"The native handle, which is used to be passed to the native APIs. Please avoid using it unless you know what is the usage of the Native API.
public SafeLLamaContextHandle NativeHandle { get; }\nSafeLLamaContextHandle
"},{"location":"xmldocs/llama.llamamodel/#encoding","title":"Encoding","text":"The encoding set for this model to deal with text input.
public Encoding Encoding { get; }\nEncoding
"},{"location":"xmldocs/llama.llamamodel/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.llamamodel/#llamamodelmodelparams-string-illamalogger","title":"LLamaModel(ModelParams, String, ILLamaLogger)","text":"public LLamaModel(ModelParams Params, string encoding, ILLamaLogger logger)\nParams ModelParams Model params.
encoding String Encoding to deal with text input.
logger ILLamaLogger The logger.
Tokenize a string.
public IEnumerable<int> Tokenize(string text, bool addBos)\ntext String
addBos Boolean Whether to add a bos to the text.
IEnumerable<Int32>
"},{"location":"xmldocs/llama.llamamodel/#detokenizeienumerableint32","title":"DeTokenize(IEnumerable<Int32>)","text":"Detokenize the tokens to text.
public string DeTokenize(IEnumerable<int> tokens)\ntokens IEnumerable<Int32>
String
"},{"location":"xmldocs/llama.llamamodel/#savestatestring","title":"SaveState(String)","text":"Save the state to specified path.
public void SaveState(string filename)\nfilename String
Get the state data as a byte array.
public Byte[] GetStateData()\nByte[]
"},{"location":"xmldocs/llama.llamamodel/#loadstatestring","title":"LoadState(String)","text":"Load the state from specified path.
public void LoadState(string filename)\nfilename String
RuntimeError
"},{"location":"xmldocs/llama.llamamodel/#loadstatebyte","title":"LoadState(Byte[])","text":"Load the state from memory.
public void LoadState(Byte[] stateData)\nstateData Byte[]
RuntimeError
"},{"location":"xmldocs/llama.llamamodel/#samplellamatokendataarray-single-mirostatetype-single-single-int32-single-single-single","title":"Sample(LLamaTokenDataArray, Single, MiroStateType, Single, Single, Int32, Single, Single, Single)","text":"Perform the sampling. Please don't use it unless you fully know what it does.
public int Sample(LLamaTokenDataArray candidates, float temperature, MiroStateType mirostat, float mirostatTau, float mirostatEta, int topK, float topP, float tfsZ, float typicalP)\ncandidates LLamaTokenDataArray
temperature Single
mirostat MiroStateType
mirostatTau Single
mirostatEta Single
topK Int32
topP Single
tfsZ Single
typicalP Single
Int32
"},{"location":"xmldocs/llama.llamamodel/#applypenaltyienumerableint32-dictionaryint32-single-int32-single-single-single-boolean","title":"ApplyPenalty(IEnumerable<Int32>, Dictionary<Int32, Single>, Int32, Single, Single, Single, Boolean)","text":"Apply the penalty for the tokens. Please don't use it unless you fully know what it does.
public LLamaTokenDataArray ApplyPenalty(IEnumerable<int> lastTokens, Dictionary<int, float> logitBias, int repeatLastTokensCount, float repeatPenalty, float alphaFrequency, float alphaPresence, bool penalizeNL)\nlastTokens IEnumerable<Int32>
logitBias Dictionary<Int32, Single>
repeatLastTokensCount Int32
repeatPenalty Single
alphaFrequency Single
alphaPresence Single
penalizeNL Boolean
LLamaTokenDataArray
"},{"location":"xmldocs/llama.llamamodel/#evalint32-int32","title":"Eval(Int32[], Int32)","text":"public int Eval(Int32[] tokens, int pastTokensCount)\ntokens Int32[]
pastTokensCount Int32
Int32 The updated pastTokensCount.
RuntimeError
"},{"location":"xmldocs/llama.llamamodel/#generateresultienumerableint32","title":"GenerateResult(IEnumerable<Int32>)","text":"internal IEnumerable<string> GenerateResult(IEnumerable<int> ids)\nids IEnumerable<Int32>
IEnumerable<String>
"},{"location":"xmldocs/llama.llamamodel/#dispose","title":"Dispose()","text":"public void Dispose()\nNamespace: LLama
The quantizer to quantize the model.
public static class LLamaQuantizer\nInheritance Object \u2192 LLamaQuantizer
"},{"location":"xmldocs/llama.llamaquantizer/#methods","title":"Methods","text":""},{"location":"xmldocs/llama.llamaquantizer/#quantizestring-string-llamaftype-int32","title":"Quantize(String, String, LLamaFtype, Int32)","text":"Quantize the model.
public static bool Quantize(string srcFileName, string dstFilename, LLamaFtype ftype, int nthread)\nsrcFileName String The model file to be quantized.
dstFilename String The path to save the quantized model.
ftype LLamaFtype The type of quantization.
nthread Int32 Thread to be used during the quantization. By default it's the physical core number.
Boolean Whether the quantization is successful.
"},{"location":"xmldocs/llama.llamaquantizer/#exceptions","title":"Exceptions","text":"ArgumentException
"},{"location":"xmldocs/llama.llamaquantizer/#quantizestring-string-string-int32","title":"Quantize(String, String, String, Int32)","text":"Quantize the model.
public static bool Quantize(string srcFileName, string dstFilename, string ftype, int nthread)\nsrcFileName String The model file to be quantized.
dstFilename String The path to save the quantized model.
ftype String The type of quantization.
nthread Int32 Thread to be used during the quantization. By default it's the physical core number.
Boolean Whether the quantization is successful.
"},{"location":"xmldocs/llama.llamaquantizer/#exceptions_1","title":"Exceptions","text":"ArgumentException
"},{"location":"xmldocs/llama.llamatransforms/","title":"LLamaTransforms","text":"Namespace: LLama
A class that contains all the transforms provided internally by LLama.
public class LLamaTransforms\nInheritance Object \u2192 LLamaTransforms
"},{"location":"xmldocs/llama.llamatransforms/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.llamatransforms/#llamatransforms_1","title":"LLamaTransforms()","text":"public LLamaTransforms()\nNamespace: LLama.Native
public struct LLamaContextParams\nInheritance Object \u2192 ValueType \u2192 LLamaContextParams
"},{"location":"xmldocs/llama.native.llamacontextparams/#fields","title":"Fields","text":""},{"location":"xmldocs/llama.native.llamacontextparams/#n_ctx","title":"n_ctx","text":"text context
public int n_ctx;\nnumber of layers to store in VRAM
public int n_gpu_layers;\nRNG seed, -1 for random
public int seed;\nuse fp16 for KV cache
public bool f16_kv;\nthe llama_eval() call computes all logits, not just the last one
public bool logits_all;\nonly load the vocabulary, no weights
public bool vocab_only;\nuse mmap if possible
public bool use_mmap;\nforce system to keep model in RAM
public bool use_mlock;\nembedding mode only
public bool embedding;\ncalled with a progress value between 0 and 1, pass NULL to disable
public IntPtr progress_callback;\ncontext pointer passed to the progress callback
public IntPtr progress_callback_user_data;\nNamespace: LLama.Native
public enum LLamaFtype\nInheritance Object \u2192 ValueType \u2192 Enum \u2192 LLamaFtype Implements IComparable, IFormattable, IConvertible
"},{"location":"xmldocs/llama.native.llamaftype/#fields","title":"Fields","text":"Name Value Description"},{"location":"xmldocs/llama.native.llamatokendata/","title":"LLamaTokenData","text":"Namespace: LLama.Native
public struct LLamaTokenData\nInheritance Object \u2192 ValueType \u2192 LLamaTokenData
"},{"location":"xmldocs/llama.native.llamatokendata/#fields","title":"Fields","text":""},{"location":"xmldocs/llama.native.llamatokendata/#id","title":"id","text":"token id
public int id;\nlog-odds of the token
public float logit;\nprobability of the token
public float p;\nLLamaTokenData(int id, float logit, float p)\nid Int32
logit Single
p Single
Namespace: LLama.Native
public struct LLamaTokenDataArray\nInheritance Object \u2192 ValueType \u2192 LLamaTokenDataArray
"},{"location":"xmldocs/llama.native.llamatokendataarray/#fields","title":"Fields","text":""},{"location":"xmldocs/llama.native.llamatokendataarray/#data","title":"data","text":"public Memory<LLamaTokenData> data;\npublic ulong size;\npublic bool sorted;\nLLamaTokenDataArray(LLamaTokenData[] data, ulong size, bool sorted)\ndata LLamaTokenData[]
size UInt64
sorted Boolean
Namespace: LLama.Native
public struct LLamaTokenDataArrayNative\nInheritance Object \u2192 ValueType \u2192 LLamaTokenDataArrayNative
"},{"location":"xmldocs/llama.native.llamatokendataarraynative/#fields","title":"Fields","text":""},{"location":"xmldocs/llama.native.llamatokendataarraynative/#data","title":"data","text":"public IntPtr data;\npublic ulong size;\npublic bool sorted;\nNamespace: LLama.Native
public class NativeApi\nInheritance Object \u2192 NativeApi
"},{"location":"xmldocs/llama.native.nativeapi/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.native.nativeapi/#nativeapi_1","title":"NativeApi()","text":"public NativeApi()\npublic static void llama_print_timings(SafeLLamaContextHandle ctx)\nctx SafeLLamaContextHandle
public static void llama_reset_timings(SafeLLamaContextHandle ctx)\nctx SafeLLamaContextHandle
Print system information
public static IntPtr llama_print_system_info()\nIntPtr
"},{"location":"xmldocs/llama.native.nativeapi/#llama_model_quantizestring-string-llamaftype-int32","title":"llama_model_quantize(String, String, LLamaFtype, Int32)","text":"public static int llama_model_quantize(string fname_inp, string fname_out, LLamaFtype ftype, int nthread)\nfname_inp String
fname_out String
ftype LLamaFtype
nthread Int32
Int32
"},{"location":"xmldocs/llama.native.nativeapi/#llama_sample_repetition_penaltysafellamacontexthandle-intptr-int32-uint64-single","title":"llama_sample_repetition_penalty(SafeLLamaContextHandle, IntPtr, Int32[], UInt64, Single)","text":"Repetition penalty described in CTRL academic paper https://arxiv.org/abs/1909.05858, with negative logit fix.
public static void llama_sample_repetition_penalty(SafeLLamaContextHandle ctx, IntPtr candidates, Int32[] last_tokens, ulong last_tokens_size, float penalty)\nctx SafeLLamaContextHandle
candidates IntPtr Pointer to LLamaTokenDataArray
last_tokens Int32[]
last_tokens_size UInt64
penalty Single
Frequency and presence penalties described in OpenAI API https://platform.openai.com/docs/api-reference/parameter-details.
public static void llama_sample_frequency_and_presence_penalties(SafeLLamaContextHandle ctx, IntPtr candidates, Int32[] last_tokens, ulong last_tokens_size, float alpha_frequency, float alpha_presence)\nctx SafeLLamaContextHandle
candidates IntPtr Pointer to LLamaTokenDataArray
last_tokens Int32[]
last_tokens_size UInt64
alpha_frequency Single
alpha_presence Single
Sorts candidate tokens by their logits in descending order and calculate probabilities based on logits.
public static void llama_sample_softmax(SafeLLamaContextHandle ctx, IntPtr candidates)\nctx SafeLLamaContextHandle
candidates IntPtr Pointer to LLamaTokenDataArray
Top-K sampling described in academic paper \"The Curious Case of Neural Text Degeneration\" https://arxiv.org/abs/1904.09751
public static void llama_sample_top_k(SafeLLamaContextHandle ctx, IntPtr candidates, int k, ulong min_keep)\nctx SafeLLamaContextHandle
candidates IntPtr Pointer to LLamaTokenDataArray
k Int32
min_keep UInt64
Nucleus sampling described in academic paper \"The Curious Case of Neural Text Degeneration\" https://arxiv.org/abs/1904.09751
public static void llama_sample_top_p(SafeLLamaContextHandle ctx, IntPtr candidates, float p, ulong min_keep)\nctx SafeLLamaContextHandle
candidates IntPtr Pointer to LLamaTokenDataArray
p Single
min_keep UInt64
Tail Free Sampling described in https://www.trentonbricken.com/Tail-Free-Sampling/.
public static void llama_sample_tail_free(SafeLLamaContextHandle ctx, IntPtr candidates, float z, ulong min_keep)\nctx SafeLLamaContextHandle
candidates IntPtr Pointer to LLamaTokenDataArray
z Single
min_keep UInt64
Locally Typical Sampling implementation described in the paper https://arxiv.org/abs/2202.00666.
public static void llama_sample_typical(SafeLLamaContextHandle ctx, IntPtr candidates, float p, ulong min_keep)\nctx SafeLLamaContextHandle
candidates IntPtr Pointer to LLamaTokenDataArray
p Single
min_keep UInt64
public static void llama_sample_temperature(SafeLLamaContextHandle ctx, IntPtr candidates, float temp)\nctx SafeLLamaContextHandle
candidates IntPtr
temp Single
Mirostat 1.0 algorithm described in the paper https://arxiv.org/abs/2007.14966. Uses tokens instead of words.
public static int llama_sample_token_mirostat(SafeLLamaContextHandle ctx, IntPtr candidates, float tau, float eta, int m, Single* mu)\nctx SafeLLamaContextHandle
candidates IntPtr A vector of llama_token_data containing the candidate tokens, their probabilities (p), and log-odds (logit) for the current position in the generated text.
tau Single The target cross-entropy (or surprise) value you want to achieve for the generated text. A higher value corresponds to more surprising or less predictable text, while a lower value corresponds to less surprising or more predictable text.
eta Single The learning rate used to update mu based on the error between the target and observed surprisal of the sampled word. A larger learning rate will cause mu to be updated more quickly, while a smaller learning rate will result in slower updates.
m Int32 The number of tokens considered in the estimation of s_hat. This is an arbitrary value that is used to calculate s_hat, which in turn helps to calculate the value of k. In the paper, they use m = 100, but you can experiment with different values to see how it affects the performance of the algorithm.
mu Single* Maximum cross-entropy. This value is initialized to be twice the target cross-entropy (2 * tau) and is updated in the algorithm based on the error between the target and observed surprisal.
Int32
"},{"location":"xmldocs/llama.native.nativeapi/#llama_sample_token_mirostat_v2safellamacontexthandle-intptr-single-single-single","title":"llama_sample_token_mirostat_v2(SafeLLamaContextHandle, IntPtr, Single, Single, Single*)","text":"Mirostat 2.0 algorithm described in the paper https://arxiv.org/abs/2007.14966. Uses tokens instead of words.
public static int llama_sample_token_mirostat_v2(SafeLLamaContextHandle ctx, IntPtr candidates, float tau, float eta, Single* mu)\nctx SafeLLamaContextHandle
candidates IntPtr A vector of llama_token_data containing the candidate tokens, their probabilities (p), and log-odds (logit) for the current position in the generated text.
tau Single The target cross-entropy (or surprise) value you want to achieve for the generated text. A higher value corresponds to more surprising or less predictable text, while a lower value corresponds to less surprising or more predictable text.
eta Single The learning rate used to update mu based on the error between the target and observed surprisal of the sampled word. A larger learning rate will cause mu to be updated more quickly, while a smaller learning rate will result in slower updates.
mu Single* Maximum cross-entropy. This value is initialized to be twice the target cross-entropy (2 * tau) and is updated in the algorithm based on the error between the target and observed surprisal.
Int32
"},{"location":"xmldocs/llama.native.nativeapi/#llama_sample_token_greedysafellamacontexthandle-intptr","title":"llama_sample_token_greedy(SafeLLamaContextHandle, IntPtr)","text":"Selects the token with the highest probability.
public static int llama_sample_token_greedy(SafeLLamaContextHandle ctx, IntPtr candidates)\nctx SafeLLamaContextHandle
candidates IntPtr Pointer to LLamaTokenDataArray
Int32
"},{"location":"xmldocs/llama.native.nativeapi/#llama_sample_tokensafellamacontexthandle-intptr","title":"llama_sample_token(SafeLLamaContextHandle, IntPtr)","text":"Randomly selects a token from the candidates based on their probabilities.
public static int llama_sample_token(SafeLLamaContextHandle ctx, IntPtr candidates)\nctx SafeLLamaContextHandle
candidates IntPtr Pointer to LLamaTokenDataArray
Int32
"},{"location":"xmldocs/llama.native.nativeapi/#llama_empty_call","title":"llama_empty_call()","text":"public static bool llama_empty_call()\nBoolean
"},{"location":"xmldocs/llama.native.nativeapi/#llama_context_default_params","title":"llama_context_default_params()","text":"public static LLamaContextParams llama_context_default_params()\nLLamaContextParams
"},{"location":"xmldocs/llama.native.nativeapi/#llama_mmap_supported","title":"llama_mmap_supported()","text":"public static bool llama_mmap_supported()\nBoolean
"},{"location":"xmldocs/llama.native.nativeapi/#llama_mlock_supported","title":"llama_mlock_supported()","text":"public static bool llama_mlock_supported()\nBoolean
"},{"location":"xmldocs/llama.native.nativeapi/#llama_init_from_filestring-llamacontextparams","title":"llama_init_from_file(String, LLamaContextParams)","text":"Various functions for loading a ggml llama model. Allocate (almost) all memory needed for the model. Return NULL on failure
public static IntPtr llama_init_from_file(string path_model, LLamaContextParams params_)\npath_model String
params_ LLamaContextParams
IntPtr
"},{"location":"xmldocs/llama.native.nativeapi/#llama_init_backend","title":"llama_init_backend()","text":"not great API - very likely to change. Initialize the llama + ggml backend Call once at the start of the program
public static void llama_init_backend()\nFrees all allocated memory
public static void llama_free(IntPtr ctx)\nctx IntPtr
Apply a LoRA adapter to a loaded model path_base_model is the path to a higher quality model to use as a base for the layers modified by the adapter. Can be NULL to use the current loaded model. The model needs to be reloaded before applying a new adapter, otherwise the adapter will be applied on top of the previous one
public static int llama_apply_lora_from_file(SafeLLamaContextHandle ctx, string path_lora, string path_base_model, int n_threads)\nctx SafeLLamaContextHandle
path_lora String
path_base_model String
n_threads Int32
Int32 Returns 0 on success
"},{"location":"xmldocs/llama.native.nativeapi/#llama_get_kv_cache_token_countsafellamacontexthandle","title":"llama_get_kv_cache_token_count(SafeLLamaContextHandle)","text":"Returns the number of tokens in the KV cache
public static int llama_get_kv_cache_token_count(SafeLLamaContextHandle ctx)\nctx SafeLLamaContextHandle
Int32
"},{"location":"xmldocs/llama.native.nativeapi/#llama_set_rng_seedsafellamacontexthandle-int32","title":"llama_set_rng_seed(SafeLLamaContextHandle, Int32)","text":"Sets the current rng seed.
public static void llama_set_rng_seed(SafeLLamaContextHandle ctx, int seed)\nctx SafeLLamaContextHandle
seed Int32
Returns the maximum size in bytes of the state (rng, logits, embedding and kv_cache) - will often be smaller after compacting tokens
public static ulong llama_get_state_size(SafeLLamaContextHandle ctx)\nctx SafeLLamaContextHandle
UInt64
"},{"location":"xmldocs/llama.native.nativeapi/#llama_copy_state_datasafellamacontexthandle-byte","title":"llama_copy_state_data(SafeLLamaContextHandle, Byte[])","text":"Copies the state to the specified destination address. Destination needs to have allocated enough memory. Returns the number of bytes copied
public static ulong llama_copy_state_data(SafeLLamaContextHandle ctx, Byte[] dest)\nctx SafeLLamaContextHandle
dest Byte[]
UInt64
"},{"location":"xmldocs/llama.native.nativeapi/#llama_set_state_datasafellamacontexthandle-byte","title":"llama_set_state_data(SafeLLamaContextHandle, Byte[])","text":"Set the state reading from the specified address Returns the number of bytes read
public static ulong llama_set_state_data(SafeLLamaContextHandle ctx, Byte[] src)\nctx SafeLLamaContextHandle
src Byte[]
UInt64
"},{"location":"xmldocs/llama.native.nativeapi/#llama_load_session_filesafellamacontexthandle-string-int32-uint64-uint64","title":"llama_load_session_file(SafeLLamaContextHandle, String, Int32[], UInt64, UInt64*)","text":"Load session file
public static bool llama_load_session_file(SafeLLamaContextHandle ctx, string path_session, Int32[] tokens_out, ulong n_token_capacity, UInt64* n_token_count_out)\nctx SafeLLamaContextHandle
path_session String
tokens_out Int32[]
n_token_capacity UInt64
n_token_count_out UInt64*
Boolean
"},{"location":"xmldocs/llama.native.nativeapi/#llama_save_session_filesafellamacontexthandle-string-int32-uint64","title":"llama_save_session_file(SafeLLamaContextHandle, String, Int32[], UInt64)","text":"Save session file
public static bool llama_save_session_file(SafeLLamaContextHandle ctx, string path_session, Int32[] tokens, ulong n_token_count)\nctx SafeLLamaContextHandle
path_session String
tokens Int32[]
n_token_count UInt64
Boolean
"},{"location":"xmldocs/llama.native.nativeapi/#llama_evalsafellamacontexthandle-int32-int32-int32-int32","title":"llama_eval(SafeLLamaContextHandle, Int32[], Int32, Int32, Int32)","text":"Run the llama inference to obtain the logits and probabilities for the next token. tokens + n_tokens is the provided batch of new tokens to process n_past is the number of tokens to use from previous eval calls
public static int llama_eval(SafeLLamaContextHandle ctx, Int32[] tokens, int n_tokens, int n_past, int n_threads)\nctx SafeLLamaContextHandle
tokens Int32[]
n_tokens Int32
n_past Int32
n_threads Int32
Int32 Returns 0 on success
"},{"location":"xmldocs/llama.native.nativeapi/#llama_eval_with_pointersafellamacontexthandle-int32-int32-int32-int32","title":"llama_eval_with_pointer(SafeLLamaContextHandle, Int32*, Int32, Int32, Int32)","text":"public static int llama_eval_with_pointer(SafeLLamaContextHandle ctx, Int32* tokens, int n_tokens, int n_past, int n_threads)\nctx SafeLLamaContextHandle
tokens Int32*
n_tokens Int32
n_past Int32
n_threads Int32
Int32
"},{"location":"xmldocs/llama.native.nativeapi/#llama_tokenizesafellamacontexthandle-string-encoding-int32-int32-boolean","title":"llama_tokenize(SafeLLamaContextHandle, String, Encoding, Int32[], Int32, Boolean)","text":"Convert the provided text into tokens. The tokens pointer must be large enough to hold the resulting tokens. Returns the number of tokens on success, no more than n_max_tokens Returns a negative number on failure - the number of tokens that would have been returned
public static int llama_tokenize(SafeLLamaContextHandle ctx, string text, Encoding encoding, Int32[] tokens, int n_max_tokens, bool add_bos)\nctx SafeLLamaContextHandle
text String
encoding Encoding
tokens Int32[]
n_max_tokens Int32
add_bos Boolean
Int32
"},{"location":"xmldocs/llama.native.nativeapi/#llama_tokenize_nativesafellamacontexthandle-sbyte-int32-int32-boolean","title":"llama_tokenize_native(SafeLLamaContextHandle, SByte[], Int32[], Int32, Boolean)","text":"public static int llama_tokenize_native(SafeLLamaContextHandle ctx, SByte[] text, Int32[] tokens, int n_max_tokens, bool add_bos)\nctx SafeLLamaContextHandle
text SByte[]
tokens Int32[]
n_max_tokens Int32
add_bos Boolean
Int32
"},{"location":"xmldocs/llama.native.nativeapi/#llama_n_vocabsafellamacontexthandle","title":"llama_n_vocab(SafeLLamaContextHandle)","text":"public static int llama_n_vocab(SafeLLamaContextHandle ctx)\nctx SafeLLamaContextHandle
Int32
"},{"location":"xmldocs/llama.native.nativeapi/#llama_n_ctxsafellamacontexthandle","title":"llama_n_ctx(SafeLLamaContextHandle)","text":"public static int llama_n_ctx(SafeLLamaContextHandle ctx)\nctx SafeLLamaContextHandle
Int32
"},{"location":"xmldocs/llama.native.nativeapi/#llama_n_embdsafellamacontexthandle","title":"llama_n_embd(SafeLLamaContextHandle)","text":"public static int llama_n_embd(SafeLLamaContextHandle ctx)\nctx SafeLLamaContextHandle
Int32
"},{"location":"xmldocs/llama.native.nativeapi/#llama_get_logitssafellamacontexthandle","title":"llama_get_logits(SafeLLamaContextHandle)","text":"Token logits obtained from the last call to llama_eval() The logits for the last token are stored in the last row Can be mutated in order to change the probabilities of the next token Rows: n_tokens Cols: n_vocab
public static Single* llama_get_logits(SafeLLamaContextHandle ctx)\nctx SafeLLamaContextHandle
Single*
"},{"location":"xmldocs/llama.native.nativeapi/#llama_get_embeddingssafellamacontexthandle","title":"llama_get_embeddings(SafeLLamaContextHandle)","text":"Get the embeddings for the input shape: [n_embd] (1-dimensional)
public static Single* llama_get_embeddings(SafeLLamaContextHandle ctx)\nctx SafeLLamaContextHandle
Single*
"},{"location":"xmldocs/llama.native.nativeapi/#llama_token_to_strsafellamacontexthandle-int32","title":"llama_token_to_str(SafeLLamaContextHandle, Int32)","text":"Token Id -> String. Uses the vocabulary in the provided context
public static IntPtr llama_token_to_str(SafeLLamaContextHandle ctx, int token)\nctx SafeLLamaContextHandle
token Int32
IntPtr Pointer to a string.
"},{"location":"xmldocs/llama.native.nativeapi/#llama_token_bos","title":"llama_token_bos()","text":"public static int llama_token_bos()\nInt32
"},{"location":"xmldocs/llama.native.nativeapi/#llama_token_eos","title":"llama_token_eos()","text":"public static int llama_token_eos()\nInt32
"},{"location":"xmldocs/llama.native.nativeapi/#llama_token_nl","title":"llama_token_nl()","text":"public static int llama_token_nl()\nInt32
"},{"location":"xmldocs/llama.native.safellamacontexthandle/","title":"SafeLLamaContextHandle","text":"Namespace: LLama.Native
public class SafeLLamaContextHandle : SafeLLamaHandleBase, System.IDisposable\nInheritance Object \u2192 CriticalFinalizerObject \u2192 SafeHandle \u2192 SafeLLamaHandleBase \u2192 SafeLLamaContextHandle Implements IDisposable
"},{"location":"xmldocs/llama.native.safellamacontexthandle/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.native.safellamacontexthandle/#isinvalid","title":"IsInvalid","text":"public bool IsInvalid { get; }\nBoolean
"},{"location":"xmldocs/llama.native.safellamacontexthandle/#isclosed","title":"IsClosed","text":"public bool IsClosed { get; }\nBoolean
"},{"location":"xmldocs/llama.native.safellamacontexthandle/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.native.safellamacontexthandle/#safellamacontexthandleintptr","title":"SafeLLamaContextHandle(IntPtr)","text":"public SafeLLamaContextHandle(IntPtr handle)\nhandle IntPtr
protected bool ReleaseHandle()\nBoolean
"},{"location":"xmldocs/llama.native.safellamahandlebase/","title":"SafeLLamaHandleBase","text":"Namespace: LLama.Native
public abstract class SafeLLamaHandleBase : System.Runtime.InteropServices.SafeHandle, System.IDisposable\nInheritance Object \u2192 CriticalFinalizerObject \u2192 SafeHandle \u2192 SafeLLamaHandleBase Implements IDisposable
"},{"location":"xmldocs/llama.native.safellamahandlebase/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.native.safellamahandlebase/#isinvalid","title":"IsInvalid","text":"public bool IsInvalid { get; }\nBoolean
"},{"location":"xmldocs/llama.native.safellamahandlebase/#isclosed","title":"IsClosed","text":"public bool IsClosed { get; }\nBoolean
"},{"location":"xmldocs/llama.native.safellamahandlebase/#methods","title":"Methods","text":""},{"location":"xmldocs/llama.native.safellamahandlebase/#tostring","title":"ToString()","text":"public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletion/","title":"ChatCompletion","text":"Namespace: LLama.OldVersion
public class ChatCompletion : System.IEquatable`1[[LLama.OldVersion.ChatCompletion, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]\nInheritance Object \u2192 ChatCompletion Implements IEquatable<ChatCompletion>
"},{"location":"xmldocs/llama.oldversion.chatcompletion/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.chatcompletion/#id","title":"Id","text":"public string Id { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletion/#object","title":"Object","text":"public string Object { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletion/#created","title":"Created","text":"public int Created { get; set; }\nInt32
"},{"location":"xmldocs/llama.oldversion.chatcompletion/#model","title":"Model","text":"public string Model { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletion/#choices","title":"Choices","text":"public ChatCompletionChoice[] Choices { get; set; }\nChatCompletionChoice[]
"},{"location":"xmldocs/llama.oldversion.chatcompletion/#usage","title":"Usage","text":"public CompletionUsage Usage { get; set; }\nCompletionUsage
"},{"location":"xmldocs/llama.oldversion.chatcompletion/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.chatcompletion/#chatcompletionstring-string-int32-string-chatcompletionchoice-completionusage","title":"ChatCompletion(String, String, Int32, String, ChatCompletionChoice[], CompletionUsage)","text":"public ChatCompletion(string Id, string Object, int Created, string Model, ChatCompletionChoice[] Choices, CompletionUsage Usage)\nId String
Object String
Created Int32
Model String
Choices ChatCompletionChoice[]
Usage CompletionUsage
public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletion/#printmembersstringbuilder","title":"PrintMembers(StringBuilder)","text":"protected bool PrintMembers(StringBuilder builder)\nbuilder StringBuilder
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletion/#gethashcode","title":"GetHashCode()","text":"public int GetHashCode()\nInt32
"},{"location":"xmldocs/llama.oldversion.chatcompletion/#equalsobject","title":"Equals(Object)","text":"public bool Equals(object obj)\nobj Object
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletion/#equalschatcompletion","title":"Equals(ChatCompletion)","text":"public bool Equals(ChatCompletion other)\nother ChatCompletion
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletion/#clone","title":"<Clone>$()","text":"public ChatCompletion <Clone>$()\nChatCompletion
"},{"location":"xmldocs/llama.oldversion.chatcompletion/#deconstructstring-string-int32-string-chatcompletionchoice-completionusage","title":"Deconstruct(String&, String&, Int32&, String&, ChatCompletionChoice[]&, CompletionUsage&)","text":"public void Deconstruct(String& Id, String& Object, Int32& Created, String& Model, ChatCompletionChoice[]& Choices, CompletionUsage& Usage)\nId String&
Object String&
Created Int32&
Model String&
Choices ChatCompletionChoice[]&
Usage CompletionUsage&
Namespace: LLama.OldVersion
public class ChatCompletionChoice : System.IEquatable`1[[LLama.OldVersion.ChatCompletionChoice, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]\nInheritance Object \u2192 ChatCompletionChoice Implements IEquatable<ChatCompletionChoice>
"},{"location":"xmldocs/llama.oldversion.chatcompletionchoice/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.chatcompletionchoice/#index","title":"Index","text":"public int Index { get; set; }\nInt32
"},{"location":"xmldocs/llama.oldversion.chatcompletionchoice/#message","title":"Message","text":"public ChatCompletionMessage Message { get; set; }\nChatCompletionMessage
"},{"location":"xmldocs/llama.oldversion.chatcompletionchoice/#finishreason","title":"FinishReason","text":"public string FinishReason { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletionchoice/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.chatcompletionchoice/#chatcompletionchoiceint32-chatcompletionmessage-string","title":"ChatCompletionChoice(Int32, ChatCompletionMessage, String)","text":"public ChatCompletionChoice(int Index, ChatCompletionMessage Message, string FinishReason)\nIndex Int32
Message ChatCompletionMessage
FinishReason String
public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletionchoice/#printmembersstringbuilder","title":"PrintMembers(StringBuilder)","text":"protected bool PrintMembers(StringBuilder builder)\nbuilder StringBuilder
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletionchoice/#gethashcode","title":"GetHashCode()","text":"public int GetHashCode()\nInt32
"},{"location":"xmldocs/llama.oldversion.chatcompletionchoice/#equalsobject","title":"Equals(Object)","text":"public bool Equals(object obj)\nobj Object
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletionchoice/#equalschatcompletionchoice","title":"Equals(ChatCompletionChoice)","text":"public bool Equals(ChatCompletionChoice other)\nother ChatCompletionChoice
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletionchoice/#clone","title":"<Clone>$()","text":"public ChatCompletionChoice <Clone>$()\nChatCompletionChoice
"},{"location":"xmldocs/llama.oldversion.chatcompletionchoice/#deconstructint32-chatcompletionmessage-string","title":"Deconstruct(Int32&, ChatCompletionMessage&, String&)","text":"public void Deconstruct(Int32& Index, ChatCompletionMessage& Message, String& FinishReason)\nIndex Int32&
Message ChatCompletionMessage&
FinishReason String&
Namespace: LLama.OldVersion
public class ChatCompletionChunk : System.IEquatable`1[[LLama.OldVersion.ChatCompletionChunk, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]\nInheritance Object \u2192 ChatCompletionChunk Implements IEquatable<ChatCompletionChunk>
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunk/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.chatcompletionchunk/#id","title":"Id","text":"public string Id { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunk/#model","title":"Model","text":"public string Model { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunk/#object","title":"Object","text":"public string Object { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunk/#created","title":"Created","text":"public int Created { get; set; }\nInt32
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunk/#choices","title":"Choices","text":"public ChatCompletionChunkChoice[] Choices { get; set; }\nChatCompletionChunkChoice[]
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunk/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.chatcompletionchunk/#chatcompletionchunkstring-string-string-int32-chatcompletionchunkchoice","title":"ChatCompletionChunk(String, String, String, Int32, ChatCompletionChunkChoice[])","text":"public ChatCompletionChunk(string Id, string Model, string Object, int Created, ChatCompletionChunkChoice[] Choices)\nId String
Model String
Object String
Created Int32
Choices ChatCompletionChunkChoice[]
public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunk/#printmembersstringbuilder","title":"PrintMembers(StringBuilder)","text":"protected bool PrintMembers(StringBuilder builder)\nbuilder StringBuilder
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunk/#gethashcode","title":"GetHashCode()","text":"public int GetHashCode()\nInt32
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunk/#equalsobject","title":"Equals(Object)","text":"public bool Equals(object obj)\nobj Object
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunk/#equalschatcompletionchunk","title":"Equals(ChatCompletionChunk)","text":"public bool Equals(ChatCompletionChunk other)\nother ChatCompletionChunk
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunk/#clone","title":"<Clone>$()","text":"public ChatCompletionChunk <Clone>$()\nChatCompletionChunk
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunk/#deconstructstring-string-string-int32-chatcompletionchunkchoice","title":"Deconstruct(String&, String&, String&, Int32&, ChatCompletionChunkChoice[]&)","text":"public void Deconstruct(String& Id, String& Model, String& Object, Int32& Created, ChatCompletionChunkChoice[]& Choices)\nId String&
Model String&
Object String&
Created Int32&
Choices ChatCompletionChunkChoice[]&
Namespace: LLama.OldVersion
public class ChatCompletionChunkChoice : System.IEquatable`1[[LLama.OldVersion.ChatCompletionChunkChoice, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]\nInheritance Object \u2192 ChatCompletionChunkChoice Implements IEquatable<ChatCompletionChunkChoice>
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkchoice/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.chatcompletionchunkchoice/#index","title":"Index","text":"public int Index { get; set; }\nInt32
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkchoice/#delta","title":"Delta","text":"public ChatCompletionChunkDelta Delta { get; set; }\nChatCompletionChunkDelta
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkchoice/#finishreason","title":"FinishReason","text":"public string FinishReason { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkchoice/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.chatcompletionchunkchoice/#chatcompletionchunkchoiceint32-chatcompletionchunkdelta-string","title":"ChatCompletionChunkChoice(Int32, ChatCompletionChunkDelta, String)","text":"public ChatCompletionChunkChoice(int Index, ChatCompletionChunkDelta Delta, string FinishReason)\nIndex Int32
Delta ChatCompletionChunkDelta
FinishReason String
public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkchoice/#printmembersstringbuilder","title":"PrintMembers(StringBuilder)","text":"protected bool PrintMembers(StringBuilder builder)\nbuilder StringBuilder
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkchoice/#gethashcode","title":"GetHashCode()","text":"public int GetHashCode()\nInt32
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkchoice/#equalsobject","title":"Equals(Object)","text":"public bool Equals(object obj)\nobj Object
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkchoice/#equalschatcompletionchunkchoice","title":"Equals(ChatCompletionChunkChoice)","text":"public bool Equals(ChatCompletionChunkChoice other)\nother ChatCompletionChunkChoice
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkchoice/#clone","title":"<Clone>$()","text":"public ChatCompletionChunkChoice <Clone>$()\nChatCompletionChunkChoice
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkchoice/#deconstructint32-chatcompletionchunkdelta-string","title":"Deconstruct(Int32&, ChatCompletionChunkDelta&, String&)","text":"public void Deconstruct(Int32& Index, ChatCompletionChunkDelta& Delta, String& FinishReason)\nIndex Int32&
Delta ChatCompletionChunkDelta&
FinishReason String&
Namespace: LLama.OldVersion
public class ChatCompletionChunkDelta : System.IEquatable`1[[LLama.OldVersion.ChatCompletionChunkDelta, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]\nInheritance Object \u2192 ChatCompletionChunkDelta Implements IEquatable<ChatCompletionChunkDelta>
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkdelta/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.chatcompletionchunkdelta/#role","title":"Role","text":"public string Role { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkdelta/#content","title":"Content","text":"public string Content { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkdelta/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.chatcompletionchunkdelta/#chatcompletionchunkdeltastring-string","title":"ChatCompletionChunkDelta(String, String)","text":"public ChatCompletionChunkDelta(string Role, string Content)\nRole String
Content String
public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkdelta/#printmembersstringbuilder","title":"PrintMembers(StringBuilder)","text":"protected bool PrintMembers(StringBuilder builder)\nbuilder StringBuilder
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkdelta/#gethashcode","title":"GetHashCode()","text":"public int GetHashCode()\nInt32
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkdelta/#equalsobject","title":"Equals(Object)","text":"public bool Equals(object obj)\nobj Object
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkdelta/#equalschatcompletionchunkdelta","title":"Equals(ChatCompletionChunkDelta)","text":"public bool Equals(ChatCompletionChunkDelta other)\nother ChatCompletionChunkDelta
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkdelta/#clone","title":"<Clone>$()","text":"public ChatCompletionChunkDelta <Clone>$()\nChatCompletionChunkDelta
"},{"location":"xmldocs/llama.oldversion.chatcompletionchunkdelta/#deconstructstring-string","title":"Deconstruct(String&, String&)","text":"public void Deconstruct(String& Role, String& Content)\nRole String&
Content String&
Namespace: LLama.OldVersion
public class ChatCompletionMessage : System.IEquatable`1[[LLama.OldVersion.ChatCompletionMessage, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]\nInheritance Object \u2192 ChatCompletionMessage Implements IEquatable<ChatCompletionMessage>
"},{"location":"xmldocs/llama.oldversion.chatcompletionmessage/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.chatcompletionmessage/#role","title":"Role","text":"public ChatRole Role { get; set; }\nChatRole
"},{"location":"xmldocs/llama.oldversion.chatcompletionmessage/#content","title":"Content","text":"public string Content { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletionmessage/#name","title":"Name","text":"public string Name { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletionmessage/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.chatcompletionmessage/#chatcompletionmessagechatrole-string-string","title":"ChatCompletionMessage(ChatRole, String, String)","text":"public ChatCompletionMessage(ChatRole Role, string Content, string Name)\nRole ChatRole
Content String
Name String
public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.chatcompletionmessage/#printmembersstringbuilder","title":"PrintMembers(StringBuilder)","text":"protected bool PrintMembers(StringBuilder builder)\nbuilder StringBuilder
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletionmessage/#gethashcode","title":"GetHashCode()","text":"public int GetHashCode()\nInt32
"},{"location":"xmldocs/llama.oldversion.chatcompletionmessage/#equalsobject","title":"Equals(Object)","text":"public bool Equals(object obj)\nobj Object
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletionmessage/#equalschatcompletionmessage","title":"Equals(ChatCompletionMessage)","text":"public bool Equals(ChatCompletionMessage other)\nother ChatCompletionMessage
Boolean
"},{"location":"xmldocs/llama.oldversion.chatcompletionmessage/#clone","title":"<Clone>$()","text":"public ChatCompletionMessage <Clone>$()\nChatCompletionMessage
"},{"location":"xmldocs/llama.oldversion.chatcompletionmessage/#deconstructchatrole-string-string","title":"Deconstruct(ChatRole&, String&, String&)","text":"public void Deconstruct(ChatRole& Role, String& Content, String& Name)\nRole ChatRole&
Content String&
Name String&
Namespace: LLama.OldVersion
public class ChatMessageRecord : System.IEquatable`1[[LLama.OldVersion.ChatMessageRecord, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]\nInheritance Object \u2192 ChatMessageRecord Implements IEquatable<ChatMessageRecord>
"},{"location":"xmldocs/llama.oldversion.chatmessagerecord/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.chatmessagerecord/#message","title":"Message","text":"public ChatCompletionMessage Message { get; set; }\nChatCompletionMessage
"},{"location":"xmldocs/llama.oldversion.chatmessagerecord/#time","title":"Time","text":"public DateTime Time { get; set; }\nDateTime
"},{"location":"xmldocs/llama.oldversion.chatmessagerecord/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.chatmessagerecord/#chatmessagerecordchatcompletionmessage-datetime","title":"ChatMessageRecord(ChatCompletionMessage, DateTime)","text":"public ChatMessageRecord(ChatCompletionMessage Message, DateTime Time)\nMessage ChatCompletionMessage
Time DateTime
public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.chatmessagerecord/#printmembersstringbuilder","title":"PrintMembers(StringBuilder)","text":"protected bool PrintMembers(StringBuilder builder)\nbuilder StringBuilder
Boolean
"},{"location":"xmldocs/llama.oldversion.chatmessagerecord/#gethashcode","title":"GetHashCode()","text":"public int GetHashCode()\nInt32
"},{"location":"xmldocs/llama.oldversion.chatmessagerecord/#equalsobject","title":"Equals(Object)","text":"public bool Equals(object obj)\nobj Object
Boolean
"},{"location":"xmldocs/llama.oldversion.chatmessagerecord/#equalschatmessagerecord","title":"Equals(ChatMessageRecord)","text":"public bool Equals(ChatMessageRecord other)\nother ChatMessageRecord
Boolean
"},{"location":"xmldocs/llama.oldversion.chatmessagerecord/#clone","title":"<Clone>$()","text":"public ChatMessageRecord <Clone>$()\nChatMessageRecord
"},{"location":"xmldocs/llama.oldversion.chatmessagerecord/#deconstructchatcompletionmessage-datetime","title":"Deconstruct(ChatCompletionMessage&, DateTime&)","text":"public void Deconstruct(ChatCompletionMessage& Message, DateTime& Time)\nMessage ChatCompletionMessage&
Time DateTime&
Namespace: LLama.OldVersion
public enum ChatRole\nInheritance Object \u2192 ValueType \u2192 Enum \u2192 ChatRole Implements IComparable, IFormattable, IConvertible
"},{"location":"xmldocs/llama.oldversion.chatrole/#fields","title":"Fields","text":"Name Value Description"},{"location":"xmldocs/llama.oldversion.chatsession-1/","title":"ChatSession<T>","text":"Namespace: LLama.OldVersion
public class ChatSession<T>\nT
Inheritance Object \u2192 ChatSession<T>
"},{"location":"xmldocs/llama.oldversion.chatsession-1/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.chatsession-1/#chatsessiont_1","title":"ChatSession(T)","text":"public ChatSession(T model)\nmodel T
public IEnumerable<string> Chat(string text, string prompt, string encoding)\ntext String
prompt String
encoding String
IEnumerable<String>
"},{"location":"xmldocs/llama.oldversion.chatsession-1/#withpromptstring-string","title":"WithPrompt(String, String)","text":"public ChatSession<T> WithPrompt(string prompt, string encoding)\nprompt String
encoding String
ChatSession<T>
"},{"location":"xmldocs/llama.oldversion.chatsession-1/#withpromptfilestring-string","title":"WithPromptFile(String, String)","text":"public ChatSession<T> WithPromptFile(string promptFilename, string encoding)\npromptFilename String
encoding String
ChatSession<T>
"},{"location":"xmldocs/llama.oldversion.chatsession-1/#withantipromptstring","title":"WithAntiprompt(String[])","text":"Set the keyword to split the return value of chat AI.
public ChatSession<T> WithAntiprompt(String[] antiprompt)\nantiprompt String[]
ChatSession<T>
"},{"location":"xmldocs/llama.oldversion.completion/","title":"Completion","text":"Namespace: LLama.OldVersion
public class Completion : System.IEquatable`1[[LLama.OldVersion.Completion, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]\nInheritance Object \u2192 Completion Implements IEquatable<Completion>
"},{"location":"xmldocs/llama.oldversion.completion/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.completion/#id","title":"Id","text":"public string Id { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.completion/#object","title":"Object","text":"public string Object { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.completion/#created","title":"Created","text":"public int Created { get; set; }\nInt32
"},{"location":"xmldocs/llama.oldversion.completion/#model","title":"Model","text":"public string Model { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.completion/#choices","title":"Choices","text":"public CompletionChoice[] Choices { get; set; }\nCompletionChoice[]
"},{"location":"xmldocs/llama.oldversion.completion/#usage","title":"Usage","text":"public CompletionUsage Usage { get; set; }\nCompletionUsage
"},{"location":"xmldocs/llama.oldversion.completion/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.completion/#completionstring-string-int32-string-completionchoice-completionusage","title":"Completion(String, String, Int32, String, CompletionChoice[], CompletionUsage)","text":"public Completion(string Id, string Object, int Created, string Model, CompletionChoice[] Choices, CompletionUsage Usage)\nId String
Object String
Created Int32
Model String
Choices CompletionChoice[]
Usage CompletionUsage
public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.completion/#printmembersstringbuilder","title":"PrintMembers(StringBuilder)","text":"protected bool PrintMembers(StringBuilder builder)\nbuilder StringBuilder
Boolean
"},{"location":"xmldocs/llama.oldversion.completion/#gethashcode","title":"GetHashCode()","text":"public int GetHashCode()\nInt32
"},{"location":"xmldocs/llama.oldversion.completion/#equalsobject","title":"Equals(Object)","text":"public bool Equals(object obj)\nobj Object
Boolean
"},{"location":"xmldocs/llama.oldversion.completion/#equalscompletion","title":"Equals(Completion)","text":"public bool Equals(Completion other)\nother Completion
Boolean
"},{"location":"xmldocs/llama.oldversion.completion/#clone","title":"<Clone>$()","text":"public Completion <Clone>$()\nCompletion
"},{"location":"xmldocs/llama.oldversion.completion/#deconstructstring-string-int32-string-completionchoice-completionusage","title":"Deconstruct(String&, String&, Int32&, String&, CompletionChoice[]&, CompletionUsage&)","text":"public void Deconstruct(String& Id, String& Object, Int32& Created, String& Model, CompletionChoice[]& Choices, CompletionUsage& Usage)\nId String&
Object String&
Created Int32&
Model String&
Choices CompletionChoice[]&
Usage CompletionUsage&
Namespace: LLama.OldVersion
public class CompletionChoice : System.IEquatable`1[[LLama.OldVersion.CompletionChoice, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]\nInheritance Object \u2192 CompletionChoice Implements IEquatable<CompletionChoice>
"},{"location":"xmldocs/llama.oldversion.completionchoice/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.completionchoice/#text","title":"Text","text":"public string Text { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.completionchoice/#index","title":"Index","text":"public int Index { get; set; }\nInt32
"},{"location":"xmldocs/llama.oldversion.completionchoice/#logprobs","title":"Logprobs","text":"public CompletionLogprobs Logprobs { get; set; }\nCompletionLogprobs
"},{"location":"xmldocs/llama.oldversion.completionchoice/#finishreason","title":"FinishReason","text":"public string FinishReason { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.completionchoice/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.completionchoice/#completionchoicestring-int32-completionlogprobs-string","title":"CompletionChoice(String, Int32, CompletionLogprobs, String)","text":"public CompletionChoice(string Text, int Index, CompletionLogprobs Logprobs, string FinishReason)\nText String
Index Int32
Logprobs CompletionLogprobs
FinishReason String
public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.completionchoice/#printmembersstringbuilder","title":"PrintMembers(StringBuilder)","text":"protected bool PrintMembers(StringBuilder builder)\nbuilder StringBuilder
Boolean
"},{"location":"xmldocs/llama.oldversion.completionchoice/#gethashcode","title":"GetHashCode()","text":"public int GetHashCode()\nInt32
"},{"location":"xmldocs/llama.oldversion.completionchoice/#equalsobject","title":"Equals(Object)","text":"public bool Equals(object obj)\nobj Object
Boolean
"},{"location":"xmldocs/llama.oldversion.completionchoice/#equalscompletionchoice","title":"Equals(CompletionChoice)","text":"public bool Equals(CompletionChoice other)\nother CompletionChoice
Boolean
"},{"location":"xmldocs/llama.oldversion.completionchoice/#clone","title":"<Clone>$()","text":"public CompletionChoice <Clone>$()\nCompletionChoice
"},{"location":"xmldocs/llama.oldversion.completionchoice/#deconstructstring-int32-completionlogprobs-string","title":"Deconstruct(String&, Int32&, CompletionLogprobs&, String&)","text":"public void Deconstruct(String& Text, Int32& Index, CompletionLogprobs& Logprobs, String& FinishReason)\nText String&
Index Int32&
Logprobs CompletionLogprobs&
FinishReason String&
Namespace: LLama.OldVersion
public class CompletionChunk : System.IEquatable`1[[LLama.OldVersion.CompletionChunk, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]\nInheritance Object \u2192 CompletionChunk Implements IEquatable<CompletionChunk>
"},{"location":"xmldocs/llama.oldversion.completionchunk/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.completionchunk/#id","title":"Id","text":"public string Id { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.completionchunk/#object","title":"Object","text":"public string Object { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.completionchunk/#created","title":"Created","text":"public int Created { get; set; }\nInt32
"},{"location":"xmldocs/llama.oldversion.completionchunk/#model","title":"Model","text":"public string Model { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.completionchunk/#choices","title":"Choices","text":"public CompletionChoice[] Choices { get; set; }\nCompletionChoice[]
"},{"location":"xmldocs/llama.oldversion.completionchunk/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.completionchunk/#completionchunkstring-string-int32-string-completionchoice","title":"CompletionChunk(String, String, Int32, String, CompletionChoice[])","text":"public CompletionChunk(string Id, string Object, int Created, string Model, CompletionChoice[] Choices)\nId String
Object String
Created Int32
Model String
Choices CompletionChoice[]
public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.completionchunk/#printmembersstringbuilder","title":"PrintMembers(StringBuilder)","text":"protected bool PrintMembers(StringBuilder builder)\nbuilder StringBuilder
Boolean
"},{"location":"xmldocs/llama.oldversion.completionchunk/#gethashcode","title":"GetHashCode()","text":"public int GetHashCode()\nInt32
"},{"location":"xmldocs/llama.oldversion.completionchunk/#equalsobject","title":"Equals(Object)","text":"public bool Equals(object obj)\nobj Object
Boolean
"},{"location":"xmldocs/llama.oldversion.completionchunk/#equalscompletionchunk","title":"Equals(CompletionChunk)","text":"public bool Equals(CompletionChunk other)\nother CompletionChunk
Boolean
"},{"location":"xmldocs/llama.oldversion.completionchunk/#clone","title":"<Clone>$()","text":"public CompletionChunk <Clone>$()\nCompletionChunk
"},{"location":"xmldocs/llama.oldversion.completionchunk/#deconstructstring-string-int32-string-completionchoice","title":"Deconstruct(String&, String&, Int32&, String&, CompletionChoice[]&)","text":"public void Deconstruct(String& Id, String& Object, Int32& Created, String& Model, CompletionChoice[]& Choices)\nId String&
Object String&
Created Int32&
Model String&
Choices CompletionChoice[]&
Namespace: LLama.OldVersion
public class CompletionLogprobs : System.IEquatable`1[[LLama.OldVersion.CompletionLogprobs, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]\nInheritance Object \u2192 CompletionLogprobs Implements IEquatable<CompletionLogprobs>
"},{"location":"xmldocs/llama.oldversion.completionlogprobs/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.completionlogprobs/#textoffset","title":"TextOffset","text":"public Int32[] TextOffset { get; set; }\nInt32[]
"},{"location":"xmldocs/llama.oldversion.completionlogprobs/#tokenlogprobs","title":"TokenLogProbs","text":"public Single[] TokenLogProbs { get; set; }\nSingle[]
"},{"location":"xmldocs/llama.oldversion.completionlogprobs/#tokens","title":"Tokens","text":"public String[] Tokens { get; set; }\nString[]
"},{"location":"xmldocs/llama.oldversion.completionlogprobs/#toplogprobs","title":"TopLogprobs","text":"public Dictionary`2[] TopLogprobs { get; set; }\nDictionary`2[]
"},{"location":"xmldocs/llama.oldversion.completionlogprobs/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.completionlogprobs/#completionlogprobsint32-single-string-dictionary2","title":"CompletionLogprobs(Int32[], Single[], String[], Dictionary`2[])","text":"public CompletionLogprobs(Int32[] TextOffset, Single[] TokenLogProbs, String[] Tokens, Dictionary`2[] TopLogprobs)\nTextOffset Int32[]
TokenLogProbs Single[]
Tokens String[]
TopLogprobs Dictionary`2[]
public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.completionlogprobs/#printmembersstringbuilder","title":"PrintMembers(StringBuilder)","text":"protected bool PrintMembers(StringBuilder builder)\nbuilder StringBuilder
Boolean
"},{"location":"xmldocs/llama.oldversion.completionlogprobs/#gethashcode","title":"GetHashCode()","text":"public int GetHashCode()\nInt32
"},{"location":"xmldocs/llama.oldversion.completionlogprobs/#equalsobject","title":"Equals(Object)","text":"public bool Equals(object obj)\nobj Object
Boolean
"},{"location":"xmldocs/llama.oldversion.completionlogprobs/#equalscompletionlogprobs","title":"Equals(CompletionLogprobs)","text":"public bool Equals(CompletionLogprobs other)\nother CompletionLogprobs
Boolean
"},{"location":"xmldocs/llama.oldversion.completionlogprobs/#clone","title":"<Clone>$()","text":"public CompletionLogprobs <Clone>$()\nCompletionLogprobs
"},{"location":"xmldocs/llama.oldversion.completionlogprobs/#deconstructint32-single-string-dictionary2","title":"Deconstruct(Int32[]&, Single[]&, String[]&, Dictionary`2[]&)","text":"public void Deconstruct(Int32[]& TextOffset, Single[]& TokenLogProbs, String[]& Tokens, Dictionary`2[]& TopLogprobs)\nTextOffset Int32[]&
TokenLogProbs Single[]&
Tokens String[]&
TopLogprobs Dictionary`2[]&
Namespace: LLama.OldVersion
public class CompletionUsage : System.IEquatable`1[[LLama.OldVersion.CompletionUsage, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]\nInheritance Object \u2192 CompletionUsage Implements IEquatable<CompletionUsage>
"},{"location":"xmldocs/llama.oldversion.completionusage/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.completionusage/#prompttokens","title":"PromptTokens","text":"public int PromptTokens { get; set; }\nInt32
"},{"location":"xmldocs/llama.oldversion.completionusage/#completiontokens","title":"CompletionTokens","text":"public int CompletionTokens { get; set; }\nInt32
"},{"location":"xmldocs/llama.oldversion.completionusage/#totaltokens","title":"TotalTokens","text":"public int TotalTokens { get; set; }\nInt32
"},{"location":"xmldocs/llama.oldversion.completionusage/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.completionusage/#completionusageint32-int32-int32","title":"CompletionUsage(Int32, Int32, Int32)","text":"public CompletionUsage(int PromptTokens, int CompletionTokens, int TotalTokens)\nPromptTokens Int32
CompletionTokens Int32
TotalTokens Int32
public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.completionusage/#printmembersstringbuilder","title":"PrintMembers(StringBuilder)","text":"protected bool PrintMembers(StringBuilder builder)\nbuilder StringBuilder
Boolean
"},{"location":"xmldocs/llama.oldversion.completionusage/#gethashcode","title":"GetHashCode()","text":"public int GetHashCode()\nInt32
"},{"location":"xmldocs/llama.oldversion.completionusage/#equalsobject","title":"Equals(Object)","text":"public bool Equals(object obj)\nobj Object
Boolean
"},{"location":"xmldocs/llama.oldversion.completionusage/#equalscompletionusage","title":"Equals(CompletionUsage)","text":"public bool Equals(CompletionUsage other)\nother CompletionUsage
Boolean
"},{"location":"xmldocs/llama.oldversion.completionusage/#clone","title":"<Clone>$()","text":"public CompletionUsage <Clone>$()\nCompletionUsage
"},{"location":"xmldocs/llama.oldversion.completionusage/#deconstructint32-int32-int32","title":"Deconstruct(Int32&, Int32&, Int32&)","text":"public void Deconstruct(Int32& PromptTokens, Int32& CompletionTokens, Int32& TotalTokens)\nPromptTokens Int32&
CompletionTokens Int32&
TotalTokens Int32&
Namespace: LLama.OldVersion
public class Embedding : System.IEquatable`1[[LLama.OldVersion.Embedding, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]\nInheritance Object \u2192 Embedding Implements IEquatable<Embedding>
"},{"location":"xmldocs/llama.oldversion.embedding/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.embedding/#object","title":"Object","text":"public string Object { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.embedding/#model","title":"Model","text":"public string Model { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.embedding/#data","title":"Data","text":"public EmbeddingData[] Data { get; set; }\nEmbeddingData[]
"},{"location":"xmldocs/llama.oldversion.embedding/#usage","title":"Usage","text":"public EmbeddingUsage Usage { get; set; }\nEmbeddingUsage
"},{"location":"xmldocs/llama.oldversion.embedding/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.embedding/#embeddingstring-string-embeddingdata-embeddingusage","title":"Embedding(String, String, EmbeddingData[], EmbeddingUsage)","text":"public Embedding(string Object, string Model, EmbeddingData[] Data, EmbeddingUsage Usage)\nObject String
Model String
Data EmbeddingData[]
Usage EmbeddingUsage
public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.embedding/#printmembersstringbuilder","title":"PrintMembers(StringBuilder)","text":"protected bool PrintMembers(StringBuilder builder)\nbuilder StringBuilder
Boolean
"},{"location":"xmldocs/llama.oldversion.embedding/#gethashcode","title":"GetHashCode()","text":"public int GetHashCode()\nInt32
"},{"location":"xmldocs/llama.oldversion.embedding/#equalsobject","title":"Equals(Object)","text":"public bool Equals(object obj)\nobj Object
Boolean
"},{"location":"xmldocs/llama.oldversion.embedding/#equalsembedding","title":"Equals(Embedding)","text":"public bool Equals(Embedding other)\nother Embedding
Boolean
"},{"location":"xmldocs/llama.oldversion.embedding/#clone","title":"<Clone>$()","text":"public Embedding <Clone>$()\nEmbedding
"},{"location":"xmldocs/llama.oldversion.embedding/#deconstructstring-string-embeddingdata-embeddingusage","title":"Deconstruct(String&, String&, EmbeddingData[]&, EmbeddingUsage&)","text":"public void Deconstruct(String& Object, String& Model, EmbeddingData[]& Data, EmbeddingUsage& Usage)\nObject String&
Model String&
Data EmbeddingData[]&
Usage EmbeddingUsage&
Namespace: LLama.OldVersion
public class EmbeddingData : System.IEquatable`1[[LLama.OldVersion.EmbeddingData, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]\nInheritance Object \u2192 EmbeddingData Implements IEquatable<EmbeddingData>
"},{"location":"xmldocs/llama.oldversion.embeddingdata/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.embeddingdata/#index","title":"Index","text":"public int Index { get; set; }\nInt32
"},{"location":"xmldocs/llama.oldversion.embeddingdata/#object","title":"Object","text":"public string Object { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.embeddingdata/#embedding","title":"Embedding","text":"public Single[] Embedding { get; set; }\nSingle[]
"},{"location":"xmldocs/llama.oldversion.embeddingdata/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.embeddingdata/#embeddingdataint32-string-single","title":"EmbeddingData(Int32, String, Single[])","text":"public EmbeddingData(int Index, string Object, Single[] Embedding)\nIndex Int32
Object String
Embedding Single[]
public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.embeddingdata/#printmembersstringbuilder","title":"PrintMembers(StringBuilder)","text":"protected bool PrintMembers(StringBuilder builder)\nbuilder StringBuilder
Boolean
"},{"location":"xmldocs/llama.oldversion.embeddingdata/#gethashcode","title":"GetHashCode()","text":"public int GetHashCode()\nInt32
"},{"location":"xmldocs/llama.oldversion.embeddingdata/#equalsobject","title":"Equals(Object)","text":"public bool Equals(object obj)\nobj Object
Boolean
"},{"location":"xmldocs/llama.oldversion.embeddingdata/#equalsembeddingdata","title":"Equals(EmbeddingData)","text":"public bool Equals(EmbeddingData other)\nother EmbeddingData
Boolean
"},{"location":"xmldocs/llama.oldversion.embeddingdata/#clone","title":"<Clone>$()","text":"public EmbeddingData <Clone>$()\nEmbeddingData
"},{"location":"xmldocs/llama.oldversion.embeddingdata/#deconstructint32-string-single","title":"Deconstruct(Int32&, String&, Single[]&)","text":"public void Deconstruct(Int32& Index, String& Object, Single[]& Embedding)\nIndex Int32&
Object String&
Embedding Single[]&
Namespace: LLama.OldVersion
public class EmbeddingUsage : System.IEquatable`1[[LLama.OldVersion.EmbeddingUsage, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]\nInheritance Object \u2192 EmbeddingUsage Implements IEquatable<EmbeddingUsage>
"},{"location":"xmldocs/llama.oldversion.embeddingusage/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.embeddingusage/#prompttokens","title":"PromptTokens","text":"public int PromptTokens { get; set; }\nInt32
"},{"location":"xmldocs/llama.oldversion.embeddingusage/#totaltokens","title":"TotalTokens","text":"public int TotalTokens { get; set; }\nInt32
"},{"location":"xmldocs/llama.oldversion.embeddingusage/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.embeddingusage/#embeddingusageint32-int32","title":"EmbeddingUsage(Int32, Int32)","text":"public EmbeddingUsage(int PromptTokens, int TotalTokens)\nPromptTokens Int32
TotalTokens Int32
public string ToString()\nString
"},{"location":"xmldocs/llama.oldversion.embeddingusage/#printmembersstringbuilder","title":"PrintMembers(StringBuilder)","text":"protected bool PrintMembers(StringBuilder builder)\nbuilder StringBuilder
Boolean
"},{"location":"xmldocs/llama.oldversion.embeddingusage/#gethashcode","title":"GetHashCode()","text":"public int GetHashCode()\nInt32
"},{"location":"xmldocs/llama.oldversion.embeddingusage/#equalsobject","title":"Equals(Object)","text":"public bool Equals(object obj)\nobj Object
Boolean
"},{"location":"xmldocs/llama.oldversion.embeddingusage/#equalsembeddingusage","title":"Equals(EmbeddingUsage)","text":"public bool Equals(EmbeddingUsage other)\nother EmbeddingUsage
Boolean
"},{"location":"xmldocs/llama.oldversion.embeddingusage/#clone","title":"<Clone>$()","text":"public EmbeddingUsage <Clone>$()\nEmbeddingUsage
"},{"location":"xmldocs/llama.oldversion.embeddingusage/#deconstructint32-int32","title":"Deconstruct(Int32&, Int32&)","text":"public void Deconstruct(Int32& PromptTokens, Int32& TotalTokens)\nPromptTokens Int32&
TotalTokens Int32&
Namespace: LLama.OldVersion
public interface IChatModel\npublic abstract string Name { get; }\nString
"},{"location":"xmldocs/llama.oldversion.ichatmodel/#methods","title":"Methods","text":""},{"location":"xmldocs/llama.oldversion.ichatmodel/#chatstring-string-string","title":"Chat(String, String, String)","text":"IEnumerable<string> Chat(string text, string prompt, string encoding)\ntext String
prompt String
encoding String
IEnumerable<String>
"},{"location":"xmldocs/llama.oldversion.ichatmodel/#initchatpromptstring-string","title":"InitChatPrompt(String, String)","text":"Init a prompt for chat and automatically produce the next prompt during the chat.
void InitChatPrompt(string prompt, string encoding)\nprompt String
encoding String
void InitChatAntiprompt(String[] antiprompt)\nantiprompt String[]
Namespace: LLama.OldVersion
public class LLamaEmbedder : System.IDisposable\nInheritance Object \u2192 LLamaEmbedder Implements IDisposable
"},{"location":"xmldocs/llama.oldversion.llamaembedder/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.llamaembedder/#llamaembedderllamaparams","title":"LLamaEmbedder(LLamaParams)","text":"public LLamaEmbedder(LLamaParams params)\nparams LLamaParams
public Single[] GetEmbeddings(string text, int n_thread, bool add_bos, string encoding)\ntext String
n_thread Int32
add_bos Boolean
encoding String
Single[]
"},{"location":"xmldocs/llama.oldversion.llamaembedder/#dispose","title":"Dispose()","text":"public void Dispose()\nNamespace: LLama.OldVersion
public class LLamaModel : IChatModel, System.IDisposable\nInheritance Object \u2192 LLamaModel Implements IChatModel, IDisposable
"},{"location":"xmldocs/llama.oldversion.llamamodel/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.oldversion.llamamodel/#name","title":"Name","text":"public string Name { get; set; }\nString
"},{"location":"xmldocs/llama.oldversion.llamamodel/#verbose","title":"Verbose","text":"public bool Verbose { get; set; }\nBoolean
"},{"location":"xmldocs/llama.oldversion.llamamodel/#nativehandle","title":"NativeHandle","text":"public SafeLLamaContextHandle NativeHandle { get; }\nSafeLLamaContextHandle
"},{"location":"xmldocs/llama.oldversion.llamamodel/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.oldversion.llamamodel/#llamamodelstring-string-boolean-int32-int32-int32-int32-int32-int32-int32-dictionaryint32-single-int32-single-single-single-single-single-int32-single-single-int32-single-single-string-string-string-string-liststring-string-string-boolean-boolean-boolean-boolean-boolean-boolean-boolean-boolean-boolean-boolean-boolean-boolean-boolean-boolean-string","title":"LLamaModel(String, String, Boolean, Int32, Int32, Int32, Int32, Int32, Int32, Int32, Dictionary<Int32, Single>, Int32, Single, Single, Single, Single, Single, Int32, Single, Single, Int32, Single, Single, String, String, String, String, List<String>, String, String, Boolean, Boolean, Boolean, Boolean, Boolean, Boolean, Boolean, Boolean, Boolean, Boolean, Boolean, Boolean, Boolean, Boolean, String)","text":"Please refer LLamaParams to find the meanings of each arg. Be sure to have set the n_gpu_layers, otherwise it will load 20 layers to gpu by default.
public LLamaModel(string model_path, string model_name, bool verbose, int seed, int n_threads, int n_predict, int n_ctx, int n_batch, int n_keep, int n_gpu_layers, Dictionary<int, float> logit_bias, int top_k, float top_p, float tfs_z, float typical_p, float temp, float repeat_penalty, int repeat_last_n, float frequency_penalty, float presence_penalty, int mirostat, float mirostat_tau, float mirostat_eta, string prompt, string path_session, string input_prefix, string input_suffix, List<string> antiprompt, string lora_adapter, string lora_base, bool memory_f16, bool random_prompt, bool use_color, bool interactive, bool embedding, bool interactive_first, bool prompt_cache_all, bool instruct, bool penalize_nl, bool perplexity, bool use_mmap, bool use_mlock, bool mem_test, bool verbose_prompt, string encoding)\nmodel_path String The model file path.
model_name String The model name.
verbose Boolean Whether to print details when running the model.
seed Int32
n_threads Int32
n_predict Int32
n_ctx Int32
n_batch Int32
n_keep Int32
n_gpu_layers Int32
logit_bias Dictionary<Int32, Single>
top_k Int32
top_p Single
tfs_z Single
typical_p Single
temp Single
repeat_penalty Single
repeat_last_n Int32
frequency_penalty Single
presence_penalty Single
mirostat Int32
mirostat_tau Single
mirostat_eta Single
prompt String
path_session String
input_prefix String
input_suffix String
antiprompt List<String>
lora_adapter String
lora_base String
memory_f16 Boolean
random_prompt Boolean
use_color Boolean
interactive Boolean
embedding Boolean
interactive_first Boolean
prompt_cache_all Boolean
instruct Boolean
penalize_nl Boolean
perplexity Boolean
use_mmap Boolean
use_mlock Boolean
mem_test Boolean
verbose_prompt Boolean
encoding String
Please refer LLamaParams to find the meanings of each arg. Be sure to have set the n_gpu_layers, otherwise it will load 20 layers to gpu by default.
public LLamaModel(LLamaParams params, string name, bool verbose, string encoding)\nparams LLamaParams The LLamaModel params
name String Model name
verbose Boolean Whether to output the detailed info.
encoding String
RuntimeError
"},{"location":"xmldocs/llama.oldversion.llamamodel/#methods","title":"Methods","text":""},{"location":"xmldocs/llama.oldversion.llamamodel/#withpromptstring-string","title":"WithPrompt(String, String)","text":"Apply a prompt to the model.
public LLamaModel WithPrompt(string prompt, string encoding)\nprompt String
encoding String
LLamaModel
"},{"location":"xmldocs/llama.oldversion.llamamodel/#exceptions_1","title":"Exceptions","text":"ArgumentException
"},{"location":"xmldocs/llama.oldversion.llamamodel/#withpromptfilestring","title":"WithPromptFile(String)","text":"Apply the prompt file to the model.
public LLamaModel WithPromptFile(string promptFileName)\npromptFileName String
LLamaModel
"},{"location":"xmldocs/llama.oldversion.llamamodel/#initchatpromptstring-string","title":"InitChatPrompt(String, String)","text":"public void InitChatPrompt(string prompt, string encoding)\nprompt String
encoding String
public void InitChatAntiprompt(String[] antiprompt)\nantiprompt String[]
Chat with the LLaMa model under interactive mode.
public IEnumerable<string> Chat(string text, string prompt, string encoding)\ntext String
prompt String
encoding String
IEnumerable<String>
"},{"location":"xmldocs/llama.oldversion.llamamodel/#exceptions_2","title":"Exceptions","text":"ArgumentException
"},{"location":"xmldocs/llama.oldversion.llamamodel/#savestatestring","title":"SaveState(String)","text":"Save the state to specified path.
public void SaveState(string filename)\nfilename String
Load the state from specified path.
public void LoadState(string filename, bool clearPreviousEmbed)\nfilename String
clearPreviousEmbed Boolean Whether to clear previous footprints of this model.
RuntimeError
"},{"location":"xmldocs/llama.oldversion.llamamodel/#tokenizestring-string","title":"Tokenize(String, String)","text":"Tokenize a string.
public List<int> Tokenize(string text, string encoding)\ntext String The utf-8 encoded string to tokenize.
encoding String
List<Int32> A list of tokens.
"},{"location":"xmldocs/llama.oldversion.llamamodel/#exceptions_4","title":"Exceptions","text":"RuntimeError If the tokenization failed.
"},{"location":"xmldocs/llama.oldversion.llamamodel/#detokenizeienumerableint32","title":"DeTokenize(IEnumerable<Int32>)","text":"Detokenize a list of tokens.
public string DeTokenize(IEnumerable<int> tokens)\ntokens IEnumerable<Int32> The list of tokens to detokenize.
String The detokenized string.
"},{"location":"xmldocs/llama.oldversion.llamamodel/#callstring-string","title":"Call(String, String)","text":"Call the model to run inference.
public IEnumerable<string> Call(string text, string encoding)\ntext String
encoding String
IEnumerable<String>
"},{"location":"xmldocs/llama.oldversion.llamamodel/#exceptions_5","title":"Exceptions","text":"RuntimeError
"},{"location":"xmldocs/llama.oldversion.llamamodel/#dispose","title":"Dispose()","text":"public void Dispose()\nNamespace: LLama.OldVersion
public struct LLamaParams\nInheritance Object \u2192 ValueType \u2192 LLamaParams
"},{"location":"xmldocs/llama.oldversion.llamaparams/#fields","title":"Fields","text":""},{"location":"xmldocs/llama.oldversion.llamaparams/#seed","title":"seed","text":"public int seed;\npublic int n_threads;\npublic int n_predict;\npublic int n_ctx;\npublic int n_batch;\npublic int n_keep;\npublic int n_gpu_layers;\npublic Dictionary<int, float> logit_bias;\npublic int top_k;\npublic float top_p;\npublic float tfs_z;\npublic float typical_p;\npublic float temp;\npublic float repeat_penalty;\npublic int repeat_last_n;\npublic float frequency_penalty;\npublic float presence_penalty;\npublic int mirostat;\npublic float mirostat_tau;\npublic float mirostat_eta;\npublic string model;\npublic string prompt;\npublic string path_session;\npublic string input_prefix;\npublic string input_suffix;\npublic List<string> antiprompt;\npublic string lora_adapter;\npublic string lora_base;\npublic bool memory_f16;\npublic bool random_prompt;\npublic bool use_color;\npublic bool interactive;\npublic bool prompt_cache_all;\npublic bool embedding;\npublic bool interactive_first;\npublic bool instruct;\npublic bool penalize_nl;\npublic bool perplexity;\npublic bool use_mmap;\npublic bool use_mlock;\npublic bool mem_test;\npublic bool verbose_prompt;\nLLamaParams(int seed, int n_threads, int n_predict, int n_ctx, int n_batch, int n_keep, int n_gpu_layers, Dictionary<int, float> logit_bias, int top_k, float top_p, float tfs_z, float typical_p, float temp, float repeat_penalty, int repeat_last_n, float frequency_penalty, float presence_penalty, int mirostat, float mirostat_tau, float mirostat_eta, string model, string prompt, string path_session, string input_prefix, string input_suffix, List<string> antiprompt, string lora_adapter, string lora_base, bool memory_f16, bool random_prompt, bool use_color, bool interactive, bool prompt_cache_all, bool embedding, bool interactive_first, bool instruct, bool penalize_nl, bool perplexity, bool use_mmap, bool use_mlock, bool mem_test, bool verbose_prompt)\nseed Int32
n_threads Int32
n_predict Int32
n_ctx Int32
n_batch Int32
n_keep Int32
n_gpu_layers Int32
logit_bias Dictionary<Int32, Single>
top_k Int32
top_p Single
tfs_z Single
typical_p Single
temp Single
repeat_penalty Single
repeat_last_n Int32
frequency_penalty Single
presence_penalty Single
mirostat Int32
mirostat_tau Single
mirostat_eta Single
model String
prompt String
path_session String
input_prefix String
input_suffix String
antiprompt List<String>
lora_adapter String
lora_base String
memory_f16 Boolean
random_prompt Boolean
use_color Boolean
interactive Boolean
prompt_cache_all Boolean
embedding Boolean
interactive_first Boolean
instruct Boolean
penalize_nl Boolean
perplexity Boolean
use_mmap Boolean
use_mlock Boolean
mem_test Boolean
verbose_prompt Boolean
Namespace: LLama
A LLamaModel what could be reset. Note that using this class will consume about 10% more memories.
public class ResettableLLamaModel : LLamaModel, System.IDisposable\nInheritance Object \u2192 LLamaModel \u2192 ResettableLLamaModel Implements IDisposable
"},{"location":"xmldocs/llama.resettablellamamodel/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.resettablellamamodel/#originalstate","title":"OriginalState","text":"The initial state of the model
public Byte[] OriginalState { get; set; }\nByte[]
"},{"location":"xmldocs/llama.resettablellamamodel/#contextsize","title":"ContextSize","text":"The context size.
public int ContextSize { get; }\nInt32
"},{"location":"xmldocs/llama.resettablellamamodel/#params","title":"Params","text":"The model params set for this model.
public ModelParams Params { get; set; }\nModelParams
"},{"location":"xmldocs/llama.resettablellamamodel/#nativehandle","title":"NativeHandle","text":"The native handle, which is used to be passed to the native APIs. Please avoid using it unless you know what is the usage of the Native API.
public SafeLLamaContextHandle NativeHandle { get; }\nSafeLLamaContextHandle
"},{"location":"xmldocs/llama.resettablellamamodel/#encoding","title":"Encoding","text":"The encoding set for this model to deal with text input.
public Encoding Encoding { get; }\nEncoding
"},{"location":"xmldocs/llama.resettablellamamodel/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.resettablellamamodel/#resettablellamamodelmodelparams-string","title":"ResettableLLamaModel(ModelParams, String)","text":"public ResettableLLamaModel(ModelParams Params, string encoding)\nParams ModelParams
encoding String
Reset the state to the initial state.
public void Reset()\nNamespace: LLama
The base class for stateful LLama executors.
public abstract class StatefulExecutorBase : LLama.Abstractions.ILLamaExecutor\nInheritance Object \u2192 StatefulExecutorBase Implements ILLamaExecutor
"},{"location":"xmldocs/llama.statefulexecutorbase/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.statefulexecutorbase/#model","title":"Model","text":"The mode used by the executor.
public LLamaModel Model { get; }\nLLamaModel
"},{"location":"xmldocs/llama.statefulexecutorbase/#methods","title":"Methods","text":""},{"location":"xmldocs/llama.statefulexecutorbase/#withsessionfilestring","title":"WithSessionFile(String)","text":"This API is currently not verified.
public StatefulExecutorBase WithSessionFile(string filename)\nfilename String
StatefulExecutorBase
"},{"location":"xmldocs/llama.statefulexecutorbase/#exceptions","title":"Exceptions","text":"ArgumentNullException
RuntimeError
"},{"location":"xmldocs/llama.statefulexecutorbase/#savesessionfilestring","title":"SaveSessionFile(String)","text":"This API has not been verified currently.
public void SaveSessionFile(string filename)\nfilename String
After running out of the context, take some tokens from the original prompt and recompute the logits in batches.
protected void HandleRunOutOfContext(int tokensToKeep)\ntokensToKeep Int32
Try to reuse the matching prefix from the session file.
protected void TryReuseMathingPrefix()\nDecide whether to continue the loop.
protected abstract bool GetLoopCondition(InferStateArgs args)\nargs InferStateArgs
Boolean
"},{"location":"xmldocs/llama.statefulexecutorbase/#preprocessinputsstring-inferstateargs","title":"PreprocessInputs(String, InferStateArgs)","text":"Preprocess the inputs before the inference.
protected abstract void PreprocessInputs(string text, InferStateArgs args)\ntext String
args InferStateArgs
Do some post processing after the inference.
protected abstract bool PostProcess(InferenceParams inferenceParams, InferStateArgs args, IEnumerable`1& extraOutputs)\ninferenceParams InferenceParams
args InferStateArgs
extraOutputs IEnumerable`1&
Boolean
"},{"location":"xmldocs/llama.statefulexecutorbase/#inferinternalinferenceparams-inferstateargs","title":"InferInternal(InferenceParams, InferStateArgs)","text":"The core inference logic.
protected abstract void InferInternal(InferenceParams inferenceParams, InferStateArgs args)\ninferenceParams InferenceParams
args InferStateArgs
Save the current state to a file.
public abstract void SaveState(string filename)\nfilename String
Get the current state data.
public abstract ExecutorBaseState GetStateData()\nExecutorBaseState
"},{"location":"xmldocs/llama.statefulexecutorbase/#loadstateexecutorbasestate","title":"LoadState(ExecutorBaseState)","text":"Load the state from data.
public abstract void LoadState(ExecutorBaseState data)\ndata ExecutorBaseState
Load the state from a file.
public abstract void LoadState(string filename)\nfilename String
Execute the inference.
public IEnumerable<string> Infer(string text, InferenceParams inferenceParams, CancellationToken cancellationToken)\ntext String
inferenceParams InferenceParams
cancellationToken CancellationToken
IEnumerable<String>
"},{"location":"xmldocs/llama.statefulexecutorbase/#inferasyncstring-inferenceparams-cancellationtoken","title":"InferAsync(String, InferenceParams, CancellationToken)","text":"Execute the inference asynchronously.
public IAsyncEnumerable<string> InferAsync(string text, InferenceParams inferenceParams, CancellationToken cancellationToken)\ntext String
inferenceParams InferenceParams
cancellationToken CancellationToken
IAsyncEnumerable<String>
"},{"location":"xmldocs/llama.statelessexecutor/","title":"StatelessExecutor","text":"Namespace: LLama
This executor infer the input as one-time job. Previous inputs won't impact on the response to current input.
public class StatelessExecutor : LLama.Abstractions.ILLamaExecutor\nInheritance Object \u2192 StatelessExecutor Implements ILLamaExecutor
"},{"location":"xmldocs/llama.statelessexecutor/#properties","title":"Properties","text":""},{"location":"xmldocs/llama.statelessexecutor/#model","title":"Model","text":"The mode used by the executor when running the inference.
public LLamaModel Model { get; }\nLLamaModel
"},{"location":"xmldocs/llama.statelessexecutor/#constructors","title":"Constructors","text":""},{"location":"xmldocs/llama.statelessexecutor/#statelessexecutorllamamodel","title":"StatelessExecutor(LLamaModel)","text":"public StatelessExecutor(LLamaModel model)\nmodel LLamaModel The LLama model.
public IEnumerable<string> Infer(string text, InferenceParams inferenceParams, CancellationToken cancellationToken)\ntext String
inferenceParams InferenceParams
cancellationToken CancellationToken
IEnumerable<String>
"},{"location":"xmldocs/llama.statelessexecutor/#inferasyncstring-inferenceparams-cancellationtoken","title":"InferAsync(String, InferenceParams, CancellationToken)","text":"public IAsyncEnumerable<string> InferAsync(string text, InferenceParams inferenceParams, CancellationToken token)\ntext String
inferenceParams InferenceParams
token CancellationToken
IAsyncEnumerable<String>
"}]} \ No newline at end of file diff --git a/site/sitemap.xml.gz b/site/sitemap.xml.gz new file mode 100644 index 00000000..393f1870 Binary files /dev/null and b/site/sitemap.xml.gz differ diff --git a/site/xmldocs/index.html b/site/xmldocs/index.html new file mode 100644 index 00000000..90641dea --- /dev/null +++ b/site/xmldocs/index.html @@ -0,0 +1,1747 @@ + + + + + + + + + + + + + + + + + + + + + +Namespace: LLama.Abstractions
+Transform history to plain text and vice versa.
+public interface IHistoryTransform
+Convert a ChatHistory instance to plain text.
+string HistoryToText(ChatHistory history)
+history ChatHistory
+The ChatHistory instance
Converts plain text to a ChatHistory instance.
+ChatHistory TextToHistory(AuthorRole role, string text)
+role AuthorRole
+The role for the author.
text String
+The chat history as plain text.
ChatHistory
+The updated history.
Namespace: LLama.Abstractions
+A high level interface for LLama models.
+public interface ILLamaExecutor
+The loaded model for this executor.
+public abstract LLamaModel Model { get; }
+Infers a response from the model.
+IEnumerable<string> Infer(string text, InferenceParams inferenceParams, CancellationToken token)
+text String
+Your prompt
inferenceParams InferenceParams
+Any additional parameters
token CancellationToken
+A cancellation token.
IAsyncEnumerable<string> InferAsync(string text, InferenceParams inferenceParams, CancellationToken token)
+text String
inferenceParams InferenceParams
token CancellationToken
Namespace: LLama.Abstractions
+Takes a stream of tokens and transforms them.
+public interface ITextStreamTransform
+Takes a stream of tokens and transforms them, returning a new stream of tokens.
+IEnumerable<string> Transform(IEnumerable<string> tokens)
+tokens IEnumerable<String>
Takes a stream of tokens and transforms them, returning a new stream of tokens asynchronously.
+IAsyncEnumerable<string> TransformAsync(IAsyncEnumerable<string> tokens)
+tokens IAsyncEnumerable<String>
Namespace: LLama.Abstractions
+An interface for text transformations. + These can be used to compose a pipeline of text transformations, such as: + - Tokenization + - Lowercasing + - Punctuation removal + - Trimming + - etc.
+public interface ITextTransform
+Takes a string and transforms it.
+string Transform(string text)
+text String
Namespace: LLama
+The main chat session class.
+public class ChatSession
+Inheritance Object → ChatSession
+The output transform used in this session.
+public ITextStreamTransform OutputTransform;
+The executor for this session.
+public ILLamaExecutor Executor { get; }
+The chat history for this session.
+public ChatHistory History { get; }
+The history transform used in this session.
+public IHistoryTransform HistoryTransform { get; set; }
+The input transform pipeline used in this session.
+public List<ITextTransform> InputTransformPipeline { get; set; }
+public ChatSession(ILLamaExecutor executor)
+executor ILLamaExecutor
+The executor for this session
Use a custom history transform.
+public ChatSession WithHistoryTransform(IHistoryTransform transform)
+transform IHistoryTransform
Add a text transform to the input transform pipeline.
+public ChatSession AddInputTransform(ITextTransform transform)
+transform ITextTransform
Use a custom output transform.
+public ChatSession WithOutputTransform(ITextStreamTransform transform)
+transform ITextStreamTransform
public void SaveSession(string path)
+path String
+The directory name to save the session. If the directory does not exist, a new directory will be created.
public void LoadSession(string path)
+path String
+The directory name to load the session.
Get the response from the LLama model with chat histories.
+public IEnumerable<string> Chat(ChatHistory history, InferenceParams inferenceParams, CancellationToken cancellationToken)
+history ChatHistory
inferenceParams InferenceParams
cancellationToken CancellationToken
Get the response from the LLama model. Note that prompt could not only be the preset words, + but also the question you want to ask.
+public IEnumerable<string> Chat(string prompt, InferenceParams inferenceParams, CancellationToken cancellationToken)
+prompt String
inferenceParams InferenceParams
cancellationToken CancellationToken
Get the response from the LLama model with chat histories.
+public IAsyncEnumerable<string> ChatAsync(ChatHistory history, InferenceParams inferenceParams, CancellationToken cancellationToken)
+history ChatHistory
inferenceParams InferenceParams
cancellationToken CancellationToken
Get the response from the LLama model with chat histories asynchronously.
+public IAsyncEnumerable<string> ChatAsync(string prompt, InferenceParams inferenceParams, CancellationToken cancellationToken)
+prompt String
inferenceParams InferenceParams
cancellationToken CancellationToken
Namespace: LLama.Common
+public enum AuthorRole
+Inheritance Object → ValueType → Enum → AuthorRole
+Implements IComparable, IFormattable, IConvertible
| Name | +Value | +Description | +
|---|---|---|
| + | + | + |
Namespace: LLama.Common
+The chat history class
+public class ChatHistory
+Inheritance Object → ChatHistory
+List of messages in the chat
+public List<Message> Messages { get; }
+Create a new instance of the chat content class
+public ChatHistory()
+Add a message to the chat history
+public void AddMessage(AuthorRole authorRole, string content)
+authorRole AuthorRole
+Role of the message author
content String
+Message content
Namespace: LLama.Common
+A queue with fixed storage size. + Currently it's only a naive implementation and needs to be further optimized in the future.
+public class FixedSizeQueue<T> : , System.Collections.IEnumerable
+T
Inheritance Object → FixedSizeQueue<T>
+Implements IEnumerable<T>, IEnumerable
public int Count { get; }
+public int Capacity { get; }
+public FixedSizeQueue(int size)
+size Int32
public FixedSizeQueue(int size, IEnumerable<T> data)
+size Int32
data IEnumerable<T>
public FixedSizeQueue<T> FillWith(T value)
+value T
Enquene an element.
+public void Enqueue(T item)
+item T
public T[] ToArray()
+T[]
public IEnumerator<T> GetEnumerator()
+IEnumerator<T>
Namespace: LLama.Common
+public interface ILLamaLogger
+Write the log in cosutomized way
+void Log(string source, string message, LogLevel level)
+source String
+The source of the log. It may be a method name or class name.
message String
+The message.
level LogLevel
+The log level.
Namespace: LLama.Common
+public class InferenceParams
+Inheritance Object → InferenceParams
+number of tokens to keep from initial prompt
+public int TokensKeep { get; set; }
+how many new tokens to predict (n_predict), set to -1 to inifinitely generate response + until it complete.
+public int MaxTokens { get; set; }
+logit bias for specific tokens
+public Dictionary<int, float> LogitBias { get; set; }
+Sequences where the model will stop generating further tokens.
+public IEnumerable<string> AntiPrompts { get; set; }
+path to file for saving/loading model eval state
+public string PathSession { get; set; }
+string to suffix user inputs with
+public string InputSuffix { get; set; }
+string to prefix user inputs with
+public string InputPrefix { get; set; }
+0 or lower to use vocab size
+public int TopK { get; set; }
+1.0 = disabled
+public float TopP { get; set; }
+1.0 = disabled
+public float TfsZ { get; set; }
+1.0 = disabled
+public float TypicalP { get; set; }
+1.0 = disabled
+public float Temperature { get; set; }
+1.0 = disabled
+public float RepeatPenalty { get; set; }
+last n tokens to penalize (0 = disable penalty, -1 = context size) (repeat_last_n)
+public int RepeatLastTokensCount { get; set; }
+frequency penalty coefficient + 0.0 = disabled
+public float FrequencyPenalty { get; set; }
+presence penalty coefficient + 0.0 = disabled
+public float PresencePenalty { get; set; }
+Mirostat uses tokens instead of words. + algorithm described in the paper https://arxiv.org/abs/2007.14966. + 0 = disabled, 1 = mirostat, 2 = mirostat 2.0
+public MiroStateType Mirostat { get; set; }
+target entropy
+public float MirostatTau { get; set; }
+learning rate
+public float MirostatEta { get; set; }
+consider newlines as a repeatable token (penalize_nl)
+public bool PenalizeNL { get; set; }
+public InferenceParams()
+Namespace: LLama.Common
+The default logger of LLamaSharp. On default it write to console. User methods of LLamaLogger.Default to change the behavior.
+ It's more recommended to inherit ILLamaLogger to cosutomize the behavior.
public sealed class LLamaDefaultLogger : ILLamaLogger
+Inheritance Object → LLamaDefaultLogger
+Implements ILLamaLogger
public static LLamaDefaultLogger Default { get; }
+public LLamaDefaultLogger EnableConsole()
+public LLamaDefaultLogger DisableConsole()
+public LLamaDefaultLogger EnableFile(string filename, FileMode mode)
+filename String
mode FileMode
public LLamaDefaultLogger DisableFile(string filename)
+filename String
public void Log(string source, string message, LogLevel level)
+source String
message String
level LogLevel
public void Info(string message)
+message String
public void Warn(string message)
+message String
public void Error(string message)
+message String
Namespace: LLama.Common
+public enum MiroStateType
+Inheritance Object → ValueType → Enum → MiroStateType
+Implements IComparable, IFormattable, IConvertible
| Name | +Value | +Description | +
|---|---|---|
| + | + | + |
Namespace: LLama.Common
+public class ModelParams
+Inheritance Object → ModelParams
+Model context size (n_ctx)
+public int ContextSize { get; set; }
+Number of layers to run in VRAM / GPU memory (n_gpu_layers)
+public int GpuLayerCount { get; set; }
+Seed for the random number generator (seed)
+public int Seed { get; set; }
+Use f16 instead of f32 for memory kv (memory_f16)
+public bool UseFp16Memory { get; set; }
+Use mmap for faster loads (use_mmap)
+public bool UseMemorymap { get; set; }
+Use mlock to keep model in memory (use_mlock)
+public bool UseMemoryLock { get; set; }
+Compute perplexity over the prompt (perplexity)
+public bool Perplexity { get; set; }
+Model path (model)
+public string ModelPath { get; set; }
+lora adapter path (lora_adapter)
+public string LoraAdapter { get; set; }
+base model path for the lora adapter (lora_base)
+public string LoraBase { get; set; }
+Number of threads (-1 = autodetect) (n_threads)
+public int Threads { get; set; }
+batch size for prompt processing (must be >=32 to use BLAS) (n_batch)
+public int BatchSize { get; set; }
+Whether to convert eos to newline during the inference.
+public bool ConvertEosToNewLine { get; set; }
+Whether to use embedding mode. (embedding) Note that if this is set to true, + The LLamaModel won't produce text response anymore.
+public bool EmbeddingMode { get; set; }
+public ModelParams(string modelPath, int contextSize, int gpuLayerCount, int seed, bool useFp16Memory, bool useMemorymap, bool useMemoryLock, bool perplexity, string loraAdapter, string loraBase, int threads, int batchSize, bool convertEosToNewLine, bool embeddingMode)
+modelPath String
+The model path.
contextSize Int32
+Model context size (n_ctx)
gpuLayerCount Int32
+Number of layers to run in VRAM / GPU memory (n_gpu_layers)
seed Int32
+Seed for the random number generator (seed)
useFp16Memory Boolean
+Whether to use f16 instead of f32 for memory kv (memory_f16)
useMemorymap Boolean
+Whether to use mmap for faster loads (use_mmap)
useMemoryLock Boolean
+Whether to use mlock to keep model in memory (use_mlock)
perplexity Boolean
+Thether to compute perplexity over the prompt (perplexity)
loraAdapter String
+Lora adapter path (lora_adapter)
loraBase String
+Base model path for the lora adapter (lora_base)
threads Int32
+Number of threads (-1 = autodetect) (n_threads)
batchSize Int32
+Batch size for prompt processing (must be >=32 to use BLAS) (n_batch)
convertEosToNewLine Boolean
+Whether to convert eos to newline during the inference.
embeddingMode Boolean
+Whether to use embedding mode. (embedding) Note that if this is set to true, The LLamaModel won't produce text response anymore.
Namespace: LLama.Exceptions
+public class RuntimeError : System.Exception, System.Runtime.Serialization.ISerializable
+Inheritance Object → Exception → RuntimeError
+Implements ISerializable
public MethodBase TargetSite { get; }
+public string Message { get; }
+public IDictionary Data { get; }
+public Exception InnerException { get; }
+public string HelpLink { get; set; }
+public string Source { get; set; }
+public int HResult { get; set; }
+public string StackTrace { get; }
+public RuntimeError()
+public RuntimeError(string message)
+message String
Namespace: LLama.Extensions
+public static class DictionaryExtension
+Inheritance Object → DictionaryExtension
+public static void Deconstruct<T1, T2>(KeyValuePair<T1, T2> pair, T1& first, T2& second)
+T1
T2
pair KeyValuePair<T1, T2>
first T1&
second T2&
public static void Update<T1, T2>(Dictionary<T1, T2> dic, IDictionary<T1, T2> other)
+T1
T2
dic Dictionary<T1, T2>
other IDictionary<T1, T2>
public static T2 GetOrDefault<T1, T2>(Dictionary<T1, T2> dic, T1 key, T2 defaultValue)
+T1
T2
dic Dictionary<T1, T2>
key T1
defaultValue T2
T2
Namespace: LLama
+The LLama executor for instruct mode.
+public class InstructExecutor : StatefulExecutorBase, LLama.Abstractions.ILLamaExecutor
+Inheritance Object → StatefulExecutorBase → InstructExecutor
+Implements ILLamaExecutor
The mode used by the executor.
+public LLamaModel Model { get; }
+public InstructExecutor(LLamaModel model, string instructionPrefix, string instructionSuffix)
+model LLamaModel
instructionPrefix String
instructionSuffix String
public ExecutorBaseState GetStateData()
+public void LoadState(ExecutorBaseState data)
+data ExecutorBaseState
public void SaveState(string filename)
+filename String
public void LoadState(string filename)
+filename String
protected bool GetLoopCondition(InferStateArgs args)
+args InferStateArgs
protected void PreprocessInputs(string text, InferStateArgs args)
+text String
args InferStateArgs
protected bool PostProcess(InferenceParams inferenceParams, InferStateArgs args, IEnumerable`1& extraOutputs)
+inferenceParams InferenceParams
args InferStateArgs
extraOutputs IEnumerable`1&
protected void InferInternal(InferenceParams inferenceParams, InferStateArgs args)
+inferenceParams InferenceParams
args InferStateArgs
Namespace: LLama
+The LLama executor for interactive mode.
+public class InteractiveExecutor : StatefulExecutorBase, LLama.Abstractions.ILLamaExecutor
+Inheritance Object → StatefulExecutorBase → InteractiveExecutor
+Implements ILLamaExecutor
The mode used by the executor.
+public LLamaModel Model { get; }
+public InteractiveExecutor(LLamaModel model)
+model LLamaModel
public ExecutorBaseState GetStateData()
+public void LoadState(ExecutorBaseState data)
+data ExecutorBaseState
public void SaveState(string filename)
+filename String
public void LoadState(string filename)
+filename String
Define whether to continue the loop to generate responses.
+protected bool GetLoopCondition(InferStateArgs args)
+args InferStateArgs
protected void PreprocessInputs(string text, InferStateArgs args)
+text String
args InferStateArgs
Return whether to break the generation.
+protected bool PostProcess(InferenceParams inferenceParams, InferStateArgs args, IEnumerable`1& extraOutputs)
+inferenceParams InferenceParams
args InferStateArgs
extraOutputs IEnumerable`1&
protected void InferInternal(InferenceParams inferenceParams, InferStateArgs args)
+inferenceParams InferenceParams
args InferStateArgs
Namespace: LLama
+The embedder for LLama, which supports getting embeddings from text.
+public class LLamaEmbedder : System.IDisposable
+Inheritance Object → LLamaEmbedder
+Implements IDisposable
public LLamaEmbedder(ModelParams params)
+params ModelParams
Get the embeddings of the text.
+public Single[] GetEmbeddings(string text, int threads, bool addBos, string encoding)
+text String
threads Int32
+Threads used for inference.
addBos Boolean
+Add bos to the text.
encoding String
public void Dispose()
+Namespace: LLama
+The abstraction of a LLama model, which holds the context in the native library.
+public class LLamaModel : System.IDisposable
+Inheritance Object → LLamaModel
+Implements IDisposable
The context size.
+public int ContextSize { get; }
+The model params set for this model.
+public ModelParams Params { get; set; }
+The native handle, which is used to be passed to the native APIs. Please avoid using it + unless you know what is the usage of the Native API.
+public SafeLLamaContextHandle NativeHandle { get; }
+The encoding set for this model to deal with text input.
+public Encoding Encoding { get; }
+public LLamaModel(ModelParams Params, string encoding, ILLamaLogger logger)
+Params ModelParams
+Model params.
encoding String
+Encoding to deal with text input.
logger ILLamaLogger
+The logger.
Tokenize a string.
+public IEnumerable<int> Tokenize(string text, bool addBos)
+text String
addBos Boolean
+Whether to add a bos to the text.
Detokenize the tokens to text.
+public string DeTokenize(IEnumerable<int> tokens)
+tokens IEnumerable<Int32>
Save the state to specified path.
+public void SaveState(string filename)
+filename String
Get the state data as a byte array.
+public Byte[] GetStateData()
+Load the state from specified path.
+public void LoadState(string filename)
+filename String
Load the state from memory.
+public void LoadState(Byte[] stateData)
+stateData Byte[]
Perform the sampling. Please don't use it unless you fully know what it does.
+public int Sample(LLamaTokenDataArray candidates, float temperature, MiroStateType mirostat, float mirostatTau, float mirostatEta, int topK, float topP, float tfsZ, float typicalP)
+candidates LLamaTokenDataArray
temperature Single
mirostat MiroStateType
mirostatTau Single
mirostatEta Single
topK Int32
topP Single
tfsZ Single
typicalP Single
Apply the penalty for the tokens. Please don't use it unless you fully know what it does.
+public LLamaTokenDataArray ApplyPenalty(IEnumerable<int> lastTokens, Dictionary<int, float> logitBias, int repeatLastTokensCount, float repeatPenalty, float alphaFrequency, float alphaPresence, bool penalizeNL)
+lastTokens IEnumerable<Int32>
logitBias Dictionary<Int32, Single>
repeatLastTokensCount Int32
repeatPenalty Single
alphaFrequency Single
alphaPresence Single
penalizeNL Boolean
public int Eval(Int32[] tokens, int pastTokensCount)
+tokens Int32[]
pastTokensCount Int32
Int32
+The updated pastTokensCount.
internal IEnumerable<string> GenerateResult(IEnumerable<int> ids)
+public void Dispose()
+Namespace: LLama
+The quantizer to quantize the model.
+public static class LLamaQuantizer
+Inheritance Object → LLamaQuantizer
+Quantize the model.
+public static bool Quantize(string srcFileName, string dstFilename, LLamaFtype ftype, int nthread)
+srcFileName String
+The model file to be quantized.
dstFilename String
+The path to save the quantized model.
ftype LLamaFtype
+The type of quantization.
nthread Int32
+Thread to be used during the quantization. By default it's the physical core number.
Boolean
+Whether the quantization is successful.
Quantize the model.
+public static bool Quantize(string srcFileName, string dstFilename, string ftype, int nthread)
+srcFileName String
+The model file to be quantized.
dstFilename String
+The path to save the quantized model.
ftype String
+The type of quantization.
nthread Int32
+Thread to be used during the quantization. By default it's the physical core number.
Boolean
+Whether the quantization is successful.
Namespace: LLama
+A class that contains all the transforms provided internally by LLama.
+public class LLamaTransforms
+Inheritance Object → LLamaTransforms
+public LLamaTransforms()
+Namespace: LLama.Native
+public struct LLamaContextParams
+Inheritance Object → ValueType → LLamaContextParams
+text context
+public int n_ctx;
+number of layers to store in VRAM
+public int n_gpu_layers;
+RNG seed, -1 for random
+public int seed;
+use fp16 for KV cache
+public bool f16_kv;
+the llama_eval() call computes all logits, not just the last one
+public bool logits_all;
+only load the vocabulary, no weights
+public bool vocab_only;
+use mmap if possible
+public bool use_mmap;
+force system to keep model in RAM
+public bool use_mlock;
+embedding mode only
+public bool embedding;
+called with a progress value between 0 and 1, pass NULL to disable
+public IntPtr progress_callback;
+context pointer passed to the progress callback
+public IntPtr progress_callback_user_data;
+Namespace: LLama.Native
+public enum LLamaFtype
+Inheritance Object → ValueType → Enum → LLamaFtype
+Implements IComparable, IFormattable, IConvertible
| Name | +Value | +Description | +
|---|---|---|
| + | + | + |
Namespace: LLama.Native
+public struct LLamaTokenData
+Inheritance Object → ValueType → LLamaTokenData
+token id
+public int id;
+log-odds of the token
+public float logit;
+probability of the token
+public float p;
+LLamaTokenData(int id, float logit, float p)
+id Int32
logit Single
p Single
Namespace: LLama.Native
+public struct LLamaTokenDataArray
+Inheritance Object → ValueType → LLamaTokenDataArray
+public Memory<LLamaTokenData> data;
+public ulong size;
+public bool sorted;
+LLamaTokenDataArray(LLamaTokenData[] data, ulong size, bool sorted)
+data LLamaTokenData[]
size UInt64
sorted Boolean
Namespace: LLama.Native
+public struct LLamaTokenDataArrayNative
+Inheritance Object → ValueType → LLamaTokenDataArrayNative
+public IntPtr data;
+public ulong size;
+public bool sorted;
+Namespace: LLama.Native
+public class NativeApi
+Inheritance Object → NativeApi
+public NativeApi()
+public static void llama_print_timings(SafeLLamaContextHandle ctx)
+public static void llama_reset_timings(SafeLLamaContextHandle ctx)
+Print system information
+public static IntPtr llama_print_system_info()
+public static int llama_model_quantize(string fname_inp, string fname_out, LLamaFtype ftype, int nthread)
+fname_inp String
fname_out String
ftype LLamaFtype
nthread Int32
Repetition penalty described in CTRL academic paper https://arxiv.org/abs/1909.05858, with negative logit fix.
+public static void llama_sample_repetition_penalty(SafeLLamaContextHandle ctx, IntPtr candidates, Int32[] last_tokens, ulong last_tokens_size, float penalty)
+candidates IntPtr
+Pointer to LLamaTokenDataArray
last_tokens Int32[]
last_tokens_size UInt64
penalty Single
Frequency and presence penalties described in OpenAI API https://platform.openai.com/docs/api-reference/parameter-details.
+public static void llama_sample_frequency_and_presence_penalties(SafeLLamaContextHandle ctx, IntPtr candidates, Int32[] last_tokens, ulong last_tokens_size, float alpha_frequency, float alpha_presence)
+candidates IntPtr
+Pointer to LLamaTokenDataArray
last_tokens Int32[]
last_tokens_size UInt64
alpha_frequency Single
alpha_presence Single
Sorts candidate tokens by their logits in descending order and calculate probabilities based on logits.
+public static void llama_sample_softmax(SafeLLamaContextHandle ctx, IntPtr candidates)
+candidates IntPtr
+Pointer to LLamaTokenDataArray
Top-K sampling described in academic paper "The Curious Case of Neural Text Degeneration" https://arxiv.org/abs/1904.09751
+public static void llama_sample_top_k(SafeLLamaContextHandle ctx, IntPtr candidates, int k, ulong min_keep)
+candidates IntPtr
+Pointer to LLamaTokenDataArray
k Int32
min_keep UInt64
Nucleus sampling described in academic paper "The Curious Case of Neural Text Degeneration" https://arxiv.org/abs/1904.09751
+public static void llama_sample_top_p(SafeLLamaContextHandle ctx, IntPtr candidates, float p, ulong min_keep)
+candidates IntPtr
+Pointer to LLamaTokenDataArray
p Single
min_keep UInt64
Tail Free Sampling described in https://www.trentonbricken.com/Tail-Free-Sampling/.
+public static void llama_sample_tail_free(SafeLLamaContextHandle ctx, IntPtr candidates, float z, ulong min_keep)
+candidates IntPtr
+Pointer to LLamaTokenDataArray
z Single
min_keep UInt64
Locally Typical Sampling implementation described in the paper https://arxiv.org/abs/2202.00666.
+public static void llama_sample_typical(SafeLLamaContextHandle ctx, IntPtr candidates, float p, ulong min_keep)
+candidates IntPtr
+Pointer to LLamaTokenDataArray
p Single
min_keep UInt64
public static void llama_sample_temperature(SafeLLamaContextHandle ctx, IntPtr candidates, float temp)
+candidates IntPtr
temp Single
Mirostat 1.0 algorithm described in the paper https://arxiv.org/abs/2007.14966. Uses tokens instead of words.
+public static int llama_sample_token_mirostat(SafeLLamaContextHandle ctx, IntPtr candidates, float tau, float eta, int m, Single* mu)
+candidates IntPtr
+A vector of llama_token_data containing the candidate tokens, their probabilities (p), and log-odds (logit) for the current position in the generated text.
tau Single
+The target cross-entropy (or surprise) value you want to achieve for the generated text. A higher value corresponds to more surprising or less predictable text, while a lower value corresponds to less surprising or more predictable text.
eta Single
+The learning rate used to update mu based on the error between the target and observed surprisal of the sampled word. A larger learning rate will cause mu to be updated more quickly, while a smaller learning rate will result in slower updates.
m Int32
+The number of tokens considered in the estimation of s_hat. This is an arbitrary value that is used to calculate s_hat, which in turn helps to calculate the value of k. In the paper, they use m = 100, but you can experiment with different values to see how it affects the performance of the algorithm.
mu Single*
+Maximum cross-entropy. This value is initialized to be twice the target cross-entropy (2 * tau) and is updated in the algorithm based on the error between the target and observed surprisal.
Mirostat 2.0 algorithm described in the paper https://arxiv.org/abs/2007.14966. Uses tokens instead of words.
+public static int llama_sample_token_mirostat_v2(SafeLLamaContextHandle ctx, IntPtr candidates, float tau, float eta, Single* mu)
+candidates IntPtr
+A vector of llama_token_data containing the candidate tokens, their probabilities (p), and log-odds (logit) for the current position in the generated text.
tau Single
+The target cross-entropy (or surprise) value you want to achieve for the generated text. A higher value corresponds to more surprising or less predictable text, while a lower value corresponds to less surprising or more predictable text.
eta Single
+The learning rate used to update mu based on the error between the target and observed surprisal of the sampled word. A larger learning rate will cause mu to be updated more quickly, while a smaller learning rate will result in slower updates.
mu Single*
+Maximum cross-entropy. This value is initialized to be twice the target cross-entropy (2 * tau) and is updated in the algorithm based on the error between the target and observed surprisal.
Selects the token with the highest probability.
+public static int llama_sample_token_greedy(SafeLLamaContextHandle ctx, IntPtr candidates)
+candidates IntPtr
+Pointer to LLamaTokenDataArray
Randomly selects a token from the candidates based on their probabilities.
+public static int llama_sample_token(SafeLLamaContextHandle ctx, IntPtr candidates)
+candidates IntPtr
+Pointer to LLamaTokenDataArray
public static bool llama_empty_call()
+public static LLamaContextParams llama_context_default_params()
+public static bool llama_mmap_supported()
+public static bool llama_mlock_supported()
+Various functions for loading a ggml llama model. + Allocate (almost) all memory needed for the model. + Return NULL on failure
+public static IntPtr llama_init_from_file(string path_model, LLamaContextParams params_)
+path_model String
params_ LLamaContextParams
not great API - very likely to change. + Initialize the llama + ggml backend + Call once at the start of the program
+public static void llama_init_backend()
+Frees all allocated memory
+public static void llama_free(IntPtr ctx)
+ctx IntPtr
Apply a LoRA adapter to a loaded model + path_base_model is the path to a higher quality model to use as a base for + the layers modified by the adapter. Can be NULL to use the current loaded model. + The model needs to be reloaded before applying a new adapter, otherwise the adapter + will be applied on top of the previous one
+public static int llama_apply_lora_from_file(SafeLLamaContextHandle ctx, string path_lora, string path_base_model, int n_threads)
+path_lora String
path_base_model String
n_threads Int32
Int32
+Returns 0 on success
Returns the number of tokens in the KV cache
+public static int llama_get_kv_cache_token_count(SafeLLamaContextHandle ctx)
+Sets the current rng seed.
+public static void llama_set_rng_seed(SafeLLamaContextHandle ctx, int seed)
+seed Int32
Returns the maximum size in bytes of the state (rng, logits, embedding + and kv_cache) - will often be smaller after compacting tokens
+public static ulong llama_get_state_size(SafeLLamaContextHandle ctx)
+Copies the state to the specified destination address. + Destination needs to have allocated enough memory. + Returns the number of bytes copied
+public static ulong llama_copy_state_data(SafeLLamaContextHandle ctx, Byte[] dest)
+dest Byte[]
Set the state reading from the specified address + Returns the number of bytes read
+public static ulong llama_set_state_data(SafeLLamaContextHandle ctx, Byte[] src)
+src Byte[]
Load session file
+public static bool llama_load_session_file(SafeLLamaContextHandle ctx, string path_session, Int32[] tokens_out, ulong n_token_capacity, UInt64* n_token_count_out)
+path_session String
tokens_out Int32[]
n_token_capacity UInt64
n_token_count_out UInt64*
Save session file
+public static bool llama_save_session_file(SafeLLamaContextHandle ctx, string path_session, Int32[] tokens, ulong n_token_count)
+path_session String
tokens Int32[]
n_token_count UInt64
Run the llama inference to obtain the logits and probabilities for the next token. + tokens + n_tokens is the provided batch of new tokens to process + n_past is the number of tokens to use from previous eval calls
+public static int llama_eval(SafeLLamaContextHandle ctx, Int32[] tokens, int n_tokens, int n_past, int n_threads)
+tokens Int32[]
n_tokens Int32
n_past Int32
n_threads Int32
Int32
+Returns 0 on success
public static int llama_eval_with_pointer(SafeLLamaContextHandle ctx, Int32* tokens, int n_tokens, int n_past, int n_threads)
+tokens Int32*
n_tokens Int32
n_past Int32
n_threads Int32
Convert the provided text into tokens. + The tokens pointer must be large enough to hold the resulting tokens. + Returns the number of tokens on success, no more than n_max_tokens + Returns a negative number on failure - the number of tokens that would have been returned
+public static int llama_tokenize(SafeLLamaContextHandle ctx, string text, Encoding encoding, Int32[] tokens, int n_max_tokens, bool add_bos)
+text String
encoding Encoding
tokens Int32[]
n_max_tokens Int32
add_bos Boolean
public static int llama_tokenize_native(SafeLLamaContextHandle ctx, SByte[] text, Int32[] tokens, int n_max_tokens, bool add_bos)
+text SByte[]
tokens Int32[]
n_max_tokens Int32
add_bos Boolean
public static int llama_n_vocab(SafeLLamaContextHandle ctx)
+public static int llama_n_ctx(SafeLLamaContextHandle ctx)
+public static int llama_n_embd(SafeLLamaContextHandle ctx)
+Token logits obtained from the last call to llama_eval() + The logits for the last token are stored in the last row + Can be mutated in order to change the probabilities of the next token + Rows: n_tokens + Cols: n_vocab
+public static Single* llama_get_logits(SafeLLamaContextHandle ctx)
+Get the embeddings for the input + shape: [n_embd] (1-dimensional)
+public static Single* llama_get_embeddings(SafeLLamaContextHandle ctx)
+Token Id -> String. Uses the vocabulary in the provided context
+public static IntPtr llama_token_to_str(SafeLLamaContextHandle ctx, int token)
+token Int32
IntPtr
+Pointer to a string.
public static int llama_token_bos()
+public static int llama_token_eos()
+public static int llama_token_nl()
+Namespace: LLama.Native
+public class SafeLLamaContextHandle : SafeLLamaHandleBase, System.IDisposable
+Inheritance Object → CriticalFinalizerObject → SafeHandle → SafeLLamaHandleBase → SafeLLamaContextHandle
+Implements IDisposable
public bool IsInvalid { get; }
+public bool IsClosed { get; }
+public SafeLLamaContextHandle(IntPtr handle)
+handle IntPtr
protected bool ReleaseHandle()
+Namespace: LLama.Native
+public abstract class SafeLLamaHandleBase : System.Runtime.InteropServices.SafeHandle, System.IDisposable
+Inheritance Object → CriticalFinalizerObject → SafeHandle → SafeLLamaHandleBase
+Implements IDisposable
public bool IsInvalid { get; }
+public bool IsClosed { get; }
+public string ToString()
+Namespace: LLama.OldVersion
+public class ChatCompletion : System.IEquatable`1[[LLama.OldVersion.ChatCompletion, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]
+Inheritance Object → ChatCompletion
+Implements IEquatable<ChatCompletion>
public string Id { get; set; }
+public string Object { get; set; }
+public int Created { get; set; }
+public string Model { get; set; }
+public ChatCompletionChoice[] Choices { get; set; }
+public CompletionUsage Usage { get; set; }
+public ChatCompletion(string Id, string Object, int Created, string Model, ChatCompletionChoice[] Choices, CompletionUsage Usage)
+Id String
Object String
Created Int32
Model String
Choices ChatCompletionChoice[]
Usage CompletionUsage
public string ToString()
+protected bool PrintMembers(StringBuilder builder)
+builder StringBuilder
public int GetHashCode()
+public bool Equals(object obj)
+obj Object
public bool Equals(ChatCompletion other)
+other ChatCompletion
public ChatCompletion <Clone>$()
+public void Deconstruct(String& Id, String& Object, Int32& Created, String& Model, ChatCompletionChoice[]& Choices, CompletionUsage& Usage)
+Id String&
Object String&
Created Int32&
Model String&
Choices ChatCompletionChoice[]&
Usage CompletionUsage&
Namespace: LLama.OldVersion
+public class ChatCompletionChoice : System.IEquatable`1[[LLama.OldVersion.ChatCompletionChoice, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]
+Inheritance Object → ChatCompletionChoice
+Implements IEquatable<ChatCompletionChoice>
public int Index { get; set; }
+public ChatCompletionMessage Message { get; set; }
+public string FinishReason { get; set; }
+public ChatCompletionChoice(int Index, ChatCompletionMessage Message, string FinishReason)
+Index Int32
Message ChatCompletionMessage
FinishReason String
public string ToString()
+protected bool PrintMembers(StringBuilder builder)
+builder StringBuilder
public int GetHashCode()
+public bool Equals(object obj)
+obj Object
public bool Equals(ChatCompletionChoice other)
+other ChatCompletionChoice
public ChatCompletionChoice <Clone>$()
+public void Deconstruct(Int32& Index, ChatCompletionMessage& Message, String& FinishReason)
+Index Int32&
Message ChatCompletionMessage&
FinishReason String&
Namespace: LLama.OldVersion
+public class ChatCompletionChunk : System.IEquatable`1[[LLama.OldVersion.ChatCompletionChunk, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]
+Inheritance Object → ChatCompletionChunk
+Implements IEquatable<ChatCompletionChunk>
public string Id { get; set; }
+public string Model { get; set; }
+public string Object { get; set; }
+public int Created { get; set; }
+public ChatCompletionChunkChoice[] Choices { get; set; }
+public ChatCompletionChunk(string Id, string Model, string Object, int Created, ChatCompletionChunkChoice[] Choices)
+Id String
Model String
Object String
Created Int32
Choices ChatCompletionChunkChoice[]
public string ToString()
+protected bool PrintMembers(StringBuilder builder)
+builder StringBuilder
public int GetHashCode()
+public bool Equals(object obj)
+obj Object
public bool Equals(ChatCompletionChunk other)
+other ChatCompletionChunk
public ChatCompletionChunk <Clone>$()
+public void Deconstruct(String& Id, String& Model, String& Object, Int32& Created, ChatCompletionChunkChoice[]& Choices)
+Id String&
Model String&
Object String&
Created Int32&
Choices ChatCompletionChunkChoice[]&
Namespace: LLama.OldVersion
+public class ChatCompletionChunkChoice : System.IEquatable`1[[LLama.OldVersion.ChatCompletionChunkChoice, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]
+Inheritance Object → ChatCompletionChunkChoice
+Implements IEquatable<ChatCompletionChunkChoice>
public int Index { get; set; }
+public ChatCompletionChunkDelta Delta { get; set; }
+public string FinishReason { get; set; }
+public ChatCompletionChunkChoice(int Index, ChatCompletionChunkDelta Delta, string FinishReason)
+Index Int32
Delta ChatCompletionChunkDelta
FinishReason String
public string ToString()
+protected bool PrintMembers(StringBuilder builder)
+builder StringBuilder
public int GetHashCode()
+public bool Equals(object obj)
+obj Object
public bool Equals(ChatCompletionChunkChoice other)
+other ChatCompletionChunkChoice
public ChatCompletionChunkChoice <Clone>$()
+public void Deconstruct(Int32& Index, ChatCompletionChunkDelta& Delta, String& FinishReason)
+Index Int32&
Delta ChatCompletionChunkDelta&
FinishReason String&
Namespace: LLama.OldVersion
+public class ChatCompletionChunkDelta : System.IEquatable`1[[LLama.OldVersion.ChatCompletionChunkDelta, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]
+Inheritance Object → ChatCompletionChunkDelta
+Implements IEquatable<ChatCompletionChunkDelta>
public string Role { get; set; }
+public string Content { get; set; }
+public ChatCompletionChunkDelta(string Role, string Content)
+Role String
Content String
public string ToString()
+protected bool PrintMembers(StringBuilder builder)
+builder StringBuilder
public int GetHashCode()
+public bool Equals(object obj)
+obj Object
public bool Equals(ChatCompletionChunkDelta other)
+other ChatCompletionChunkDelta
public ChatCompletionChunkDelta <Clone>$()
+public void Deconstruct(String& Role, String& Content)
+Role String&
Content String&
Namespace: LLama.OldVersion
+public class ChatCompletionMessage : System.IEquatable`1[[LLama.OldVersion.ChatCompletionMessage, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]
+Inheritance Object → ChatCompletionMessage
+Implements IEquatable<ChatCompletionMessage>
public ChatRole Role { get; set; }
+public string Content { get; set; }
+public string Name { get; set; }
+public ChatCompletionMessage(ChatRole Role, string Content, string Name)
+Role ChatRole
Content String
Name String
public string ToString()
+protected bool PrintMembers(StringBuilder builder)
+builder StringBuilder
public int GetHashCode()
+public bool Equals(object obj)
+obj Object
public bool Equals(ChatCompletionMessage other)
+other ChatCompletionMessage
public ChatCompletionMessage <Clone>$()
+public void Deconstruct(ChatRole& Role, String& Content, String& Name)
+Role ChatRole&
Content String&
Name String&
Namespace: LLama.OldVersion
+public class ChatMessageRecord : System.IEquatable`1[[LLama.OldVersion.ChatMessageRecord, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]
+Inheritance Object → ChatMessageRecord
+Implements IEquatable<ChatMessageRecord>
public ChatCompletionMessage Message { get; set; }
+public DateTime Time { get; set; }
+public ChatMessageRecord(ChatCompletionMessage Message, DateTime Time)
+Message ChatCompletionMessage
Time DateTime
public string ToString()
+protected bool PrintMembers(StringBuilder builder)
+builder StringBuilder
public int GetHashCode()
+public bool Equals(object obj)
+obj Object
public bool Equals(ChatMessageRecord other)
+other ChatMessageRecord
public ChatMessageRecord <Clone>$()
+public void Deconstruct(ChatCompletionMessage& Message, DateTime& Time)
+Message ChatCompletionMessage&
Time DateTime&
Namespace: LLama.OldVersion
+public enum ChatRole
+Inheritance Object → ValueType → Enum → ChatRole
+Implements IComparable, IFormattable, IConvertible
| Name | +Value | +Description | +
|---|---|---|
| + | + | + |
Namespace: LLama.OldVersion
+public class ChatSession<T>
+T
Inheritance Object → ChatSession<T>
+public ChatSession(T model)
+model T
public IEnumerable<string> Chat(string text, string prompt, string encoding)
+text String
prompt String
encoding String
public ChatSession<T> WithPrompt(string prompt, string encoding)
+prompt String
encoding String
public ChatSession<T> WithPromptFile(string promptFilename, string encoding)
+promptFilename String
encoding String
Set the keyword to split the return value of chat AI.
+public ChatSession<T> WithAntiprompt(String[] antiprompt)
+antiprompt String[]
Namespace: LLama.OldVersion
+public class Completion : System.IEquatable`1[[LLama.OldVersion.Completion, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]
+Inheritance Object → Completion
+Implements IEquatable<Completion>
public string Id { get; set; }
+public string Object { get; set; }
+public int Created { get; set; }
+public string Model { get; set; }
+public CompletionChoice[] Choices { get; set; }
+public CompletionUsage Usage { get; set; }
+public Completion(string Id, string Object, int Created, string Model, CompletionChoice[] Choices, CompletionUsage Usage)
+Id String
Object String
Created Int32
Model String
Choices CompletionChoice[]
Usage CompletionUsage
public string ToString()
+protected bool PrintMembers(StringBuilder builder)
+builder StringBuilder
public int GetHashCode()
+public bool Equals(object obj)
+obj Object
public bool Equals(Completion other)
+other Completion
public Completion <Clone>$()
+public void Deconstruct(String& Id, String& Object, Int32& Created, String& Model, CompletionChoice[]& Choices, CompletionUsage& Usage)
+Id String&
Object String&
Created Int32&
Model String&
Choices CompletionChoice[]&
Usage CompletionUsage&
Namespace: LLama.OldVersion
+public class CompletionChoice : System.IEquatable`1[[LLama.OldVersion.CompletionChoice, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]
+Inheritance Object → CompletionChoice
+Implements IEquatable<CompletionChoice>
public string Text { get; set; }
+public int Index { get; set; }
+public CompletionLogprobs Logprobs { get; set; }
+public string FinishReason { get; set; }
+public CompletionChoice(string Text, int Index, CompletionLogprobs Logprobs, string FinishReason)
+Text String
Index Int32
Logprobs CompletionLogprobs
FinishReason String
public string ToString()
+protected bool PrintMembers(StringBuilder builder)
+builder StringBuilder
public int GetHashCode()
+public bool Equals(object obj)
+obj Object
public bool Equals(CompletionChoice other)
+other CompletionChoice
public CompletionChoice <Clone>$()
+public void Deconstruct(String& Text, Int32& Index, CompletionLogprobs& Logprobs, String& FinishReason)
+Text String&
Index Int32&
Logprobs CompletionLogprobs&
FinishReason String&
Namespace: LLama.OldVersion
+public class CompletionChunk : System.IEquatable`1[[LLama.OldVersion.CompletionChunk, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]
+Inheritance Object → CompletionChunk
+Implements IEquatable<CompletionChunk>
public string Id { get; set; }
+public string Object { get; set; }
+public int Created { get; set; }
+public string Model { get; set; }
+public CompletionChoice[] Choices { get; set; }
+public CompletionChunk(string Id, string Object, int Created, string Model, CompletionChoice[] Choices)
+Id String
Object String
Created Int32
Model String
Choices CompletionChoice[]
public string ToString()
+protected bool PrintMembers(StringBuilder builder)
+builder StringBuilder
public int GetHashCode()
+public bool Equals(object obj)
+obj Object
public bool Equals(CompletionChunk other)
+other CompletionChunk
public CompletionChunk <Clone>$()
+public void Deconstruct(String& Id, String& Object, Int32& Created, String& Model, CompletionChoice[]& Choices)
+Id String&
Object String&
Created Int32&
Model String&
Choices CompletionChoice[]&
Namespace: LLama.OldVersion
+public class CompletionLogprobs : System.IEquatable`1[[LLama.OldVersion.CompletionLogprobs, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]
+Inheritance Object → CompletionLogprobs
+Implements IEquatable<CompletionLogprobs>
public Int32[] TextOffset { get; set; }
+public Single[] TokenLogProbs { get; set; }
+public String[] Tokens { get; set; }
+public Dictionary`2[] TopLogprobs { get; set; }
+public CompletionLogprobs(Int32[] TextOffset, Single[] TokenLogProbs, String[] Tokens, Dictionary`2[] TopLogprobs)
+TextOffset Int32[]
TokenLogProbs Single[]
Tokens String[]
TopLogprobs Dictionary`2[]
public string ToString()
+protected bool PrintMembers(StringBuilder builder)
+builder StringBuilder
public int GetHashCode()
+public bool Equals(object obj)
+obj Object
public bool Equals(CompletionLogprobs other)
+other CompletionLogprobs
public CompletionLogprobs <Clone>$()
+public void Deconstruct(Int32[]& TextOffset, Single[]& TokenLogProbs, String[]& Tokens, Dictionary`2[]& TopLogprobs)
+TextOffset Int32[]&
TokenLogProbs Single[]&
Tokens String[]&
TopLogprobs Dictionary`2[]&
Namespace: LLama.OldVersion
+public class CompletionUsage : System.IEquatable`1[[LLama.OldVersion.CompletionUsage, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]
+Inheritance Object → CompletionUsage
+Implements IEquatable<CompletionUsage>
public int PromptTokens { get; set; }
+public int CompletionTokens { get; set; }
+public int TotalTokens { get; set; }
+public CompletionUsage(int PromptTokens, int CompletionTokens, int TotalTokens)
+PromptTokens Int32
CompletionTokens Int32
TotalTokens Int32
public string ToString()
+protected bool PrintMembers(StringBuilder builder)
+builder StringBuilder
public int GetHashCode()
+public bool Equals(object obj)
+obj Object
public bool Equals(CompletionUsage other)
+other CompletionUsage
public CompletionUsage <Clone>$()
+public void Deconstruct(Int32& PromptTokens, Int32& CompletionTokens, Int32& TotalTokens)
+PromptTokens Int32&
CompletionTokens Int32&
TotalTokens Int32&
Namespace: LLama.OldVersion
+public class Embedding : System.IEquatable`1[[LLama.OldVersion.Embedding, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]
+Inheritance Object → Embedding
+Implements IEquatable<Embedding>
public string Object { get; set; }
+public string Model { get; set; }
+public EmbeddingData[] Data { get; set; }
+public EmbeddingUsage Usage { get; set; }
+public Embedding(string Object, string Model, EmbeddingData[] Data, EmbeddingUsage Usage)
+Object String
Model String
Data EmbeddingData[]
Usage EmbeddingUsage
public string ToString()
+protected bool PrintMembers(StringBuilder builder)
+builder StringBuilder
public int GetHashCode()
+public bool Equals(object obj)
+obj Object
public bool Equals(Embedding other)
+other Embedding
public Embedding <Clone>$()
+public void Deconstruct(String& Object, String& Model, EmbeddingData[]& Data, EmbeddingUsage& Usage)
+Object String&
Model String&
Data EmbeddingData[]&
Usage EmbeddingUsage&
Namespace: LLama.OldVersion
+public class EmbeddingData : System.IEquatable`1[[LLama.OldVersion.EmbeddingData, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]
+Inheritance Object → EmbeddingData
+Implements IEquatable<EmbeddingData>
public int Index { get; set; }
+public string Object { get; set; }
+public Single[] Embedding { get; set; }
+public EmbeddingData(int Index, string Object, Single[] Embedding)
+Index Int32
Object String
Embedding Single[]
public string ToString()
+protected bool PrintMembers(StringBuilder builder)
+builder StringBuilder
public int GetHashCode()
+public bool Equals(object obj)
+obj Object
public bool Equals(EmbeddingData other)
+other EmbeddingData
public EmbeddingData <Clone>$()
+public void Deconstruct(Int32& Index, String& Object, Single[]& Embedding)
+Index Int32&
Object String&
Embedding Single[]&
Namespace: LLama.OldVersion
+public class EmbeddingUsage : System.IEquatable`1[[LLama.OldVersion.EmbeddingUsage, LLamaSharp, Version=0.4.0.0, Culture=neutral, PublicKeyToken=null]]
+Inheritance Object → EmbeddingUsage
+Implements IEquatable<EmbeddingUsage>
public int PromptTokens { get; set; }
+public int TotalTokens { get; set; }
+public EmbeddingUsage(int PromptTokens, int TotalTokens)
+PromptTokens Int32
TotalTokens Int32
public string ToString()
+protected bool PrintMembers(StringBuilder builder)
+builder StringBuilder
public int GetHashCode()
+public bool Equals(object obj)
+obj Object
public bool Equals(EmbeddingUsage other)
+other EmbeddingUsage
public EmbeddingUsage <Clone>$()
+public void Deconstruct(Int32& PromptTokens, Int32& TotalTokens)
+PromptTokens Int32&
TotalTokens Int32&
Namespace: LLama.OldVersion
+public interface IChatModel
+public abstract string Name { get; }
+IEnumerable<string> Chat(string text, string prompt, string encoding)
+text String
prompt String
encoding String
Init a prompt for chat and automatically produce the next prompt during the chat.
+void InitChatPrompt(string prompt, string encoding)
+prompt String
encoding String
void InitChatAntiprompt(String[] antiprompt)
+antiprompt String[]
Namespace: LLama.OldVersion
+public class LLamaEmbedder : System.IDisposable
+Inheritance Object → LLamaEmbedder
+Implements IDisposable
public LLamaEmbedder(LLamaParams params)
+params LLamaParams
public Single[] GetEmbeddings(string text, int n_thread, bool add_bos, string encoding)
+text String
n_thread Int32
add_bos Boolean
encoding String
public void Dispose()
+Namespace: LLama.OldVersion
+public class LLamaModel : IChatModel, System.IDisposable
+Inheritance Object → LLamaModel
+Implements IChatModel, IDisposable
public string Name { get; set; }
+public bool Verbose { get; set; }
+public SafeLLamaContextHandle NativeHandle { get; }
+Please refer LLamaParams to find the meanings of each arg. Be sure to have set the n_gpu_layers, otherwise it will
+ load 20 layers to gpu by default.
public LLamaModel(string model_path, string model_name, bool verbose, int seed, int n_threads, int n_predict, int n_ctx, int n_batch, int n_keep, int n_gpu_layers, Dictionary<int, float> logit_bias, int top_k, float top_p, float tfs_z, float typical_p, float temp, float repeat_penalty, int repeat_last_n, float frequency_penalty, float presence_penalty, int mirostat, float mirostat_tau, float mirostat_eta, string prompt, string path_session, string input_prefix, string input_suffix, List<string> antiprompt, string lora_adapter, string lora_base, bool memory_f16, bool random_prompt, bool use_color, bool interactive, bool embedding, bool interactive_first, bool prompt_cache_all, bool instruct, bool penalize_nl, bool perplexity, bool use_mmap, bool use_mlock, bool mem_test, bool verbose_prompt, string encoding)
+model_path String
+The model file path.
model_name String
+The model name.
verbose Boolean
+Whether to print details when running the model.
seed Int32
n_threads Int32
n_predict Int32
n_ctx Int32
n_batch Int32
n_keep Int32
n_gpu_layers Int32
logit_bias Dictionary<Int32, Single>
top_k Int32
top_p Single
tfs_z Single
typical_p Single
temp Single
repeat_penalty Single
repeat_last_n Int32
frequency_penalty Single
presence_penalty Single
mirostat Int32
mirostat_tau Single
mirostat_eta Single
prompt String
path_session String
input_prefix String
input_suffix String
antiprompt List<String>
lora_adapter String
lora_base String
memory_f16 Boolean
random_prompt Boolean
use_color Boolean
interactive Boolean
embedding Boolean
interactive_first Boolean
prompt_cache_all Boolean
instruct Boolean
penalize_nl Boolean
perplexity Boolean
use_mmap Boolean
use_mlock Boolean
mem_test Boolean
verbose_prompt Boolean
encoding String
Please refer LLamaParams to find the meanings of each arg. Be sure to have set the n_gpu_layers, otherwise it will
+ load 20 layers to gpu by default.
public LLamaModel(LLamaParams params, string name, bool verbose, string encoding)
+params LLamaParams
+The LLamaModel params
name String
+Model name
verbose Boolean
+Whether to output the detailed info.
encoding String
Apply a prompt to the model.
+public LLamaModel WithPrompt(string prompt, string encoding)
+prompt String
encoding String
Apply the prompt file to the model.
+public LLamaModel WithPromptFile(string promptFileName)
+promptFileName String
public void InitChatPrompt(string prompt, string encoding)
+prompt String
encoding String
public void InitChatAntiprompt(String[] antiprompt)
+antiprompt String[]
Chat with the LLaMa model under interactive mode.
+public IEnumerable<string> Chat(string text, string prompt, string encoding)
+text String
prompt String
encoding String
Save the state to specified path.
+public void SaveState(string filename)
+filename String
Load the state from specified path.
+public void LoadState(string filename, bool clearPreviousEmbed)
+filename String
clearPreviousEmbed Boolean
+Whether to clear previous footprints of this model.
Tokenize a string.
+public List<int> Tokenize(string text, string encoding)
+text String
+The utf-8 encoded string to tokenize.
encoding String
List<Int32>
+A list of tokens.
RuntimeError
+If the tokenization failed.
Detokenize a list of tokens.
+public string DeTokenize(IEnumerable<int> tokens)
+tokens IEnumerable<Int32>
+The list of tokens to detokenize.
String
+The detokenized string.
Call the model to run inference.
+public IEnumerable<string> Call(string text, string encoding)
+text String
encoding String
public void Dispose()
+Namespace: LLama.OldVersion
+public struct LLamaParams
+Inheritance Object → ValueType → LLamaParams
+public int seed;
+public int n_threads;
+public int n_predict;
+public int n_ctx;
+public int n_batch;
+public int n_keep;
+public int n_gpu_layers;
+public Dictionary<int, float> logit_bias;
+public int top_k;
+public float top_p;
+public float tfs_z;
+public float typical_p;
+public float temp;
+public float repeat_penalty;
+public int repeat_last_n;
+public float frequency_penalty;
+public float presence_penalty;
+public int mirostat;
+public float mirostat_tau;
+public float mirostat_eta;
+public string model;
+public string prompt;
+public string path_session;
+public string input_prefix;
+public string input_suffix;
+public List<string> antiprompt;
+public string lora_adapter;
+public string lora_base;
+public bool memory_f16;
+public bool random_prompt;
+public bool use_color;
+public bool interactive;
+public bool prompt_cache_all;
+public bool embedding;
+public bool interactive_first;
+public bool instruct;
+public bool penalize_nl;
+public bool perplexity;
+public bool use_mmap;
+public bool use_mlock;
+public bool mem_test;
+public bool verbose_prompt;
+LLamaParams(int seed, int n_threads, int n_predict, int n_ctx, int n_batch, int n_keep, int n_gpu_layers, Dictionary<int, float> logit_bias, int top_k, float top_p, float tfs_z, float typical_p, float temp, float repeat_penalty, int repeat_last_n, float frequency_penalty, float presence_penalty, int mirostat, float mirostat_tau, float mirostat_eta, string model, string prompt, string path_session, string input_prefix, string input_suffix, List<string> antiprompt, string lora_adapter, string lora_base, bool memory_f16, bool random_prompt, bool use_color, bool interactive, bool prompt_cache_all, bool embedding, bool interactive_first, bool instruct, bool penalize_nl, bool perplexity, bool use_mmap, bool use_mlock, bool mem_test, bool verbose_prompt)
+seed Int32
n_threads Int32
n_predict Int32
n_ctx Int32
n_batch Int32
n_keep Int32
n_gpu_layers Int32
logit_bias Dictionary<Int32, Single>
top_k Int32
top_p Single
tfs_z Single
typical_p Single
temp Single
repeat_penalty Single
repeat_last_n Int32
frequency_penalty Single
presence_penalty Single
mirostat Int32
mirostat_tau Single
mirostat_eta Single
model String
prompt String
path_session String
input_prefix String
input_suffix String
antiprompt List<String>
lora_adapter String
lora_base String
memory_f16 Boolean
random_prompt Boolean
use_color Boolean
interactive Boolean
prompt_cache_all Boolean
embedding Boolean
interactive_first Boolean
instruct Boolean
penalize_nl Boolean
perplexity Boolean
use_mmap Boolean
use_mlock Boolean
mem_test Boolean
verbose_prompt Boolean
Namespace: LLama
+A LLamaModel what could be reset. Note that using this class will consume about 10% more memories.
+public class ResettableLLamaModel : LLamaModel, System.IDisposable
+Inheritance Object → LLamaModel → ResettableLLamaModel
+Implements IDisposable
The initial state of the model
+public Byte[] OriginalState { get; set; }
+The context size.
+public int ContextSize { get; }
+The model params set for this model.
+public ModelParams Params { get; set; }
+The native handle, which is used to be passed to the native APIs. Please avoid using it + unless you know what is the usage of the Native API.
+public SafeLLamaContextHandle NativeHandle { get; }
+The encoding set for this model to deal with text input.
+public Encoding Encoding { get; }
+public ResettableLLamaModel(ModelParams Params, string encoding)
+Params ModelParams
encoding String
Reset the state to the initial state.
+public void Reset()
+Namespace: LLama
+The base class for stateful LLama executors.
+public abstract class StatefulExecutorBase : LLama.Abstractions.ILLamaExecutor
+Inheritance Object → StatefulExecutorBase
+Implements ILLamaExecutor
The mode used by the executor.
+public LLamaModel Model { get; }
+This API is currently not verified.
+public StatefulExecutorBase WithSessionFile(string filename)
+filename String
This API has not been verified currently.
+public void SaveSessionFile(string filename)
+filename String
After running out of the context, take some tokens from the original prompt and recompute the logits in batches.
+protected void HandleRunOutOfContext(int tokensToKeep)
+tokensToKeep Int32
Try to reuse the matching prefix from the session file.
+protected void TryReuseMathingPrefix()
+Decide whether to continue the loop.
+protected abstract bool GetLoopCondition(InferStateArgs args)
+args InferStateArgs
Preprocess the inputs before the inference.
+protected abstract void PreprocessInputs(string text, InferStateArgs args)
+text String
args InferStateArgs
Do some post processing after the inference.
+protected abstract bool PostProcess(InferenceParams inferenceParams, InferStateArgs args, IEnumerable`1& extraOutputs)
+inferenceParams InferenceParams
args InferStateArgs
extraOutputs IEnumerable`1&
The core inference logic.
+protected abstract void InferInternal(InferenceParams inferenceParams, InferStateArgs args)
+inferenceParams InferenceParams
args InferStateArgs
Save the current state to a file.
+public abstract void SaveState(string filename)
+filename String
Get the current state data.
+public abstract ExecutorBaseState GetStateData()
+Load the state from data.
+public abstract void LoadState(ExecutorBaseState data)
+data ExecutorBaseState
Load the state from a file.
+public abstract void LoadState(string filename)
+filename String
Execute the inference.
+public IEnumerable<string> Infer(string text, InferenceParams inferenceParams, CancellationToken cancellationToken)
+text String
inferenceParams InferenceParams
cancellationToken CancellationToken
Execute the inference asynchronously.
+public IAsyncEnumerable<string> InferAsync(string text, InferenceParams inferenceParams, CancellationToken cancellationToken)
+text String
inferenceParams InferenceParams
cancellationToken CancellationToken
Namespace: LLama
+This executor infer the input as one-time job. Previous inputs won't impact on the + response to current input.
+public class StatelessExecutor : LLama.Abstractions.ILLamaExecutor
+Inheritance Object → StatelessExecutor
+Implements ILLamaExecutor
The mode used by the executor when running the inference.
+public LLamaModel Model { get; }
+public StatelessExecutor(LLamaModel model)
+model LLamaModel
+The LLama model.
public IEnumerable<string> Infer(string text, InferenceParams inferenceParams, CancellationToken cancellationToken)
+text String
inferenceParams InferenceParams
cancellationToken CancellationToken
public IAsyncEnumerable<string> InferAsync(string text, InferenceParams inferenceParams, CancellationToken token)
+text String
inferenceParams InferenceParams
token CancellationToken