205 lines
5.3 KiB
Markdown
205 lines
5.3 KiB
Markdown
# LLamaBatch
|
|
|
|
Namespace: LLama.Native
|
|
|
|
A batch allows submitting multiple tokens to multiple sequences simultaneously
|
|
|
|
```csharp
|
|
public class LLamaBatch
|
|
```
|
|
|
|
Inheritance [Object](https://docs.microsoft.com/en-us/dotnet/api/system.object) → [LLamaBatch](./llama.native.llamabatch.md)
|
|
|
|
## Properties
|
|
|
|
### **TokenCount**
|
|
|
|
The number of tokens in this batch
|
|
|
|
```csharp
|
|
public int TokenCount { get; private set; }
|
|
```
|
|
|
|
#### Property Value
|
|
|
|
[Int32](https://docs.microsoft.com/en-us/dotnet/api/system.int32)<br>
|
|
|
|
### **SequenceCapacity**
|
|
|
|
Maximum number of sequences a token can be assigned to (automatically grows if exceeded)
|
|
|
|
```csharp
|
|
public int SequenceCapacity { get; private set; }
|
|
```
|
|
|
|
#### Property Value
|
|
|
|
[Int32](https://docs.microsoft.com/en-us/dotnet/api/system.int32)<br>
|
|
|
|
## Constructors
|
|
|
|
### **LLamaBatch()**
|
|
|
|
Create a new batch for submitting inputs to llama.cpp
|
|
|
|
```csharp
|
|
public LLamaBatch()
|
|
```
|
|
|
|
## Methods
|
|
|
|
### **ToNativeBatch(LLamaNativeBatch&)**
|
|
|
|
```csharp
|
|
internal GroupDisposable ToNativeBatch(LLamaNativeBatch& batch)
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
`batch` [LLamaNativeBatch&](./llama.native.llamanativebatch&.md)<br>
|
|
|
|
#### Returns
|
|
|
|
[GroupDisposable](./llama.native.groupdisposable.md)<br>
|
|
|
|
### **Add(LLamaToken, LLamaPos, ReadOnlySpan<LLamaSeqId>, Boolean)**
|
|
|
|
Add a single token to the batch at the same position in several sequences
|
|
|
|
```csharp
|
|
public int Add(LLamaToken token, LLamaPos pos, ReadOnlySpan<LLamaSeqId> sequences, bool logits)
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
`token` [LLamaToken](./llama.native.llamatoken.md)<br>
|
|
The token to add
|
|
|
|
`pos` [LLamaPos](./llama.native.llamapos.md)<br>
|
|
The position to add it att
|
|
|
|
`sequences` [ReadOnlySpan<LLamaSeqId>](https://docs.microsoft.com/en-us/dotnet/api/system.readonlyspan-1)<br>
|
|
The set of sequences to add this token to
|
|
|
|
`logits` [Boolean](https://docs.microsoft.com/en-us/dotnet/api/system.boolean)<br>
|
|
|
|
#### Returns
|
|
|
|
[Int32](https://docs.microsoft.com/en-us/dotnet/api/system.int32)<br>
|
|
The index that the token was added at. Use this for GetLogitsIth
|
|
|
|
**Remarks:**
|
|
|
|
https://github.com/ggerganov/llama.cpp/blob/ad939626577cd25b462e8026cc543efb71528472/common/common.cpp#L829C2-L829C2
|
|
|
|
### **Add(LLamaToken, LLamaPos, List<LLamaSeqId>, Boolean)**
|
|
|
|
Add a single token to the batch at the same position in several sequences
|
|
|
|
```csharp
|
|
public int Add(LLamaToken token, LLamaPos pos, List<LLamaSeqId> sequences, bool logits)
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
`token` [LLamaToken](./llama.native.llamatoken.md)<br>
|
|
The token to add
|
|
|
|
`pos` [LLamaPos](./llama.native.llamapos.md)<br>
|
|
The position to add it att
|
|
|
|
`sequences` [List<LLamaSeqId>](https://docs.microsoft.com/en-us/dotnet/api/system.collections.generic.list-1)<br>
|
|
The set of sequences to add this token to
|
|
|
|
`logits` [Boolean](https://docs.microsoft.com/en-us/dotnet/api/system.boolean)<br>
|
|
|
|
#### Returns
|
|
|
|
[Int32](https://docs.microsoft.com/en-us/dotnet/api/system.int32)<br>
|
|
The index that the token was added at. Use this for GetLogitsIth
|
|
|
|
**Remarks:**
|
|
|
|
https://github.com/ggerganov/llama.cpp/blob/ad939626577cd25b462e8026cc543efb71528472/common/common.cpp#L829C2-L829C2
|
|
|
|
### **Add(LLamaToken, LLamaPos, LLamaSeqId, Boolean)**
|
|
|
|
Add a single token to the batch at a certain position for a single sequences
|
|
|
|
```csharp
|
|
public int Add(LLamaToken token, LLamaPos pos, LLamaSeqId sequence, bool logits)
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
`token` [LLamaToken](./llama.native.llamatoken.md)<br>
|
|
The token to add
|
|
|
|
`pos` [LLamaPos](./llama.native.llamapos.md)<br>
|
|
The position to add it att
|
|
|
|
`sequence` [LLamaSeqId](./llama.native.llamaseqid.md)<br>
|
|
The sequence to add this token to
|
|
|
|
`logits` [Boolean](https://docs.microsoft.com/en-us/dotnet/api/system.boolean)<br>
|
|
|
|
#### Returns
|
|
|
|
[Int32](https://docs.microsoft.com/en-us/dotnet/api/system.int32)<br>
|
|
The index that the token was added at. Use this for GetLogitsIth
|
|
|
|
**Remarks:**
|
|
|
|
https://github.com/ggerganov/llama.cpp/blob/ad939626577cd25b462e8026cc543efb71528472/common/common.cpp#L829C2-L829C2
|
|
|
|
### **AddRange(ReadOnlySpan<LLamaToken>, LLamaPos, LLamaSeqId, Boolean)**
|
|
|
|
Add a range of tokens to a single sequence, start at the given position.
|
|
|
|
```csharp
|
|
public int AddRange(ReadOnlySpan<LLamaToken> tokens, LLamaPos start, LLamaSeqId sequence, bool logitsLast)
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
`tokens` [ReadOnlySpan<LLamaToken>](https://docs.microsoft.com/en-us/dotnet/api/system.readonlyspan-1)<br>
|
|
The tokens to add
|
|
|
|
`start` [LLamaPos](./llama.native.llamapos.md)<br>
|
|
The starting position to add tokens at
|
|
|
|
`sequence` [LLamaSeqId](./llama.native.llamaseqid.md)<br>
|
|
The sequence to add this token to
|
|
|
|
`logitsLast` [Boolean](https://docs.microsoft.com/en-us/dotnet/api/system.boolean)<br>
|
|
Whether the final token should generate logits
|
|
|
|
#### Returns
|
|
|
|
[Int32](https://docs.microsoft.com/en-us/dotnet/api/system.int32)<br>
|
|
The index that the final token was added at. Use this for GetLogitsIth
|
|
|
|
### **Clear()**
|
|
|
|
Set TokenCount to zero for this batch
|
|
|
|
```csharp
|
|
public void Clear()
|
|
```
|
|
|
|
### **GetLogitPositions(Span<ValueTuple<LLamaSeqId, Int32>>)**
|
|
|
|
Get the positions where logits can be sampled from
|
|
|
|
```csharp
|
|
internal Span<ValueTuple<LLamaSeqId, int>> GetLogitPositions(Span<ValueTuple<LLamaSeqId, int>> dest)
|
|
```
|
|
|
|
#### Parameters
|
|
|
|
`dest` [Span<ValueTuple<LLamaSeqId, Int32>>](https://docs.microsoft.com/en-us/dotnet/api/system.span-1)<br>
|
|
|
|
#### Returns
|
|
|
|
[Span<ValueTuple<LLamaSeqId, Int32>>](https://docs.microsoft.com/en-us/dotnet/api/system.span-1)<br>
|