LLamaSharp/docs/Examples/BatchedExecutorFork.md

BatchedExecutor Fork - Generate Multiple Completions With Shared Memory

This example demonstrates using the BatchedExecutor to split one sequence into multiple sequences. See the source code here.

Sequences share memory up to the point they were split, meaning no extra memory is consumed by creating a fork. Inference runs for all sequences simultaneously, this means that running two sequences does not take twice as much time as running one.

An example output, starting with the prompt Not many people know that:

Not many people know that
└── , in the 17th century, a military band led by Captain Charles
    ├──  Bossler of Baden, Germany, composed and played a music suite titled
    │   ├──  the "Civil Psalm," in order to rally German Protestants during
    │   │   ├──  the Thirty Years' War.  This tune became popular among German soldiers,
    │   │   │   ├──  and its popularity continued long after the war
    │   │   │   └──  and, eventually, reached France. The
    │   │   └──  the Thirty Years' War.This music, with its clear call
    │   │       ├──  to arms and strong Christian themes, helped
    │   │       └──  to arms and unwavering belief
    │   └──  "Baden's First National Symphony," with lyrics by a young Wol
    │       ├── fgang Amadeus Mozart. The story of the composition's creation
    │       │   ├──  has long been forgotten. But the B
    │       │   └──  was popularized by a novelty book
    │       └── fgang Amadeus Mozart. It's said that this music brought
    │           ├──  peace to Europe, at least for a
    │           └──  the troops together during difficult times. It
    └──  Newdick played a mournful dirge to accompany the procession of
        ├──  the head of King Charles I. It is the scene that opens my latest book
        │   ├── , "Death and Taxes." The book follows a British army captain named
        │   │   ├──  Marcus as he seeks revenge for his wife
        │   │   └──  William Darnay who becomes involved in
        │   └── , A King, A Pawn and a Prince. The murder of the king
        │       ├──  and the civil war that followed are the
        │       └──  is a watershed moment in the political
        └──  the coffin of William Shakespeare, as it was carried to its final resting place
            ├── . That is the least that can be said for a man who is often regarded
            │   ├──  as the greatest writer in the English language
            │   └──  as the greatest writer the English language has
            └──  at Stratford-upon-Avon.  Shakespeare, of course
                ├── , was a famous English poet and play
                └── , was one of the greatest playwright

Forked sequences can be used for many possible things. For example

• Evaluating the system prompt once and forking for each independent conversation.
• Saving a "checkpoint" in a conversation to return to later.
• Beam Search.
• Splitting a conversation, generating completions from several different "agents", and taking the best response.