node-llama-cpp

Appearance

Type Alias: SequenceEvaluateOptions ​

ts
type SequenceEvaluateOptions = {
  temperature: number;
  minP: number;
  topK: number;
  topP: number;
  seed: number;
  grammarEvaluationState:   | LlamaGrammarEvaluationState
     | () => 
     | LlamaGrammarEvaluationState
     | undefined;
  repeatPenalty: LlamaContextSequenceRepeatPenalty;
  tokenBias:   | TokenBias
     | () => TokenBias;
  evaluationPriority: EvaluationPriority;
  contextShift: ContextShiftOptions;
  yieldEogToken: boolean;
};

Defined in: evaluator/LlamaContext/types.ts:276

Type declaration ​

temperature? ​

ts
optional temperature: number;

minP? ​

ts
optional minP: number;

topK? ​

ts
optional topK: number;

topP? ​

ts
optional topP: number;

seed? ​

ts
optional seed: number;

Used to control the randomness of the generated text.

Change the seed to get different results.

Defaults to the current epoch time.

Only relevant when using temperature.

grammarEvaluationState? ​

ts
optional grammarEvaluationState: 
  | LlamaGrammarEvaluationState
  | () => 
  | LlamaGrammarEvaluationState
  | undefined;

repeatPenalty? ​

ts
optional repeatPenalty: LlamaContextSequenceRepeatPenalty;

tokenBias? ​

ts
optional tokenBias: 
  | TokenBias
  | () => TokenBias;

Adjust the probability of tokens being generated. Can be used to bias the model to generate tokens that you want it to lean towards, or to avoid generating tokens that you want it to avoid.

evaluationPriority? ​

ts
optional evaluationPriority: EvaluationPriority;

When a lot of tokens are queued for the next batch, more than the configured batchSize, the tokens for each sequence will be evaluated based on the strategy chosen for the context. By default, the "maximumParallelism" strategy is used, which will try to evaluate as many sequences in parallel as possible, but at some point, it'll have to choose which sequences to evaluate more tokens of, so it'll prioritize the sequences with the highest evaluation priority. Also, a custom strategy can be used to prioritize the sequences differently, but generally, the higher the evaluation priority is, the more likely and more tokens will be evaluated for that sequence in the next queued batch.

contextShift? ​

ts
optional contextShift: ContextShiftOptions;

Override the sequence context shift options for this evaluation

See ContextShiftOptions for more information.

yieldEogToken? ​

ts
optional yieldEogToken: boolean;

Yield an EOG (End Of Generation) token (like EOS and EOT) when it's generated. When false the generation will stop when an EOG token is generated and the token won't be yielded. Defaults to false.