node-llama-cpp

Appearance

Type Alias: SequenceEvaluateOptions ​

ts
type SequenceEvaluateOptions = {
  temperature: number;
  minP: number;
  topK: number;
  topP: number;
  seed: number;
  grammarEvaluationState:   | LlamaGrammarEvaluationState
     | () => 
     | LlamaGrammarEvaluationState
     | undefined;
  repeatPenalty: LlamaContextSequenceRepeatPenalty;
  tokenBias:   | TokenBias
     | () => TokenBias;
  evaluationPriority: EvaluationPriority;
  contextShift: ContextShiftOptions;
  yieldEogToken: boolean;
};

Defined in: evaluator/LlamaContext/types.ts:276

Properties ​

temperature? ​

ts
optional temperature: number;

Defined in: evaluator/LlamaContext/types.ts:277

minP? ​

ts
optional minP: number;

Defined in: evaluator/LlamaContext/types.ts:277

topK? ​

ts
optional topK: number;

Defined in: evaluator/LlamaContext/types.ts:277

topP? ​

ts
optional topP: number;

Defined in: evaluator/LlamaContext/types.ts:277

seed? ​

ts
optional seed: number;

Defined in: evaluator/LlamaContext/types.ts:288

Used to control the randomness of the generated text.

Change the seed to get different results.

Defaults to the current epoch time.

Only relevant when using temperature.

grammarEvaluationState? ​

ts
optional grammarEvaluationState: 
  | LlamaGrammarEvaluationState
  | () => 
  | LlamaGrammarEvaluationState
  | undefined;

Defined in: evaluator/LlamaContext/types.ts:289

repeatPenalty? ​

ts
optional repeatPenalty: LlamaContextSequenceRepeatPenalty;

Defined in: evaluator/LlamaContext/types.ts:290

tokenBias? ​

ts
optional tokenBias: 
  | TokenBias
  | () => TokenBias;

Defined in: evaluator/LlamaContext/types.ts:297

Adjust the probability of tokens being generated. Can be used to bias the model to generate tokens that you want it to lean towards, or to avoid generating tokens that you want it to avoid.

evaluationPriority? ​

ts
optional evaluationPriority: EvaluationPriority;

Defined in: evaluator/LlamaContext/types.ts:308

When a lot of tokens are queued for the next batch, more than the configured batchSize, the tokens for each sequence will be evaluated based on the strategy chosen for the context. By default, the "maximumParallelism" strategy is used, which will try to evaluate as many sequences in parallel as possible, but at some point, it'll have to choose which sequences to evaluate more tokens of, so it'll prioritize the sequences with the highest evaluation priority. Also, a custom strategy can be used to prioritize the sequences differently, but generally, the higher the evaluation priority is, the more likely and more tokens will be evaluated for that sequence in the next queued batch.

contextShift? ​

ts
optional contextShift: ContextShiftOptions;

Defined in: evaluator/LlamaContext/types.ts:315

Override the sequence context shift options for this evaluation

See ContextShiftOptions for more information.

yieldEogToken? ​

ts
optional yieldEogToken: boolean;

Defined in: evaluator/LlamaContext/types.ts:322

Yield an EOG (End Of Generation) token (like EOS and EOT) when it's generated. When false the generation will stop when an EOG token is generated and the token won't be yielded. Defaults to false.