node-llama-cpp

Appearance

Type Alias: SequenceEvaluateOptions ​

ts
type SequenceEvaluateOptions = {
  temperature?: number;
  minP?: number;
  topK?: number;
  topP?: number;
  seed?: number;
  grammarEvaluationState?:   | LlamaGrammarEvaluationState
     | () => 
     | LlamaGrammarEvaluationState
     | undefined;
  repeatPenalty?: LlamaContextSequenceRepeatPenalty;
  tokenBias?:   | TokenBias
     | () => TokenBias;
  evaluationPriority?: EvaluationPriority;
  contextShift?: ContextShiftOptions;
  yieldEogToken?: boolean;
};

Defined in: evaluator/LlamaContext/types.ts:292

Properties ​

temperature? ​

ts
optional temperature: number;

Defined in: evaluator/LlamaContext/types.ts:293

minP? ​

ts
optional minP: number;

Defined in: evaluator/LlamaContext/types.ts:293

topK? ​

ts
optional topK: number;

Defined in: evaluator/LlamaContext/types.ts:293

topP? ​

ts
optional topP: number;

Defined in: evaluator/LlamaContext/types.ts:293

seed? ​

ts
optional seed: number;

Defined in: evaluator/LlamaContext/types.ts:304

Used to control the randomness of the generated text.

Change the seed to get different results.

Defaults to the current epoch time.

Only relevant when using temperature.

grammarEvaluationState? ​

ts
optional grammarEvaluationState: 
  | LlamaGrammarEvaluationState
  | () => 
  | LlamaGrammarEvaluationState
  | undefined;

Defined in: evaluator/LlamaContext/types.ts:305

repeatPenalty? ​

ts
optional repeatPenalty: LlamaContextSequenceRepeatPenalty;

Defined in: evaluator/LlamaContext/types.ts:306

tokenBias? ​

ts
optional tokenBias: 
  | TokenBias
  | () => TokenBias;

Defined in: evaluator/LlamaContext/types.ts:313

Adjust the probability of tokens being generated. Can be used to bias the model to generate tokens that you want it to lean towards, or to avoid generating tokens that you want it to avoid.

evaluationPriority? ​

ts
optional evaluationPriority: EvaluationPriority;

Defined in: evaluator/LlamaContext/types.ts:324

When a lot of tokens are queued for the next batch, more than the configured batchSize, the tokens for each sequence will be evaluated based on the strategy chosen for the context. By default, the "maximumParallelism" strategy is used, which will try to evaluate as many sequences in parallel as possible, but at some point, it'll have to choose which sequences to evaluate more tokens of, so it'll prioritize the sequences with the highest evaluation priority. Also, a custom strategy can be used to prioritize the sequences differently, but generally, the higher the evaluation priority is, the more likely and more tokens will be evaluated for that sequence in the next queued batch.

contextShift? ​

ts
optional contextShift: ContextShiftOptions;

Defined in: evaluator/LlamaContext/types.ts:331

Override the sequence context shift options for this evaluation

See ContextShiftOptions for more information.

yieldEogToken? ​

ts
optional yieldEogToken: boolean;

Defined in: evaluator/LlamaContext/types.ts:338

Yield an EOG (End Of Generation) token (like EOS and EOT) when it's generated. When false the generation will stop when an EOG token is generated and the token won't be yielded. Defaults to false.