node-llama-cpp

Appearance

Type Alias: SequenceEvaluateOptions ​

ts
type SequenceEvaluateOptions = {
  temperature?: number;
  minP?: number;
  topK?: number;
  topP?: number;
  seed?: number;
  grammarEvaluationState?:   | LlamaGrammarEvaluationState
     | () => 
     | LlamaGrammarEvaluationState
     | undefined;
  repeatPenalty?: LlamaContextSequenceRepeatPenalty;
  tokenBias?:   | TokenBias
     | () => TokenBias;
  evaluationPriority?: EvaluationPriority;
  contextShift?: ContextShiftOptions;
  yieldEogToken?: boolean;
};

Defined in: evaluator/LlamaContext/types.ts:297

Properties ​

temperature? ​

ts
optional temperature: number;

Defined in: evaluator/LlamaContext/types.ts:298

minP? ​

ts
optional minP: number;

Defined in: evaluator/LlamaContext/types.ts:298

topK? ​

ts
optional topK: number;

Defined in: evaluator/LlamaContext/types.ts:298

topP? ​

ts
optional topP: number;

Defined in: evaluator/LlamaContext/types.ts:298

seed? ​

ts
optional seed: number;

Defined in: evaluator/LlamaContext/types.ts:309

Used to control the randomness of the generated text.

Change the seed to get different results.

Defaults to the current epoch time.

Only relevant when using temperature.

grammarEvaluationState? ​

ts
optional grammarEvaluationState: 
  | LlamaGrammarEvaluationState
  | () => 
  | LlamaGrammarEvaluationState
  | undefined;

Defined in: evaluator/LlamaContext/types.ts:310

repeatPenalty? ​

ts
optional repeatPenalty: LlamaContextSequenceRepeatPenalty;

Defined in: evaluator/LlamaContext/types.ts:311

tokenBias? ​

ts
optional tokenBias: 
  | TokenBias
  | () => TokenBias;

Defined in: evaluator/LlamaContext/types.ts:318

Adjust the probability of tokens being generated. Can be used to bias the model to generate tokens that you want it to lean towards, or to avoid generating tokens that you want it to avoid.

evaluationPriority? ​

ts
optional evaluationPriority: EvaluationPriority;

Defined in: evaluator/LlamaContext/types.ts:329

When a lot of tokens are queued for the next batch, more than the configured batchSize, the tokens for each sequence will be evaluated based on the strategy chosen for the context. By default, the "maximumParallelism" strategy is used, which will try to evaluate as many sequences in parallel as possible, but at some point, it'll have to choose which sequences to evaluate more tokens of, so it'll prioritize the sequences with the highest evaluation priority. Also, a custom strategy can be used to prioritize the sequences differently, but generally, the higher the evaluation priority is, the more likely and more tokens will be evaluated for that sequence in the next queued batch.

contextShift? ​

ts
optional contextShift: ContextShiftOptions;

Defined in: evaluator/LlamaContext/types.ts:336

Override the sequence context shift options for this evaluation

See ContextShiftOptions for more information.

yieldEogToken? ​

ts
optional yieldEogToken: boolean;

Defined in: evaluator/LlamaContext/types.ts:343

Yield an EOG (End Of Generation) token (like EOS and EOT) when it's generated. When false the generation will stop when an EOG token is generated and the token won't be yielded. Defaults to false.