node-llama-cpp

Appearance

Type Alias: LlamaDecisionContextOptions ​

ts
type LlamaDecisionContextOptions = {
  chatWrapper?: "auto" | ChatWrapper;
  contextSize?:   | "auto"
     | number
     | {
     min?: number;
     max?: number;
   };
  batchSize?: number;
  parallelQuestions?: number;
  threads?: number;
  createSignal?: AbortSignal;
  ignoreMemorySafetyChecks?: boolean;
};

Defined in: evaluator/LlamaDecisionContext/LlamaDecisionContext.ts:17

Properties ​

chatWrapper? ​

ts
optional chatWrapper: "auto" | ChatWrapper;

Defined in: evaluator/LlamaDecisionContext/LlamaDecisionContext.ts:19

"auto" is used by default

contextSize? ​

ts
optional contextSize: 
  | "auto"
  | number
  | {
  min?: number;
  max?: number;
};

Defined in: evaluator/LlamaDecisionContext/LlamaDecisionContext.ts:33

The number of tokens the model can see at once.

  • "auto" - adapt to the current VRAM state and attempt to set the context size as high as possible up to the size the model was trained on.
  • number - set the context size to a specific number of tokens. If there's not enough VRAM, an error will be thrown. Use with caution.
  • {min?: number, max?: number} - adapt to the current VRAM state and attempt to set the context size as high as possible up to the size the model was trained on, but at least min and at most max.

Defaults to {max: 4096}.

batchSize? ​

ts
optional batchSize: number;

Defined in: evaluator/LlamaDecisionContext/LlamaDecisionContext.ts:39

prompt processing batch size

parallelQuestions? ​

ts
optional parallelQuestions: number;

Defined in: evaluator/LlamaDecisionContext/LlamaDecisionContext.ts:46

The number of questions to support evaluating in parallel.

Defaults to 4.

threads? ​

ts
optional threads: number;

Defined in: evaluator/LlamaDecisionContext/LlamaDecisionContext.ts:52

number of threads to use to evaluate tokens. set to 0 to use the maximum threads supported by the current machine hardware

createSignal? ​

ts
optional createSignal: AbortSignal;

Defined in: evaluator/LlamaDecisionContext/LlamaDecisionContext.ts:55

An abort signal to abort the context creation

ignoreMemorySafetyChecks? ​

ts
optional ignoreMemorySafetyChecks: boolean;

Defined in: evaluator/LlamaDecisionContext/LlamaDecisionContext.ts:63

Ignore insufficient memory errors and continue with the context creation. Can cause the process to crash if there's not enough VRAM for the new context.

Defaults to false.