Class: InputLookupTokenPredictor

Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:22

Attempts to find the last few generated tokens in the input (prompt) tokens to predict the next tokens.

This is useful in input-grounded tasks (when the model frequently repeats some of the input tokens in the output, such as in text summarization or modifying code).

This works in all completion classes, including LlamaChatSession, LlamaChat, and LlamaCompletion.

Based on https://github.com/apoorvumang/prompt-lookup-decoding.

See

Using Token Predictors: Input Lookup Token Predictor

Extends

• TokenPredictor

Constructors

Constructor

new InputLookupTokenPredictor(options?: {
  patternLength?: {
     min?: number;
     max?: number;
  };
  predictionLength?: {
     min?: number;
     max?: number;
  };
}): InputLookupTokenPredictor;

Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:33

Parameters

Parameter	Type	Description
options	{ patternLength?: { min?: number; max?: number; }; predictionLength?: { min?: number; max?: number; }; }	-
options.patternLength?	{ min?: number; max?: number; }	-
options.patternLength.min?	number	Min pattern length to look for in the input tokens. Defaults to 1.
options.patternLength.max?	number	Max pattern length to look for in the input tokens. Set to 0 to disable the max pattern size. Defaults to 0.
options.predictionLength?	{ min?: number; max?: number; }	-
options.predictionLength.min?	number	Minimum number of tokens to predict. Defaults to 1.
options.predictionLength.max?	number	Maximum number of tokens to predict. Defaults to 3.

Returns

InputLookupTokenPredictor

Overrides

TokenPredictor.constructor

Accessors

patternMinLength

Get Signature

get patternMinLength(): number;

Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:86

Returns

number

---

patternMaxLength

Get Signature

get patternMaxLength(): number;

Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:90

Returns

number

---

predictionMinLength

Get Signature

get predictionMinLength(): number;

Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:94

Returns

number

---

predictionMaxLength

Get Signature

get predictionMaxLength(): number;

Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:98

Returns

number

Methods

stop()

stop(untilPredictionsExhausted?: boolean): 
  | void
  | Promise<void>;

Defined in: evaluator/LlamaContext/TokenPredictor.ts:52

Stops the prediction process when it runs in the background.

Parameters

Parameter	Type	Description
untilPredictionsExhausted?	boolean	If true, the prediction process should not resume until the current predictions are exhausted.

Returns

| void | Promise<void>

Inherited from

TokenPredictor.stop

---

reset()

reset(__namedParameters: {
  stateTokens: Token[];
}): void;

Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:102

Resets the state of the predictor.

Called before the generation starts.

Parameters

Parameter	Type
__namedParameters	{ stateTokens: Token[]; }
__namedParameters.stateTokens	Token[]

Returns

void

Overrides

TokenPredictor.reset

---

updateInputTokens()

updateInputTokens(tokens: Token[]): void;

Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:110

Called with the input tokens before the generation starts when using LlamaChatSession, LlamaChat, and LlamaCompletion.

Parameters

Parameter	Type
tokens	Token[]

Returns

void

Overrides

TokenPredictor.updateInputTokens

---

pushTokens()

pushTokens(tokens: Token[]): void;

Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:116

Parameters

Parameter	Type
tokens	Token[]

Returns

void

Overrides

TokenPredictor.pushTokens

---

predictTokens()

predictTokens(): Token[];

Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:124

Predicts the next tokens based on the current state.

If the generation should wait until the minimum predications are ready, this method should return a promise that resolves when the minimum predictions are ready.

A background prediction process can be started when this function is called, so that the next predictions will be ready when this function is called again.

Returns

Token[]

Overrides

TokenPredictor.predictTokens

---

dispose()

dispose(): void;

Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:169

Returns

void

Overrides

TokenPredictor.dispose