Class: InputLookupTokenPredictor
Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:22
Attempts to find the last few generated tokens in the input (prompt) tokens to predict the next tokens.
This is useful in input-grounded tasks (when the model frequently repeats some of the input tokens in the output, such as in text summarization or modifying code).
This works in all completion classes, including LlamaChatSession
, LlamaChat
, and LlamaCompletion
.
Based on https://github.com/apoorvumang/prompt-lookup-decoding.
See
Using Token Predictors: Input Lookup Token Predictor
Extends
Constructors
Constructor
new InputLookupTokenPredictor(options: {
patternLength?: {
min?: number;
max?: number;
};
predictionLength?: {
min?: number;
max?: number;
};
}): InputLookupTokenPredictor;
Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:33
Parameters
Parameter | Type | Description |
---|---|---|
options | { patternLength? : { min? : number ; max? : number ; }; predictionLength? : { min? : number ; max? : number ; }; } | - |
options.patternLength? | { min? : number ; max? : number ; } | - |
options.patternLength.min? | number | Min pattern length to look for in the input tokens. Defaults to 1 . |
options.patternLength.max? | number | Max pattern length to look for in the input tokens. Set to 0 to disable the max pattern size. Defaults to 0 . |
options.predictionLength? | { min? : number ; max? : number ; } | - |
options.predictionLength.min? | number | Minimum number of tokens to predict. Defaults to 1 . |
options.predictionLength.max? | number | Maximum number of tokens to predict. Defaults to 3 . |
Returns
InputLookupTokenPredictor
Overrides
Accessors
patternMinLength
Get Signature
get patternMinLength(): number;
Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:86
Returns
number
patternMaxLength
Get Signature
get patternMaxLength(): number;
Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:90
Returns
number
predictionMinLength
Get Signature
get predictionMinLength(): number;
Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:94
Returns
number
predictionMaxLength
Get Signature
get predictionMaxLength(): number;
Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:98
Returns
number
Methods
stop()
stop(untilPredictionsExhausted?: boolean):
| void
| Promise<void>;
Defined in: evaluator/LlamaContext/TokenPredictor.ts:52
Stops the prediction process when it runs in the background.
Parameters
Parameter | Type | Description |
---|---|---|
untilPredictionsExhausted? | boolean | If true, the prediction process should not resume until the current predictions are exhausted. |
Returns
| void
| Promise
<void
>
Inherited from
reset()
reset(__namedParameters: {
stateTokens: Token[];
}): void;
Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:102
Resets the state of the predictor.
Called before the generation starts.
Parameters
Parameter | Type |
---|---|
__namedParameters | { stateTokens : Token []; } |
__namedParameters.stateTokens | Token [] |
Returns
void
Overrides
updateInputTokens()
updateInputTokens(tokens: Token[]): void;
Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:110
Called with the input tokens before the generation starts when using LlamaChatSession
, LlamaChat
, and LlamaCompletion
.
Parameters
Parameter | Type |
---|---|
tokens | Token [] |
Returns
void
Overrides
TokenPredictor
.updateInputTokens
pushTokens()
pushTokens(tokens: Token[]): void;
Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:116
Parameters
Parameter | Type |
---|---|
tokens | Token [] |
Returns
void
Overrides
predictTokens()
predictTokens(): Token[];
Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:124
Predicts the next tokens based on the current state.
If the generation should wait until the minimum predications are ready, this method should return a promise that resolves when the minimum predictions are ready.
A background prediction process can be started when this function is called, so that the next predictions will be ready when this function is called again.
Returns
Token
[]
Overrides
dispose()
dispose(): void;
Defined in: evaluator/LlamaContext/tokenPredictors/InputLookupTokenPredictor.ts:169
Returns
void