Function: tokenizeText()

function tokenizeText(text: string | LlamaText, tokenizer: Tokenizer): Token[];

Defined in: utils/LlamaText.ts:597

Tokenize the given input using the given tokenizer, whether it's a string or a LlamaText

Parameters

Parameter	Type
text	string | LlamaText
tokenizer	Tokenizer

Returns

Token[]