Function: tokenizeText()
ts
function tokenizeText(text: string | LlamaText, tokenizer: Tokenizer): Token[]Defined in: utils/LlamaText.ts:594
Tokenize the given input using the given tokenizer, whether it's a string or a LlamaText
Parameters
| Parameter | Type |
|---|---|
text | string | LlamaText |
tokenizer | Tokenizer |
Returns
Token[]