Function: tokenizeText()

function tokenizeText(text: string | LlamaText, tokenizer: Tokenizer): Token[]

Tokenize the given input using the given tokenizer, whether it's a string or a LlamaText

Parameters

Parameter	Type
`text`	`string` \| `LlamaText`
`tokenizer`	`Tokenizer`