Type Alias: LlamaOptions
type LlamaOptions = {
gpu: | "auto"
| LlamaGpuType
| {
type: "auto";
exclude: LlamaGpuType[];
};
logLevel: LlamaLogLevel;
logger: (level: LlamaLogLevel, message: string) => void;
build: "auto" | "never" | "forceRebuild" | "try";
cmakeOptions: Record<string, string>;
existingPrebuiltBinaryMustMatchBuildOptions: boolean;
usePrebuiltBinaries: boolean;
progressLogs: boolean;
skipDownload: boolean;
maxThreads: number;
vramPadding: number | (totalVram: number) => number;
ramPadding: number | (totalRam: number) => number;
debug: boolean;
};
Defined in: bindings/getLlama.ts:37
Properties
gpu?
optional gpu:
| "auto"
| LlamaGpuType
| {
type: "auto";
exclude: LlamaGpuType[];
};
Defined in: bindings/getLlama.ts:50
The compute layer implementation type to use for llama.cpp.
"auto"
: Automatically detect and use the best GPU available (Metal on macOS, and CUDA or Vulkan on Windows and Linux)"metal"
: Use Metal. Only supported on macOS. Enabled by default on Apple Silicon Macs."cuda"
: Use CUDA."vulkan"
: Use Vulkan.false
: Disable any GPU support and only use the CPU.
"auto"
by default.
logLevel?
optional logLevel: LlamaLogLevel;
Defined in: bindings/getLlama.ts:59
Set the minimum log level for llama.cpp. Defaults to "warn"
.
logger()?
optional logger: (level: LlamaLogLevel, message: string) => void;
Defined in: bindings/getLlama.ts:64
Set a custom logger for llama.cpp logs.
Parameters
Parameter | Type |
---|---|
level | LlamaLogLevel |
message | string |
Returns
void
build?
optional build: "auto" | "never" | "forceRebuild" | "try";
Defined in: bindings/getLlama.ts:86
Set what build method to use.
"auto"
: If a local build is found, use it. Otherwise, if a prebuilt binary is found, use it. Otherwise, build from source."never"
: If a local build is found, use it. Otherwise, if a prebuilt binary is found, use it. Otherwise, throw aNoBinaryFoundError
error."forceRebuild"
: Always build from source. Be cautious with this option, as it will cause the build to fail on Windows when the binaries are in use by another process."try"
: If a local build is found, use it. Otherwise, try to build from source and use the resulting binary. If building from source fails, use a prebuilt binary if found.
When running from inside an Asar archive in Electron, building from source is not possible, so it'll never build from source. To allow building from source in Electron apps, make sure you ship node-llama-cpp
as an unpacked module.
Defaults to "auto"
. On Electron, defaults to "never"
.
cmakeOptions?
optional cmakeOptions: Record<string, string>;
Defined in: bindings/getLlama.ts:91
Set custom CMake options for llama.cpp
existingPrebuiltBinaryMustMatchBuildOptions?
optional existingPrebuiltBinaryMustMatchBuildOptions: boolean;
Defined in: bindings/getLlama.ts:97
When a prebuilt binary is found, only use it if it was built with the same build options as the ones specified in buildOptions
. Disabled by default.
usePrebuiltBinaries?
optional usePrebuiltBinaries: boolean;
Defined in: bindings/getLlama.ts:103
Use prebuilt binaries if they match the build options. Enabled by default.
progressLogs?
optional progressLogs: boolean;
Defined in: bindings/getLlama.ts:109
Print binary compilation progress logs. Enabled by default.
skipDownload?
optional skipDownload: boolean;
Defined in: bindings/getLlama.ts:116
Don't download llama.cpp source if it's not found. When set to true
, and llama.cpp source is not found, a NoBinaryFoundError
error will be thrown. Disabled by default.
maxThreads?
optional maxThreads: number;
Defined in: bindings/getLlama.ts:127
The maximum number of threads to use for the Llama instance.
Set to 0
to have no thread limit.
When not using a GPU, defaults to the number of CPU cores that are useful for math (.cpuMathCores
), or 4
, whichever is higher.
When using a GPU, there's no limit by default.
vramPadding?
optional vramPadding: number | (totalVram: number) => number;
Defined in: bindings/getLlama.ts:137
Pad the available VRAM for the memory size calculations, as these calculations are not always accurate. Recommended to ensure stability. This only affects the calculations of "auto"
in function options and is not reflected in the getVramState
function.
Defaults to 6%
of the total VRAM or 1GB, whichever is lower. Set to 0
to disable.
ramPadding?
optional ramPadding: number | (totalRam: number) => number;
Defined in: bindings/getLlama.ts:148
Pad the available RAM for the memory size calculations, as these calculations are not always accurate. Recommended to ensure stability.
Defaults to 25%
of the total RAM or 6GB (1GB on Linux), whichever is lower. Set to 0
to disable.
Since the OS also needs RAM to function, the default value can get up to 6GB on Windows and macOS, and 1GB on Linux.
debug?
optional debug: boolean;
Defined in: bindings/getLlama.ts:158
Enable debug mode to find issues with llama.cpp. Makes logs print directly to the console from llama.cpp
and not through the provided logger.
Defaults to false
.
The default can be set using the NODE_LLAMA_CPP_DEBUG
environment variable.