Building from source
node-llama-cpp
ships with pre-built binaries for macOS, Linux and Windows.
In case binaries are not available for your platform or fail to load, it'll fallback to download a release of llama.cpp
and build it from source with cmake
.
Downloading a release
To download a release of llama.cpp
and build it from source you can use the CLI download
command.
npx --no node-llama-cpp download
npx --no node-llama-cpp download
NOTE
node-llama-cpp
ships with a git bundle of the release of llama.cpp
it was built with, so when you run the download
command without specifying a specific release or repo, it will use the bundled git bundle instead of downloading the release from GitHub.
This is useful for building from source on machines that aren't connected to the internet.
INFO
If cmake
is not installed on your machine, node-llama-cpp
will automatically download cmake
to an internal directory and try to use it to build llama.cpp
from source.
If the build fails, make sure you have the required dependencies of cmake
installed on your machine. More info is available here (you don't have to install cmake
or cmake-js
, just the dependencies).
If the build fails on macOS with the error "/usr/bin/cc" is not able to compile a simple test program
, try running xcode-select --install
to install the Xcode command line tools.
download
and build
commands
The difference between the download
and build
commands is that the download
command downloads a release of llama.cpp
and builds it, while the build
command builds the llama.cpp
release that's already downloaded.
You can only use the build
command after you've already downloaded a release of llama.cpp
with the download
command.
To only download a release of llama.cpp
without building it, use the download
command with the --skipBuild
option:
npx --no node-llama-cpp download --skipBuild
npx --no node-llama-cpp download --skipBuild
Customizing the build
Meta: To configure Metal support see the Metal support guide.
CUDA: To configure CUDA support see the CUDA support guide.
llama.cpp
has cmake build options that can be configured to customize the build. You can find documentation for these options here.
To build node-llama-cpp
with any of these options, set an environment variable of an option prefixed with NODE_LLAMA_CPP_CMAKE_OPTION_
before running the download
or build
commands.
Downloading a newer release
Every new release of node-llama-cpp
ships with the latest release of llama.cpp
that was available at the time of the release, so relying on the latest version of node-llama-cpp
should be enough for most use cases.
However, you may want to download a newer release of llama.cpp
(llama.cpp
releases) and build it from source to get the latest features and bug fixes before a new version of node-llama-cpp
is released.
A new release may contain breaking changes, so it won't necessarily work properly or even compile at all, so do this with caution.
You can do this by specifying the --release
option with the release tag you want to download:
npx --no node-llama-cpp download --release "b1350"
npx --no node-llama-cpp download --release "b1350"
You can find the release tag on the
llama.cpp
releases page:
You can also opt to download the latest release available:
npx --no node-llama-cpp download --release latest
npx --no node-llama-cpp download --release latest