node-llama-cpp

Run AI models locally on your machine

_{Pre-built bindings are provided with a fallback to building from source with cmake}

✨ v3.0 is here! ✨

Features

• Run LLMs locally on your machine
• Metal, CUDA and Vulkan support
• Pre-built binaries are provided, with a fallback to building from source without node-gyp or Python
• Adapts to your hardware automatically, no need to configure anything
• A Complete suite of everything you need to use LLMs in your projects
• Use the CLI to chat with a model without writing any code
• Up-to-date with the latest llama.cpp. Download and compile the latest release with a single CLI command
• Enforce a model to generate output in a parseable format, like JSON, or even force it to follow a specific JSON schema
• Provide a model with functions it can call on demand to retrieve information of perform actions
• Embedding support
• Great developer experience with full TypeScript support, and complete documentation
• Much more

Documentation

• Getting started guide
• API reference
• CLI help
• Blog
• Changelog
• Roadmap

Try It Without Installing

Chat with a model in your terminal using a single command:

1npx -y node-llama-cpp chat

Installation

1npm install node-llama-cpp

This package comes with pre-built binaries for macOS, Linux and Windows.

If binaries are not available for your platform, it'll fallback to download a release of llama.cpp and build it from source with cmake. To disable this behavior, set the environment variable NODE_LLAMA_CPP_SKIP_DOWNLOAD to true.

Usage

1import {fileURLToPath} from "url";
2import path from "path";
3import {getLlama, LlamaChatSession} from "node-llama-cpp";
4
5const __dirname = path.dirname(fileURLToPath(import.meta.url));
6
7const llama = await getLlama();
8const model = await llama.loadModel({
9    modelPath: path.join(__dirname, "models", "Meta-Llama-3.1-8B-Instruct.Q4_K_M.gguf")
10});
11const context = await model.createContext();
12const session = new LlamaChatSession({
13    contextSequence: context.getSequence()
14});
15
16
17const q1 = "Hi there, how are you?";
18console.log("User: " + q1);
19
20const a1 = await session.prompt(q1);
21console.log("AI: " + a1);
22
23
24const q2 = "Summarize what you said";
25console.log("User: " + q2);
26
27const a2 = await session.prompt(q2);
28console.log("AI: " + a2);

For more examples, see the getting started guide

Contributing

To contribute to node-llama-cpp read the contribution guide.

Acknowledgements

• llama.cpp: ggerganov/llama.cpp

If you like this repo, star it ✨