NODE-LLAMA

Run llama cpp locally inside your Node environment.

Build Status

| OS | Node 18 | Node 20 | Node 22 | |---------|---------|---------|---------| | Ubuntu | | | | | macOS | | | | | Windows | | | |

Package Info

Reasoning

Sometimes you just need a small model that can run anywhere and can't be bothered with making REST calls to services like OpenRouter or Ollama. This project is super simple, NodeJS native inference based on llamacpp project and with no need for external services.

Install NPM, download a model, and run it. Simple as.

Features

• Minimal dependencies (mostly CMake and GCC) and no need for external services
• High performance, full speed of llamacpp with a thin layer of Node
• Supports most LLM models
• Easy to use API
• Command line for direct inference and model download

Installation

npm install @duck4i/llama

Please note that you need CMake and GCC installed if you don't have it already, as the plugin is cpp based.

sudo apt-get install -y build-essential cmake g++

Usage

import { RunInference } = from "@duck4i/llama";

const system_prompt = "The following is a conversation with an AI assistant. The assistant is helpful, creative, clever, and very friendly.";
const user_prompt = "What is life expectancy of a duck?";

const inference = RunInference("model.gguf", user_prompt, system_prompt, /*optional*/ 512);

console.log("Answer", inference);

It is likely you will want async functions for better memory management with multiple prompts, which is done like this:

import { LoadModelAsync, CreateContextAsync, RunInferenceAsync, ReleaseContextAsync, ReleaseModelAsync } = from "@duck4i/llama";

const system_prompt = "The following is a conversation with an AI assistant. The assistant is helpful, creative, clever, and very friendly.";
const prompts = [
    "How old can ducks get?",
    "Why are ducks so cool?",
    "Is there a limit on number of ducks I can own?"
]

const model = await LoadModelAsync("model.gguf");
const ctx = await CreateContextAsync(model, /*optional n_ctx*/ 0, /*optional flash_att*/ true);
console.log("Model loaded", model);

for (const prompt of prompts) {
    const inference = await RunInferenceAsync(model, ctx, prompt, system_prompt, /*optional max tokens*/ 512);
    console.log("Answer:", inference);
}

await ReleaseContextAsync(model);
await ReleaseModelAsync(model);

Model format

Its likely you will want more control over the model, so you can push the complete formatted prompt to it with prefix !#, like this:

const system = "You are ...";
const user = "...";

//  QWEN example (prefix !# will get removed before reaching the llm)
const prompt = `"!#<|im_start|>system ${system}<|im_end|><|im_start|>user ${user}<|im_end|><|im_start|>assistant"`;

const reply = await RunInferenceAsync(modelHandle, prompt, /*optional max token*/ 128)

Getting tokens from model is done by GetModelToken method.

const eos = GetModelToken(modelHandle, "EOS");
const bos = GetModelToken(modelHandle, "BOS");
const eot = GetModelToken(modelHandle, "EOT");
const sep = GetModelToken(modelHandle, "SEP");
const cls = GetModelToken(modelHandle, "CLS");
const nl = GetModelToken(modelHandle, "NL");

Logging control

You can control log levels coming from llamacpp like this:

import { SetLogLevel } = from '@duck4i/llama';

// 0 - none, 1 - debug, 2 - info, 3 - warn, 4 - error
SetLogLevel(1);

Command line

# Download model
npx llama-download -u https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-fp16.gguf?download=true -p model.gguf

# Run inference
npx llama-run -m model.gguf -p "How old can ducks get?"

# Run with system prompt
npx llama-run -m model.gguf -p "How old can ducks get?" -s "[System prompt...]"

Supported Models

All models supported by llamacpp natively are supported here too, so do check their repository.

Please keep in mind that CUDA is not enabled yet due to complex dependencies so keep the model size in check.

On MacOS, the Metal backend should come included.

Contributing

Contributions are welcome! Please open an issue or submit a pull request.

License

This project is licensed under the MIT License.