inference-server

Libraries and server to build AI applications doing local inference in node. Use it within your application, or as a microservice. Adapters to llama.cpp via node-llama-cpp and gpt4all, transformers.js and @lmagder/node-stable-diffusion-cpp.

The project includes a model resource pool, an inference queue and a HTTP API server. Model file management is abstracted away as much as possible - configure a URL and go. This package is useful for quick model evaluations and experiments (in JavaScript), small-scale chatbots, resource efficient assistants on edge devices, or any applications where private & offline are interesting criteria. For other - not node-based solutions - check out the related solutions section.

⚠️ This package is currently in beta. APIs may change. Things may break. Help is welcome.

Features

• Configure as many models as you want, they will be downloaded and cached to disk. You may provide them as abs file paths if you already have models downloaded.
• Adjust the pool concurrency, and the models maxInstances, ttl and contextSize to fit your usecase. Combine multiple pools for more complex setups.
• Can be tuned to either use no resources when idle or to always keep a model ready with context preloaded.
• A chat session cache that will effectively reuse context across multiple turns or stateless requests.
• OpenAI spec API endpoints. See HTTP API docs for details. A "native" HTTP API is not yet implemented.
• BYO web server or use the provided express server and middleware. Incoming requests are queued - stall, if needed - and processed as soon as resources are available.
• Have as many ModelServers running as you want, they can share the same cache directory. (Multiple processes can as well)
• Use the ModelPool class directly for a lowerlevel transaction-like API to aquire/release model instances.
• Use custom engines to combine multiple models (or do RAG) behind the scenes.

Usage

Example with minimal configuration:

import { InferenceServer } from 'inference-server'

const localModels = new InferenceServer({
  log: 'info', // default is 'warn'
  models: {
    'my-model': { // Identifiers can use a-zA-Z0-9_:\-\.
      // Required are `task`, `engine`, `url` and/or `file`.
      task: 'text-completion', // text-completion models can be used for chat and text generation tasks
      engine: 'node-llama-cpp', // each engine comes with a peer dep. `npm install node-llama-cpp@3`
      url: 'https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF/blob/main/smollm2-1.7b-instruct-q4_k_m.gguf',
    },
  },
})
await localModels.start()
const result = await localModels.processChatCompletionTask({
  model: 'my-model',
  messages: [
    {
      role: 'user',
      content: 'Why are bananas rather blue than bread at night?',
    },
  ],
})
console.debug(result)
localModels.stop()

Or, to launch an OpenAI-compatible HTTP server with two concurrent instances of the same model:

import { InferenceServer, createExpressServer } from 'inference-server'
import OpenAI from 'openai'

const localModels = new InferenceServer({
  log: 'info', // default is 'warn'
  concurrency: 2, // two inference processes may run at the same time
  models: {
    'smollm': {
      task: 'text-completion',
      engine: 'node-llama-cpp',
      url: 'https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF/blob/main/smollm2-1.7b-instruct-q4_k_m.gguf',
      maxInstances: 2, // two instances of this model may be loaded into memory
      device: {
        cpuThreads: 4, // limit cpu threads so we dont occupy all cores
      }
    },
  },
})
localModels.start()
const httpServer = createExpressServer(localModels)
httpServer.listen(3000)

const client = new OpenAI({
  baseURL: 'http://localhost:3000/openai/v1/',
  apiKey: 'yes',
})
const completion = await client.beta.chat.completions.stream({
  stream_options: { include_usage: true },
  model: 'smollm',
  messages: [
    { role: 'user', content: 'lets count to 10, but only whisper every second number' },
  ],
})
for await (const chunk of completion) {
  if (chunk.choices[0]?.delta?.content) {
    process.stdout.write(chunk.choices[0].delta.content)
  }
}
localModels.stop()
httpServer.close()

In addition to the OpenAI endpoints, the server created by createExpressServer will also display some internal state at /. The endpoint implementations are also exported from the package and can be used with any http server.

More usage examples:

• Using all available options / model options API doc ./examples/all-options.
• Custom engines ./tests/engines/experiments.test.ts.
• A chat cli ./examples/chat-cli.
• concurrency behavior ./examples/concurrency.
• Using the ModelPool directly ./examples/pool.
• Using the express middleware ./examples/express.

Currently supported inference engines are:

| Engine | Peer Dependency | | --- | --- | | node-llama-cpp | node-llama-cpp >= 3.0.0 | | gpt4all | gpt4all >= 4.0.0 | | transformers-js | @huggingface/transformers >= 3.0.0 | | stable-diffusion-cpp | @lmagder/node-stable-diffusion-cpp >= 0.1.6 |

See engine docs for more information on each.

Limitations and Known Issues

Only one model can run on GPU at a time

Llama.cpp bindings currently do not support running multiple models on gpu at the same time. This can/will likely be improved in the future. See GPU docs for more information on how to work around that.

System Messages

System role messages are supported only as the first message in a chat completion session. All other system messages will be ignored. This is only for simplicity reasons and might change in the future.

Chat Context Cache / Reusing the correct instance on stateless requests

Note that the current context cache implementation only works if (apart from the final user message) the same messages are resent in the same order. This is because the messages will be hashed to be compared during follow up turns, to match requests to the correct session. If no hash matches everything will still work, but slower. Because a fresh context will be used and the whole input conversation will be reingested, instead of just the new user message.

Function Calling

Only available when using node-llama-cpp and a model that supports function calling. tool_choice can currently not be controlled and will always be auto. GBNF grammars cannot be used together with function calling.

Huge node_modules when installing all engines

CUDA binaries are distributed with each engine seperately, which leads to an extra 0.5-1GB of disk use. Unfortunately there is nothing I can do about that.

Can't use larger models with transformers.js

Models using the ONNX external data format are currently (not supported in node.js+transformers.js).

TODO / Roadmap

Not in any particular order:

• [x] transformer.js text + image embeddings
• [x] transformer.js multimodal image/text embeddings (see jina-clip-v1 and nomic-embed-vision issues.)
• [x] Allow "prefilling" (partial) assistant responses like outlined here
• [x] non-chat text completions: Allow reuse of context
• [x] non-chat text completions: Support preloading of prefixes
• [x] Add stable-diffusion engine
• [x] Implement more transformer.js tasks
• [x] CLI
• [x] Rename ModelServer -> InferenceServer and refactor task types
• [x] Support for transformer.js text-completion and chat-completion tasks
• [ ] Add some light jsdoc for server/pool/store methods
• [ ] Should support adding a json schema grammar on completion requests (not just a grammar name)
• [ ] utilize node-llama-cpp's support to reuse LlamaModel weights with multiple contexts (across instances)
• [ ] Add n parameter support to node-llama-cpp+transformers chat completions
• [ ] Infill completion support https://github.com/withcatai/node-llama-cpp/blob/beta/src/evaluator/LlamaCompletion.ts#L322-L336
• [ ] Support HF auth and support HF-like model repositories on other domains
• [ ] Rework GPU+device usage / lock (Support multiple models on gpu in cases where its possible)
• [ ] Add engine interfaces for resource use (and estimates, see https://github.com/ggerganov/llama.cpp/issues/4315 and https://github.com/withcatai/node-llama-cpp/blob/beta/src/gguf/insights/utils/resolveContextContextSizeOption.ts)
• [ ] Allow configuring a pools max memory usage
• [ ] Support logits with transformers and node-llama-cpp
• [ ] Test deno/bun support
• [ ] Add image generation endpoint in oai api
• [ ] Add transcript endpoint in oai api
• [ ] Add createSpeech endpoint in oai api
• [ ] Read more model metadata and expose it in API+CLI
• [ ] Replace express with tinyhttp
• [ ] Find a way to type available custom engines (and their options?)

Contributing

If you are using this package - let me know where you would like this to go. Code also welcome. You find things im planning to do (eventually) above, and the wishlist below.

Possible Future Goals

• Create a separate HTTP API thats independent of the OpenAI spec and stateful. See discussion.
• Add a clientside library (React hooks?) for use of above API.
• Provide a Docker image. And maybe a Prometheus endpoint.

Currently not the Goals

• A facade to LLM cloud hoster HTTP API's. The strengths here are local/private/offline use.
• Worry too much about authentication or rate limiting or misuse. Host this with caution, it's likely DDoS-able.
• Some kind of distributed or multi-node setup. That should probably be something designed for this purpose from the ground up.
• Other common related tooling like vector stores, Chat GUIs, etc. Scope would probably get out of hand.

Related Solutions

If you look at this package, you might also want to take a look at these other solutions:

• ollama API - Uses llama.cpp and provides a HTTP API. Also has experimental OpenAI API compatibility.
• llama.cpp Server - The official llama.cpp HTTP API.
• VLLM - A more production ready solution for hosting large language models.
• LM Studio - Also has a local server.
• LocalAI - Similar project in go.
• Petals - Local (and distributed!) inference in python.
• cortex.cpp