generative-ts

a typescript library for building LLM applications+agents

Install

To install everything:

npm i generative-ts

You can also do more granular installs of scoped packages if you want to optimize your builds further (see packages)

Usage

AWS Bedrock

API docs: createAwsBedrockModelProvider

import {
  AmazonTitanTextApi,
  createAwsBedrockModelProvider
} from "generative-ts";

// Bedrock supports many different APIs and models. See API docs (above) for full list.
const titanText = createAwsBedrockModelProvider({
  api: AmazonTitanTextApi,
  modelId: "amazon.titan-text-express-v1",
  // If your code is running in an AWS Environment (eg, Lambda) authorization will happen automatically. Otherwise, explicitly pass in `auth`
});

const response = await titanText.sendRequest({ 
  $prompt:"Brief history of NY Mets:" 
  // all other options for the specified `api` available here
});

console.log(response.results[0]?.outputText);

Cohere

API docs: createCohereModelProvider

import { createCohereModelProvider } from "generative-ts";

const commandR = createCohereModelProvider({
  modelId: "command-r-plus", // Cohere defined model ID
  // you can explicitly pass auth here, otherwise by default it is read from process.env
});

const response = await commandR.sendRequest({
  $prompt:"Brief History of NY Mets:",
  preamble: "Talk like Jafar from Aladdin",
  // all other Cohere /generate options available here
});

console.log(response.text);

Google Cloud VertexAI

API docs: createVertexAiModelProvider

import { createVertexAiModelProvider } from "@packages/gcloud-vertex-ai";

const gemini = await createVertexAiModelProvider({
  modelId: "gemini-1.0-pro", // VertexAI defined model ID
  // you can explicitly pass auth here, otherwise by default it is read from process.env
});

const response = await gemini.sendRequest({
  $prompt:"Brief History of NY Mets:",
  // all other Gemini options available here
});

console.log(response.data.candidates[0]);

Groq

API docs: createGroqModelProvider

import { createGroqModelProvider } from "generative-ts";

const llama3 = createGroqModelProvider({
  modelId: "llama3-70b-8192", // Groq defined model ID
  // you can explicitly pass auth here, otherwise by default it is read from process.env
});

const response = await llama3.sendRequest({ 
  $prompt:"Brief History of NY Mets:" 
  // all other OpenAI ChatCompletion options available here (Groq uses the OpenAI ChatCompletion API for all the models it hosts)
});

console.log(response.choices[0]?.message.content);

Huggingface Inference

API docs: createHuggingfaceInferenceModelProvider

import { 
  createHuggingfaceInferenceModelProvider, 
  HfTextGenerationTaskApi 
} from "generative-ts";

// Huggingface Inference supports many different APIs and models. See API docs (above) for full list.
const gpt2 = createHuggingfaceInferenceModelProvider({
  api: HfTextGenerationTaskApi,
  modelId: "gpt2",
  // you can explicitly pass auth here, otherwise by default it is read from process.env
});

const response = await gpt2.sendRequest({ 
  $prompt:"Hello," 
  // all other options for the specified `api` available here
});

console.log(response[0]?.generated_text);

LMStudio

API docs: createLmStudioModelProvider

import { createLmStudioModelProvider } from "generative-ts";

const llama3 = createLmStudioModelProvider({
  modelId: "lmstudio-community/Meta-Llama-3-70B-Instruct-GGUF", // a ID of a model you have downloaded in LMStudio
});

const response = await llama3.sendRequest({ 
  $prompt:"Brief History of NY Mets:" 
  // all other OpenAI ChatCompletion options available here (LMStudio uses the OpenAI ChatCompletion API for all the models it hosts)
});

console.log(response.choices[0]?.message.content);

Mistral

API docs: createMistralModelProvider

import { createMistralModelProvider } from "generative-ts";

const mistralLarge = createMistralModelProvider({
  modelId: "mistral-large-latest", // Mistral defined model ID
  // you can explicitly pass auth here, otherwise by default it is read from process.env
});

const response = await mistralLarge.sendRequest({ 
  $prompt:"Brief History of NY Mets:" 
  // all other Mistral ChatCompletion API options available here
});

console.log(response.choices[0]?.message.content);

OpenAI

API docs: createOpenAiChatModelProvider

import { createOpenAiChatModelProvider } from "generative-ts";

const gpt = createOpenAiChatModelProvider({
  modelId: "gpt-4-turbo", // OpenAI defined model ID
  // you can explicitly pass auth here, otherwise by default it is read from process.env
});

const response = await gpt.sendRequest({
  $prompt:"Brief History of NY Mets:",
  max_tokens: 100,
  // all other OpenAI ChatCompletion options available here
});

console.log(response.choices[0]?.message.content);

Custom HTTP Client

todo;

Supported Providers and Models

See Usage for how to use each provider.

|Provider|Models|Model APIs| |-|-|-| |AWS Bedrock|Multiple hosted models|Native model APIs| |Cohere|Command / Command R+|Cohere /generate and /chat| |Google Vertex AI|Gemini x.y|Gemini; OpenAI in preview| |Groq|Multiple hosted models|OpenAI ChatCompletion| |Huggingface Inference|Open-source|Huggingface Inference APIs| |LMStudio (localhost)|Open-source (must be downloaded)|OpenAI ChatCompletion| |Mistral|Mistral x.y|Mistral ChatCompletion| |OpenAI|GPT x.y|OpenAI ChatCompletion| |Azure (coming soon)|| |Replicate (coming soon)|| |Anthropic (coming soon)|| |Fireworks (coming soon)||

It's also easy to add your own TODO LINK

Packages

If you're using a modern bundler, just install generative-ts to get everything. Modern bundlers support tree-shaking, so your final bundle won't include unused code. (Note: we distribute both ESM and CJS bundles for compatibility.) If you prefer to avoid unnecessary downloads, or you're operating under constraints where tree-shaking isn't an option, we offer scoped packages under @generative-ts/ with specific functionality for more fine-grained installs.

|Package|Description|| |-|-|-| | generative-ts | Everything | Includes all scoped packages listed below | | @generative-ts/core | Core functionality (zero dependencies) | Interfaces, classes, utilities, etc | | @generative-ts/gcloud-vertex-ai | Google Cloud VertexAI ModelProvider | Uses Application Default Credentials (ADC) to properly authenticate in GCloud environments | | @generative-ts/aws-bedrock | AWS Bedrock ModelProvider | Uses aws4 to properly authenticate when running in AWS environments |

Report Bugs / Submit Feature Requests

Please submit all issues here: https://github.com/Econify/generative-ts/issues

Contributing

To get started developing, optionally fork and then clone the repository and run:

nvm use
npm ci

To run examples and integration/e2e tests, create an .env file by running cp .env.example .env and then add values where necessary

Publishing

The "main" generative-ts package and the scoped @generative-ts packages both are controlled by the generative-ts npm organization. Releases are published via circleci job upon pushes of tags that have a name starting with release/. The job requires an NPM token that has publishing permissions to both generative-ts and @generative-ts. Currently this is a "granular" token set to expire every 30 days, created by @jnaglick, set in a circleci context.