@friendliai/ai-provider

Learn how to use the FriendliAI provider for the Vercel AI SDK.

Installation

You can install the package via npm:

npm i @friendliai/ai-provider

Credentials

The tokens required for model usage can be obtained from the Friendli suite.

To use the provider, you need to set the FRIENDLI_TOKEN environment variable with your personal access token.

export FRIENDLI_TOKEN="YOUR_FRIENDLI_TOKEN"

Check the FriendliAI documentation for more information.

Provider Instance

import { friendli } from "@friendliai/ai-provider";

Language Models

You can create FriendliAI models using a provider instance. The first argument is the model id, e.g. meta-llama-3.1-8b-instruct.

const model = friendli("meta-llama-3.1-8b-instruct");

Example: Generating text

You can use FriendliAI language models to generate text with the generateText function:

import { friendli } from "@friendliai/ai-provider";
import { generateText } from 'ai'

const { text } = await generateText({
  model: friendli('meta-llama-3.1-8b-instruct')
  prompt: 'What is the meaning of life?',
})

Example: Using Enforcing Patterns (Regex)

Specify a specific pattern (e.g., CSV), character sets, or specific language characters (e.g., Korean Hangul characters) for your LLM's output.

import { friendli } from "@friendliai/ai-provider";
import { generateText } from "ai";

const { text } = await generateText({
  model: friendli("meta-llama-3.1-8b-instruct", {
    regex: "[\n ,.?!0-9\uac00-\ud7af]*",
  }),
  maxTokens: 40,
  prompt: "who is the first king of the Joseon Dynasty?",
});

console.log(text);

Example: Using built-in tools (Beta)

If you use @friendliai/ai-provider, you can use the built-in tools via the tools option.

Built-in tools allow models to use tools to generate better answers. For example, a web:search tool can provide up-to-date answers to current questions.

import { friendli } from "@friendliai/ai-provider";
import { convertToCoreMessages, streamText } from "ai";

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = await streamText({
    model: friendli("meta-llama-3.1-8b-instruct", {
      tools: [
        { type: "web:search" },
        { type: "math:calculator" },
        { type: "code:python-interpreter" }, // and more tools..!!
      ],
    }),
    messages: convertToCoreMessages(messages),
  });

  return result.toDataStreamResponse();
}

FriendliAI language models can also be used in the streamText, generateObject, streamObject, and streamUI functions. (see AI SDK Core and AI SDK RSC).

OpenAI Compatibility

We can also use @ai-sdk/openai with OpenAI compatibility.

import { createOpenAI } from "@ai-sdk/openai";

const friendli = createOpenAI({
  baseURL: "https://api.friendli.ai/serverless/v1",
  apiKey: process.env.FRIENDLI_TOKEN,
});