genkitx-promptfoo is a community plugin for using Promptfoo with Firebase Genkit. Built by Yuki Nagae.

This Genkit plugin allows to use Promptfoo.

[!WARNING] This version is experimental and may have API changes and critical bugs. Use it for prototypes or hobby projects, not in production.

Installation

Install the plugin in your project with your favorite package manager:

• npm install genkitx-promptfoo
• yarn add genkitx-promptfoo
• pnpm add genkitx-promptfoo

Usage

Configuration

import { promptfooEval } from 'genkitx-promptfoo';

configureGenkit({
  plugins: [
    promptfooEval({
      metrics: [
        {
          type: 'contains',
          value: 'Hello, World!',
        },
        {
          type: 'regex',
          value: '^Hello, World!$',
        },
        {
          type: 'javascript',
          value: "output.includes('Hello, World!')",
        },
        {
          type: 'similar',
          value: 'Aloha, World!',
          threshold: 0.8,
          provider: 'vertex:embedding:text-embedding-004',
        },
        {
          type: 'llm-rubric',
          value: 'It is a friendly greeting.',
          provider: 'vertex:gemini-1.5-flash',
        },
      ],
    }),
  ],
});

Basic examples

You can specify the Promptfoo metrics below:

• contains / icontains

{
  type: 'contains',
  value: 'The expected substring',
},

• contains-all

{
  type: 'contains-all',
  value: [
   'Value 1',
   'Value 2',
   'Value 3',
  ],
},

• contains-any

{
  type: 'contains-any',
  value: [
   'Value 1',
   'Value 2',
   'Value 3',
  ],
},

• regex

{
  type: 'regex',
  value: '\\d{4}', // Matches a 4-digit number
},

• contains-json / is-json

{
  type: 'contains-json',
  value:
  {
    'required': ['latitude', 'longitude'],
    'type': 'object',
    'properties':
      {
        'latitude': { 'type': 'number', 'minimum': -90, 'maximum': 90 },
        'longitude': { 'type': 'number', 'minimum': -180, 'maximum': 180 },
      },
  }
},

• contains-sql / is-sql

To use this assertion, you need to install the node-sql-parser package. You can install it using npm: npm install node-sql-parser.

{
  type: 'contains-sql',
},

• equals

{
  type: 'equals',
  value: 'The expected output',
},

• contains-xml / is-xml

{
  type: 'contains-xml',
},

• javascript

{
  type: 'javascript',
  value: "output.includes('Hello, World!')",
},

• starts-with

{
  type: 'starts-with',
  value: 'Yes',
},

• similar

{
  type: 'similar',
  value: 'The expected output',
  threshold: 0.8,
  provider: 'vertex:embedding:text-embedding-004',
},

• llm-rubric

{
  type: 'llm-rubric',
  value: 'Is not apologetic and provides a clear, concise answer',
  provider: 'vertex:gemini-1.5-flash',
},

Contributing

Want to contribute to the project? That's awesome! Head over to our Contribution Guidelines.

Need support?

[!NOTE]
This repository depends on Google's Firebase Genkit. For issues and questions related to Genkit, please refer to instructions available in Genkit's repository.

Reach out by opening a discussion on Github Discussions.

Credits

This plugin is proudly maintained by Yuki Nagae Yuki Nagae.

I got the inspiration, structure and patterns to create this plugin from the Genkit Community Plugins repository built by the Fire Compnay as well as the github plugin.

License

This project is licensed under the Apache 2.0 License.