npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

pretzelduck

v2.2.1

Published

Use natural language to write end-to-end tests in Playwright.

Downloads

116

Readme

Pretzel Duck

Use natural language to write end-to-end tests in Playwright.

import { PretzelDuck, openai } from "pretzelduck";
import { test } from "@playwright/test";

const { describe, beforeEach } = test;

const { it } = new PretzelDuck(test, openai("gpt-4o"));

describe("search functionality", () => {
	beforeEach(async ({ page }) => {
		await page.goto("https://github.com/");
	});

	it(
		"should find and favorite the Pretzel Duck repository",
		"the favorite button should be highlighted yellow on a GitHub repository titled Pretzel Duck",
	);
});

Why

End-to-end (E2E) testing in its current form is often not practical for various reasons:

  • Writing end-to-end tests takes a long time.
  • Often, in order to use end-to-end tests effectively, you need to maintain a mock API or have control over your environment's data.
  • Seemingly small UI changes can break end-to-end tests, requiring you spend additional time fixing them.

From the author:

As an engineer at John Deere, my team tirelessly works to meet feature deadlines and deliver bug fixes. Moreover, we don't control the data in our quality assurance environment. Though we wanted to save integration testing time, the investment required to write end-to-end tests has never proven to be worth it. The Pretzel Duck project was a shower thought of mine to help my team overcome this opportunity cost. Though it can be quite expensive to run an LLM, especially in this fashion, the engineering time outweighs this cost ten-fold.

Features

  • Simple to use with good defaults
    • Just provide a natural language goal and assertion for every test
  • Interactions such as clicking, inputting text, scrolling, and hitting enter
  • Highly configurable with a BYOM (bring your own model) approach
  • Use existing Playwright DX features like tracing and UI
  • Saves engineering time

Getting Started

I don't have an existing Playwright test suite

Before proceeding, please follow the introductory Playwright documentation to create your first test suite. This usually takes around 5-10 minutes.

Next, modify your playwright.config.ts to increase the test timeout time. Because Pretzel Duck repeatedly makes calls to your model of choice, testing times can become lengthy. As you refine your goal prompts, you may be able to decrease the testing suite timeout. We recommend a minute to start.

export default defineConfig({
    // ...
    timeout: 60000
    // ...
});

I have an existing Playwright test suite

You'll first need to install Pretzel Duck via npm.

npm i pretzelduck

You'll also need to install the AI SDK of your choice. This example uses OpenAI.

Next, create a pretzelduck.ts file that's accessible from your testing suite.

import { PretzelDuck } from 'pretzelduck';
import { openai } from '@ai-sdk/openai';
import { test } from '@playwright/test';

/*
    The `model` expected by Pretzel Duck is identical to those created by the Vercel AI SDK.
    We support any model that allows for multi-modal input (image and text).
    
    This example uses `openai`, but others are available.
    You will need to provide an API key using the respective environment variables.
*/
const model = openai('gpt-4o');

export const pretzelDuck = new PretzelDuck(test, model);

In your testing suite, you're now ready to write your first natural language end-to-end test! Open up an existing .spec.ts or .test.ts file, or use the one example.spec.ts generated by Playwright.

import { test } from "@playwright/test";
import { pretzelDuck } from "./pretzelduck";

// Setup the page using the base Playwright `test` utility.
test.beforeEach(async ({ page }) => {
	await page.goto("https://www.example.com/");
});

pretzelDuck.test(
	"find more information about protocol registries", // goal
	"you should be on a page titled Protocol Registries", // assertion indicating the goal has been reached
);

That's it, test away! In your console, enter the following:

npx playwright test --ui

From the UI, run the respective test and see the model perform in real-time.

Advanced

Though Pretzel Duck strives to be a 0-configuration solution, more advanced use cases may require some fine-tuning. Fortunately, you can configure Pretzel Duck instance-by-instance and test-by-test.

The following options can be provided as a trailing parameter on both the PretzelDuck class and the test function:

type TestOptions = {
	interactions: {
		maxInteractions: number;
		disabled: InteractionLabels[];
		interactables: {
			disabledElements: InteractableElements[];
			disabledRoles: InteractableRoles[];
		};
	};
	assertions: {
		maxRetries: number;
		temperature: number;
		immediate: boolean;
	};
	decisions: {
		maxRetries: number;
		progressions: {
			enabled: boolean;
			type: 'forced';
		};
		temperature: number;
		maxEntropy: number;
		useScreenshots: boolean;
		useVisibleHtml: boolean;
		useHistory: boolean;
	};
};

Here's an example of the configuration in use:

import { PretzelDuck } from "pretzelduck";
import { test } from "@playwright/test";

export const pretzelDuck = new PretzelDuck(test, /* Your Model Here */, {
	interactions: {
		disabled: ["submit"],
		interactables: {
			disabledElements: ["select"],
			disabledRoles: ["menuitem"],
		},
	},
	decisions: {
		temperature: 0.8,
	},
});

pretzelDuck.test(
	"find the e-mail address of the website owner",
	"an e-mail address should be visible on screen",
	{
		decisions: {
			progressions: {
				enabled: false,
			},
			useVisibleHtml: false,
		},
	},
);

Architecture

image