Safetest: Next Generation UI Testing Library

Safetest is a powerful UI testing library that combines Playwright, Jest/Vitest, and React for a powerful end-to-end testing solution for applications and component testing. With Safetest, you can easily test the functionality and appearance of your application, ensuring that it works as expected and looks great on all devices.

Safetest provides a seamless testing experience by integrating with your existing development environment and offering a familiar, easy-to-use API for creating and managing tests.

Features

• Playwright Integration: Run your tests on real browsers using Playwright. Safetest automatically handles browser management, so you can focus on just writing tests.
  • Screenshot diffing via jest-image-snapshot
  • Video recording
  • Trace Viewer
  • Full control over network layer
  • Powerful overrides for complex test cases
• Jest Integration: Safetest leverages the Jest test runner. Write your tests using familiar Jest syntax and benefit from its powerful assertion library and mocking capabilities.
• Vitest Integration: Safetest can also use the Vitest runner. If you have a vite project you'll probably want to use this
• React Support: Safetest is designed with React applications in mind, so you can easily test your components and their interactions. This allows for focused testing of individual components, for Example testing that <Header admin={true}> behaves as expected.
• Framework agnostic: Safetest also works with other frameworks like Vue, Svelte, and Angular. See the examples folder for more details. Safetest even works to component test a NextJS application
• Easy Setup: Safetest is easy to set up and configure, so you can start writing tests in no time. No need to worry about complex configurations or dependencies; Safetest takes care of it all.
• Easy Auth Hooks: If your app is testing an authenticated application, Safetest provides hooks to handles the auth flow in a reusable method across all your tests.

Why Safetest?

Fundamentally UI tests come in two flavors: Integration test and E2E test.

Integration tests are usually ran via react-testing-library or similar. They are fast, easy to write, and test the components within an application. However they are limited in that they don't actually run the application or render actual items to a screen so regressions like having a bad z-index that causes the submit button to be un-clickable won't be caught by these tests. Another common issue is that while it's easy to write the setup for the test, it's hard to write the events needed to cause things on the page to happen. For example, displaying a fancy <Dropdown /> isn't as simple as just calling fireEvent.click('select') since the js-dom doesn't perfectly match the real browser, so you end up needing to mouseover the label and then click the select. Along the same lines figuring out how to enter text on a smart <Input /> has a similar battle. Figuring out the exact incantation to make this happen is hard and brittle. Debugging why something stopped working is also hard since you can't just open the browser and see what's going on.

E2E tests like Cypress and Playwright are great for testing the actual application. They use a real browser and run against the actual application. They're able to test things like z-index issues, etc. However they lack the ability to test components in isolation, which is why some teams will end up having a Storybook adjacent build to point the E2E test at. Another issue is that it's hard to setup the different test fixtures. For example, if you want to test that an admin user has an edit button on the page while a regular user doesn't, you'll find some ways to override the auth service to return different results. Similarly, component testing isn't possible when we have external service dependencies like OAuth since Cypress and Playwright component testing do not run against an actual instance of the app, so any auth gating can make rendering components impossible.

Essentially we end up with this breakdown:

| Integration Tests Pros | Integration Test Cons | E2E Test Pros | E2E Test Cons | | --------------------------- | ------------------------------------- | -------------------------------- | ----------------------------- | | Easy setup | Hard to drive | Easy to drive | Hard to setup | | Fast | Hard to debug | Easy to debug | Slow | | Mock services | Hard to override network | Easy to override network | No service mocking | | | Can't test clickability of element | Can test clickability of element | | | | Can't test z-index | Can test z-index | | | | Can't easily set value on <Input /> | Easy to set value on <Input /> | | | | No screenshot testing | Screenshot testing | | | | No video recording | Video recording | | | | No trace viewer | Trace viewer | | | No confidence app works E2E | Confidence components work | Confidence app works E2E | No confidence components work |

It's almost like the two are complementary, if only there was a way to combine the two...

This is essentially what Safetest is trying to solve. It's a way to combine the best of both worlds. It allows us to write tests that are easy to setup, easy to drive, and can test the components in isolation, while also being able to test the application as a whole, test clickability of elements, and do screenshot testing, video recording, trace viewer, etc..

Consider this example:

// Header.tsx
export const Header = ({ admin }: { admin?: boolean }) => (
  <div className='header'>
    <div className='header-title'>The App</div>
    <div className='header-user'>
      <div className='header-user-name'>admin</div>
      {admin && <div className='header-user-admin'>admin</div>}
      <div className='header-user-logout'>Logout</div>
    </div>
  </div>
);

// Header.safetest.tsx
import { describe, it, expect } from 'safetest/jest';
import { render } from 'safetest/react';
import { Header } from './Header';

describe('Header', () => {
  it('can render a regular header', async () => {
    const { page } = await render(<Header />);
    await expect(page.locator('text=Logout')).toBeVisible();
    await expect(page.locator('text=admin')).not.toBeVisible();
    expect(await page.screenshot()).toMatchImageSnapshot();
  });

  it('can render an admin header', async () => {
    const { page } = await render(<Header admin={true} />);
    await expect(page.locator('text=Logout')).toBeVisible();
    await expect(page.locator('text=admin')).toBeVisible();
    expect(await page.screenshot()).toMatchImageSnapshot();
  });
});

SafeTest artifacts and reports

SafeTest can output a report of the tests that were run, including a trace viewer of each test, a video of the test execution, and a link to open the tested component in the deployed environment. SafeTest tests SafeTest with SafeTest.

See the reports that is generated.

See the Reporting section for more details.

Getting Started

Note: To quickly try the one of the example app in the project, follow the SafeTest Development section below.

To get started with Safetest, follow these steps:

1. Install Safetest as a dependency in your project:

   npm install --save-dev safetest

   Or, if you're using Yarn:

   yarn add --dev safetest

   The following instructions assume you're using create-react-app. Look in the examples folder for other setup configurations.

2. Add run command to package.json scripts:

   Add the following line to your package.json scripts section:

   {
     "scripts": {
       "safetest": "OPT_URL=${TARGET_URL:-http://localhost:3000} react-scripts --inspect test --runInBand --testMatch '**/*.safetest.{j,t}s{,x}' --setupFilesAfterEnv ./setup-safetest.tsx",
       "safetest:ci": "rm -f artifacts.json && OPT_URL=${DEPLOYED_URL} OPT_CI=1 OPT_DOCKER=1 npm run safetest -- --watchAll=false --ci=1 --json --outputFile=results.json",
       "safetest:regenerate-screenshots": "OPT_DOCKER=1 npm run safetest -- --watchAll=false --update-snapshot"
     }
   }

   The preceding script runs the default runner (react-scripts) with a couple of flags and environment variables to make sure Safetest is loaded and run with jest and that all .safetest.tsx test files are tested. You may need to adjust based on your specific setup, e.g., using craco or react-app-rewired instead.

   ---

   If you're using Vitest you'd use these instead:

   {
     "scripts": {
       "safetest": "OPT_URL=${OPT_URL:-http://localhost:3000/} vitest --config vite.safetest.config",
       "safetest:ci": "rm -f artifacts.json && OPT_URL=${DEPLOYED_URL} OPT_CI=1 OPT_DOCKER=1 OPT_ARTIFACTS=artifacts.json npm run safetest -- --run --bail=5",
       "safetest:regenerate-screenshots": "OPT_DOCKER=1 npm run safetest -- --run --update"
     }
   }

   ---

   A note about the OPT_URL and similar variables. This is used to pass flags to Safetest which will flow through different testing frameworks spawning threads or any other mechanism that would make command line flag passing practically impossible</s,all>

3. Add setup-safetest.tsx file:

   Create a file called setup-safetest.tsx in the root of your project and add the following code:

   import { setup } from 'safetest/setup';
   
   setup({
     bootstrappedAt: require.resolve('./src/main.tsx'),
   });

   This file is the minimal setup required to get Safetest working with your project. It's also where you can configure Safetest by specifying options to the setup function.

   If you're using vitest you'll need to add a vitest.safetest.config.ts file with the following config:

   /// <reference types="vitest" />
   
   import { defineConfig } from 'vite';
   
   // https://vitejs.dev/config/
   export default defineConfig({
     test: {
       globals: true,
       testTimeout: 30000,
       reporters: ['basic', 'json'],
       outputFile: 'results.json',
       setupFiles: ['setup-safetest'],
       include: ['**/*.safetest.?(c|m)[jt]s?(x)'],
       threads: process.env.CI ? true : false,
       inspect: process.env.CI ? false : true,
     },
   });

4. Bootstrapping your application

   In order for Safetest to be able to work with your application, you need to bootstrap it to load. This is done by modifying your application's entry point (usually src/index.tsx) as follows:

    import ReactDOM from "react-dom";
   +import { bootstrap } from 'safetest/react';
    import App from "./App";
   
   -ReactDOM.render(
   -  <App />,
   -  document.getElementById("app")
   -);
   +const container = document.getElementById("app");
   +const element = <App />;
   +
   +const isDev = process.env.NODE_ENV !== 'production';
   +
   +bootstrap({
   +  element,
   +  render: (element) => ReactDOM.render(element, container),
   +  // If using React 18:
   +  // render: (element) => ReactDOM.createRoot(container).render(element),
   +
   +  // Add one of the following depending on your bundler...
   +
   +  // Webpack:
   +  webpackContext: isDev && import.meta.webpackContext('.', {
   +    recursive: true,
   +    regExp: /\.safetest$/,
   +    mode: 'lazy'
   +  })
   +
   +  // Vite:
   +  // importGlob: isDev && import.meta.glob('./**/*.safetest.{j,t}s{,x}'),
   +
   +  // Using the `npx safetest generate-import-map src/Bootstrap.tsx src > src/imports.tsx` syntax:
   +  // imports, // Where imports is defined as `import imports from './imports';`
   +
   +  // Other:
   +  // import: isDev && async (s) => import(`${s.replace(/.*src/, '.').replace(/\.safetest$/, '')}.safetest`),
   +
   +});

   The above magic import makes use of Webpack Context Or Vite Glob import (or whatever flavor dynamic import is available) to bundle the .safetest.tsx files in your project separately. This allows you to write tests for your application in the same project as your application, without having to worry about setting up a separate test project or about the tests being loaded when loading your application in a non-test context. The isDev check is only really needed if you don't want to leak your tests into production, but it's not strictly necessary. In this project it's turned off for the examples since I want to test against the final deployed app to keep things simple.

5. Creating your first tests

   Now that you've set up Safetest, you can start writing your first tests. Create a file called src/App.safetest.tsx and add the following code:

   import { describe, it, expect } from 'safetest/jest';
   import { render } from 'safetest/react';
   
   import { Header } from './Header';
   
   // Whole App testing
   describe('App', () => {
     it('renders without crashing', async () => {
       const { page } = await render();
       await expect(page.locator('text=Welcome to The App')).toBeVisible();
     });
   });
   
   // Component testing
   describe('Header', () => {
     it('renders without crashing', async () => {
       const { page } = await render(<Header />);
       await expect(page.locator('text=Logout')).toBeVisible();
       expect(await page.screenshot()).toMatchImageSnapshot();
     });
   });

6. Running your tests

   Important: Your app needs to already be running to run the tests. Safetest will not start your app for you!

   Now that you've created your first tests, you can run it using the following command:

   npm run safetest

   Additionally, you can pass it a a number of custom options via environment variables prefixed with OPT_. (This is needed since the test runner may run in subprocesses and command line args aren't always passed through but environment variables always are.) Some examples: OPT_HEADED, OPT_URL, OPT_ARTIFACTS, OPT_DOCKER, OPT_CI, etc. For example to see the browser window while the tests are running:

   OPT_HEADED=1 npm run safetest

7. Integrating into CI

   Assuming part of your CI pipeline deploys the app to some url https://my-app.com, you can add a step to the CI pipeline by either adding a script to manually invoke the following:

   OPT_CI=1 OPT_DOCKER=1 OPT_URL=https://my-app.com npm run safetest -- --watchAll=false --ci=1 --json --outputFile=results.json

See the Reporting section about how to get an HTML report of the results with links to:

• Trace viewer of each test
• Video of test execution
• Ability to open tested component in the deployed environment

See here for a the reports (and apps) of each the example projects.

Writing Tests

Since Safetest is bootstrapped within the application, essentially every test is a component test. If you don't specify a component in render, then it will just render the default component (for example <App /> in the getting started section). render also allows passing a function which will be called with the default component as an argument. This is useful for overriding props or wrapping the component in a provider.

If you just want to test your application as a whole, you can use this syntax

const { page } = await render();

and just pretend everything after that line is a @playwright/test test.

The following section showcases a couple of common testing scenarios:

Testing a component

import { describe, it, expect } from 'safetest/jest';
import { render } from 'safetest/react';
import { Header } from './Header';

describe('Header', () => {
  it('can render a regular header', async () => {
    const { page } = await render(<Header />);
    await expect(page.locator('text=Logout')).toBeVisible();
    await expect(page.locator('text=admin')).not.toBeVisible();
    expect(await page.screenshot()).toMatchImageSnapshot();
  });

  it('can render an admin header', async () => {
    const { page } = await render(<Header admin={true} />);
    await expect(page.locator('text=Logout')).toBeVisible();
    await expect(page.locator('text=admin')).toBeVisible();
    expect(await page.screenshot()).toMatchImageSnapshot();
  });
});

Snapshot testing

Safetest comes out of the box with snapshot testing enabled via jest-image-snapshot. A simple example of this is shown above. You can also mask over or remove DOM elements before the snapshot to have deterministic tests. A common scenario for this is to remove a date field from the UI before taking a snapshot, since the value will be different every time and will cause the screenshots not to match.

import { describe, it, expect } from 'safetest/jest';
import { render } from 'safetest/react';

describe('Snapshot', () => {
  it('works with date fields', async () => {
    const { page } = await render();
    await page.evaluate(() => document.querySelector('.header-date')?.remove());
    expect(await page.screenshot()).toMatchImageSnapshot();
  });
});

There is also a mask option you can pass to page.screenshot({ mask: ... }); however that only covers over the element. If the element's width changes across tests, the snapshot diffs will still fail.

Deterministic snapshots

Due to hardware and platform differences between dev machines and CI environments, there will be slight rendering differences between snapshots generated locally and in CI. To solve this problem and to ensure that a consistent and reproducible test setup is used, Safetest can run your tests in a docker container. This should be used in CI by default (via the safetest:ci script). To run your tests in docker locally or to generate updated snapshots which will match CI, you can run:
OPT_DOCKER=1 npm run safetest and yarn safetest:regenerate-screenshots respectively.

Note that you can also run this on "headed" mode, which will open a browser window connected to the debugPort within the docker container as show below:

Mocks and spies

Safetest also has the ability to provide mocks and spies to component props which you can assert against in your tests. This is useful for testing that components behave as you'd expect.

import { describe, it, expect, browserMock } from 'safetest/jest';
import { render } from 'safetest/react';
import { Header } from './Header';

describe('Header', () => {
  /* ... */

  it('calls the passed logout handler when clicked', async () => {
    const spy = browserMock.fn();
    const { page } = await render(<Header handleLogout={spy} />);
    await page.locator('text=Logout').click();
    expect(await spy).toHaveBeenCalled();
  });
});

Communicating between node and the browser.

In order to make Safetest work, the test code is run in both node and the browser (see the How Safetest works section for more details about this). What this means is that we have full control over both what happens in node as well as the browser as the test is running. This allows us to do some powerful communication between the two environments. One of these items is the ability to make assertions in node from the browser as seen above (the await spy was not a typo, it's also type safe so don't worry about forgetting it). render also returns a bridge function which we can use to coordinate some complex use cases. For example, here's how we'd test that a loader component can recover from an error:

// MoreLoader.tsx
interface LoaderProps<T> {
  getData: (lastId?: string) => Promise<T[]>;
  renderItem: (t: T, index: number) => React.ReactNode;
}

// Pretend this is a real component
export const MoreLoader = <T>(props: LoaderProps<T>) => {
  /* ... */
};

describe('MoreLoader', () => {
  it('can recover from errors', async () => {
    let nextIndex = 0;
    let error = false;
    const { page, bridge } = render(
      <MoreLoader<number>
        getData={async () => {
          if (error) throw new Error('Error');
          return nextIndex++;
        }}
        renderItem={(d) => <>Number is {d}</>}
      />
    );
    await expect(page.locator('text=Number is 0')).toBeVisible();
    await page.locator('.load-more').click();
    await expect(page.locator('text=Number is 1')).toBeVisible();
    await bridge(() => nextIndex = 10)
    await page.locator('.load-more').click();
    await expect(page.locator('text=Number is 10')).toBeVisible();
    await bridge(() => error = true);
    await page.locator('.load-more').click();
    await expect(page.locator('text=Error loading item')).toBeVisible();
    await bridge(() => error = false);
    await page.locator('.load-more').click();
    await expect(page.locator('text=Number is 11')).toBeVisible();
  });
});

Overrides

Sometimes the bridge function doesn't cover all your use cases. For example, if you want to test that a component can recover from an error, you'll need to be able to override some logic within the component to simulate an error. For this use case, Safetest provides the createOverride function. This function allows you to override any value within the component. For example let's pretend we have this existing component:

// Records.tsx
export const Records = () => {
  const { records, loading, error } = useGetRecordsQuery();
  if (loading) return <Loader />;
  if (error) return <Error error={error} />;
  return <RecordList records={records} />;
};

Here's how to create and use an override for the useGetRecordsQuery hook to simulate an error:

 // Records.tsx
+ const UseGetRecordsQuery = createOverride(useGetRecordsQuery)

 export const Records = () => {
+  const useGetRecordQuery = UseGetRecordsQuery.useValue()
   const { records, loading, error } = useGetRecordsQuery();
   if (loading) return <Loader />;
   if (error) return <Error error={error} />;
   return <RecordList records={records} />;
 };

The test would look like this:

describe('Records', () => {
  it('Has a loading state', async () => {
    const { page } = render(
      <UseGetRecordQuery.Override with={(old) => ({ ...old(), loading: true })}>
        <Records />
      </UseGetRecordQuery.Override>
    );
    await expect(await page.locator('text=Loading')).toBeVisible();
  });

  it('Has an error state', async () => {
    const { page } = render(
      <UseGetRecordQuery.Override
        with={(old) => ({ ...old(), error: new Error('Test Error') })}
      >
        <Records />
      </UseGetRecordQuery.Override>
    );
    await expect(await page.locator('text=Test Error')).toBeVisible();
  });

  it('Has a loaded state', async () => {
    const { page } = render(
      <UseGetRecordQuery.Override
        with={(old) => ({
          ...old(),
          loading: false,
          value: [{ name: 'Tester' }],
        })}
      >
        <Records />
      </UseGetRecordQuery.Override>
    );
    await expect(await page.locator('text=Tester')).toBeVisible();
  });
});

This isn't limited to overriding a hook or a service, we can override anything we want. For example we can override the Date.now() to get consistent time stamps in our tests. A powerful use case for this is when we have a component that combines 3 graphql calls, we can test what happens if only one of those call fails, etc.

Targeted Injecting in App

Since Safetest is bootstrapped within the application, we can test anything into the application. This ensures that even seemingly complex use cases can be tested. Here are some examples of this:

• Our app makes a bunch of GPRC calls and we want to test that if one of them fails the page doesn't crash. This isn't feasible using attempts to override the network calls, unless we can understand binary.

  // Page.tsx
  const myServiceClient = new MyServiceClient('http://localhost:8080');
  const useGrpc = (dataType: string) => {
    const [data, setData] = React.useState(null);
    const [error, setError] = React.useState(null);
    const [loading, setLoading] = React.useState(false);
  
    React.useMemo(() => {
      setLoading(true);
      const request = new DataRequest();
      request.setType(dataType);
      myServiceClient.getData(request, {}, (err, response) => {
        setLoading(false);
        if (err) {
          setData(null);
          setError(err);
        } else {
          setData(response.toObject());
          setError(null);
        }
      });
    }, [dataType]);
  
    return { loading, error, data };
  };
  
  export const Page = () => {
    const settingsPanel = useGrpc('settings');
    const alertsPanel = useGrpc('alerts');
    const todosPanel = useGrpc('todos');
    return (
      <Grid>
        <Settings settings={settingsPanel} />
        <Alerts alerts={alertsPanel} />
        <Todos todos={todosPanel} />
      </Grid>
    );
  };

   // Updated Page.tsx
   // ... snip
  +export const UseGrpc = createOverride(useGrpc);
  
   export const Page = () => {
  +  const useGrpc = UseGrpc.useValue();
     const settingsPanel = useGrpc('settings');
     const alertsPanel = useGrpc('alerts');
     const todosPanel = useGrpc('todos');
     return (
       <Grid>
         <Settings settings={settingsPanel} />
         <Alerts alerts={alertsPanel} />
         <Todos todos={todosPanel} />
       </Grid>
     );
   };

  // Page.safetest.tsx
  describe('Page', () => {
    it('can handle an error on the Alerts pane', async () => {
      const UseGrpcOverride = UseGrpc.Override;
      const { page } = render((app) => (
        <UseGrpcOverride
          with={(old) => {
            return (dataType) => {
              const oldValue = old(dataType);
              if (dataType === 'alerts') {
                return {
                  ...oldValue,
                  error: new Error('Test Error'),
                };
              }
              return oldValue;
            };
          }}
        >
          {app}
        </UseGrpcOverride>
      ));
      await page.locator('.view-panels').click();
      await expect(page.locator('text=Error loading alerts')).toBeVisible();
      await expect(
        page.locator('text=Error loading settings')
      ).not.toBeVisible();
      await expect(page.locator('text=Error loading todos')).not.toBeVisible();
    });
  });

• Part of our build process is to regenerate the graphql schema and now we want to test one of those calls failing, overriding the network calls means that anytime the schema changes we'll need to update our tests.

• There was a regression with debouncing/throttling and not using cached results in an autocomplete and we want to make sure it doesn't break in the future.

• We want to test that an absolutely positioned element set to the users width preference doesn't cover over a clickable element.

With the tools above we can test pretty much any scenario we can think of. The guiding principle of Safetest is to make any test possible, no matter how complex or involved the test is.

Providers and Contexts

An issue that React Testing Library and component testing libraries need to contend with is rewrapping the component in required providers and contexts. This is due to react-query, react-redux, react-router, etc., all requiring a provider to be present in the tree. However, since Safetest is bootstrapped with the application, we can just shuffle around some code to make this works for all use cases. All that's required is to move the Providers/Contexts to a separate file (for example src/Providers.tsx) and use it when bootstrapping the application:

src/Provider.tsx

import React from 'react';
import { QueryClient, QueryClientProvider } from 'react-query';
import { ApolloClient, ApolloProvider } from '@apollo/client';

// Create provider clients.
const apolloClient = new ApolloClient({});
const queryClient = new QueryClient({});

export const Provider: React.FC = ({ children }) => (
  <QueryClientProvider client={new QueryClient()}>
    <ApolloProvider client={apolloClient}>{children}</ApolloProvider>
  </QueryClientProvider>
);

src/index.tsx

import { bootstrap } from 'safetest/react';
import { Provider } from './Provider';
bootstrap({
  element,
  render: (elem) => ReactDOM.render(<Provider>{elem}</Provider>, container),
  importGlob: import.meta.glob('./**/*.safetest.{j,t}s{,x}'),
});

Now you never need to think again about providers and contexts, just use render as you normally would.

Authentication

Most corporate applications require some form of authentication. Safetest provides a simple way to handle authentication in your tests. Ideally you already have a service to generate a cookie for you, but if you don't, you'll need to create one. Here's an example of a service that generates a cookie for you:

// auth.ts
import { Cookie, Page, chromium } from 'playwright';

let cookies: Cookie[];

const getCookies = async () => {
  const browser = await chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();
  await page.goto('https://my-app.com/login');
  await page.fill('input[name="username"]', process.env.USERNAME);
  await page.fill('input[name="password"]', process.env.PASSWORD);
  await page.click('button[type="submit"]');
  await page.waitForNavigation();
  cookies = await context.cookies();
  await browser.close();
};

const addCookies = async (page: Page) => {
  await page.context().addCookies(cookies);
};

To use it, you need to add the following code to your setup-safetest.tsx file:

// setup-safetest.tsx
import { setup } from 'safetest/setup';

import { getCookies, addCookies } from './auth';

beforeAll(getCookies);

setup({
  bootstrappedAt: require.resolve('./src/main.tsx'),
  ciOptions: { usingArtifactsDir: 'artifacts' },
  hooks: { beforeNavigate: [addCookies] },
});

Debugging and Troubleshooting

Safetest takes advantage of playwright and jest to provide a lot of debugging and troubleshooting tools. Here are some of the most useful ones. The script copied in the package.json file will open a debug port that you can connect to with the node inspector. You can just add a debugger statement in your test and the node-inspector will just catch it. Alternately to can add a launch.json file with these run properties and have vscode auto-attach to the process.

The render also returns a pause method that will pause the execution of the page and allow you to inspect the page in the browser and to continue to use the playwright page object.

it('can pause', async () => {
  const { page, pause } = await render();
  debugger;
  await pause();
  await expect(page.locator('text=Welcome to The App')).toBeVisible();
});

We can even run code live against this test!

Reporting

Safetest published an HTML Test Reporter that can be used to view the results of your tests. To use it, you just need to process the results json to include information about the artifact, you can use the cli command that safetest provides:

npx safetest add-artifact-info artifacts.json results.json

Now you can either publish the node_modules/safetest/report.html standalone html file or import the Report component from safetest/report and use it in your application.

import { Report } from 'safetest/report';

export const MyReport: React.FunctionComponent = () => {
  return (
    <Report
      getTestUrl={(filename, test) => {
        const relativeFile = `./${filename}`.replace(/\.[jt]sx?$/g, '');
        const testName = test.trim().replace(/ /g, '+');
        return `${process.env.DEPLOYED_URL}?test_path=${relativeFile}&test_name=${testName}`;
      }}
      renderArtifact={(type, path) => {
        if (type === 'video')
          return <video src={`${process.env.DEPLOYED_URL}${path}`} controls />;
        // etc
      }}
    />
  );
};

Here's an example of the vite-react example app report

Note that part of the report is a link to the test in the deployed environment. This is done by passing the getTestUrl prop to the Report component. This is useful for debugging tests that fail in CI but not locally. The renderArtifact prop is used to render the artifacts that are generated by the tests. This is useful for rendering videos, trace viewer, etc. As an example here's a link to a Vite/React test and the associated trace and video artifacts

src__nother_safetest_tsx_Main2_can_do_many_interactions_fast-attempt-0.webm

How Safetest works

Safetest is a combination of a few different technologies glued together intelligently to leverage the best parts of each. The essential technologies used are

• A test runner (this can be Jest or Vitest, feel free to open a PR for other test runners, it's pretty easy to add)
• A browser automation library (Playwright is the default and only one used currently, this will be a bit harder to extend)
• A UI framework (React is the main example used in this Readme, but there are adapters for vue, svelte, and angular, feel free to open a PR for other frameworks)

Take a look at the examples folder to see different combinations of these technologies. Please feel free to open a PR with more examples.

When the runner first starts it will build a mapping of the test structure. For example suppose we have a test file src/App.safetest.tsx with the following contents:

import { describe, it, expect } from 'safetest/jest';
import { render } from 'safetest/react';

import { Header } from './components/header';

describe('App', () => {
  it('renders the app', async () => {
    const { page } = await render();
    await expect(page.locator('text=Welcome to The App')).toBeVisible();
  });

  it('can render a regular header', async () => {
    const { page } = await render(<Header />);
    await expect(page.locator('text=Logout')).toBeVisible();
    await expect(page.locator('text=admin')).not.toBeVisible();
    expect(await page.screenshot()).toMatchImageSnapshot();
  });

  it('can render an admin header', async () => {
    const { page } = await render(<Header admin={true} />);
    await expect(page.locator('text=Logout')).toBeVisible();
    await expect(page.locator('text=admin')).toBeVisible();
    expect(await page.screenshot()).toMatchImageSnapshot();
  });
});

Safetest will build a tree of the tests and their structure:

{
  "App": {
    "renders the app": async () => { /* ... */ },
    "can render a regular header": async () => { /* ... */ },
    "can render an admin header": async () => { /* ... */ },
  }
}

The test runner continues running, i.e., the "renders the app" test runs and hits the render() function, resulting in Safetest opening a browser and navigating to the page. Safetest controls the browser instance, exposes a "magic" function, gets info about the currently executing test "App renders the app". There's also an exposed, magic function that will be called when the browser page is "ready".

On the browser side of things, when the call to bootstrap is called, the following happens:

• Safetest will check if there's a a "magic" function available that will give us information about the current executing test.

  • If there is no test info available Safetest will render the page as normal and the bootstrapping process is done.

• Safetest will now call the import function that was passed to bootstrap with the name of the test file.

• This will allow Safetest to build that same mapping in the browser.

• Safetest will now execute the mapping["app renders the app"] function.

• Safetest will hit the render function. Safetest will now render this component on the page.

• Safetest will now call the magic exposed function to signal that the page is ready for testing.

  Back in node...

• The await render(...) call now resolves and we can continue with the test.

SafeTest Development

To develop Safetest, you can use the examples folder to test your changes. The examples folder is a monorepo that contains a few different examples of how to use Safetest. To get started you'll need to follow these steps

git clone https://github.com/kolodny/safetest # or your fork
cd safetest
npm install --legacy-peer-deps # legacy-peer-deps is needed due to how the repo can't load all the peer deps

cd examples/vite-react-ts
npm install # We'll use this app as our main example

Now you'll need three different terminal sessions

# Session 1 is used to build the library (this is run in the root of the project)
npm run build -- --watch

# Session 2 is used to run the example app (this is run in the examples/vite-react-ts folder)
npm run dev -- --force # Force vite to not use the cache

# Session 3 is used to run the tests (this is run in the examples/vite-react-ts folder)
npx cross-env OPT_HEADED=1 npm run safetest

You should now see the tests running in the browser and the results in the terminal.