Repeato CLI Test Runner

Introduction

Repeato CLI is a headless test runner that allows you to execute test batches designed with Repeato Studio.

We try to rely as much a possible on open standards, so all the workspace data is stored in either image formats or text based formats (mostly JSON). That makes it possible to generate tests not just with Repeato Studio, but also with tools of your choice. That's also true for the test run artefacts: All data is stored as image data (screenshots), XML (JUnit export, can be used for Jenkins) and JSON.

Repeato CLI is built using Node.js which makes it possible to run it on almost any system. However, it has dependencies to native modules, which makes it necessary to build parts of it for the target OS and platform.

Installation:

Node.js version 16 is strongly recommended. You can use nvm to switch to the right Node.js version: nvm use 16. You can also choose to use another Node.js version, but this requires compiling our native modules on your end. We don't recommend doing this, because it can require some effort to get it working.

Install and run instantly via NPX:

The recommended way of using this package is via npx, which downloads and runs Repeato-CLI right away:

npx @repeato/cli-testrunner

If you run Repeato as part of your CI/CD, you might want to fix the version of the test runner, so you don't run into a situation where your setup breaks because of updates we ship. (Although those should usually not cause any problems!)

npx @repeato/[email protected]

Installation via NPM:

npm -g @repeato/cli-testrunner

...and then run it like this:

cli-testrunner --help

Command Line Arguments:

The testrunner essentially needs two inputs:

1. The path to your workspace containing you batch
2. The batch ID of the batch you want to execute

cli-testrunner --help will print all the required and optional params:

Runs a selected test batch of your Repeato workspace

Usage via npx:
  $ npx @repeato/cli-testrunner [params]

Usage when installed globally via npm:
  $ cli-testrunner [params]

Required params:
  --accessToken: Your Repeato access token. Can be generated here: https://clients.repeato.app/
  --workspaceDir: Path to the workspace directory. See also https://www.repeato.app/documentation/test-workspaces/
  --batchId: Which batch should be executed. 
    Just execute cli-testrunner --workspaceDir [workspace-dir] to see a list of contained batches

Optional params:
  --deviceId: The device ID of the device you want to connect to
    List Android devices via 'adb devices' or iOS simulators via 'xcrun simctl list'
  --runMode: "AllTests" | "OnlyFailed" (default: "AllTests")
  --timeoutFactor: Since some CI servers sometimes don't provide as much performance as local machines, 
    it can be useful to increase the timeout when running on a server by a factor. E.g pass 1.5 to increase the timeouts by 50% (default: 1.0)
  --waitDurationBetweenSteps: Additional wait time between step executions can help during debugging or to increase test stability. 
    (default: 0, milliseconds)
  --enableUtf8Keyboard: "true" | "false". Enable the Repeato test automation keyboard. 
    Set this to "false" if you want to use the default keyboard of your device. (default: "true")
  --outputDir: Override the default export path for batch run reports (default is: [workspaceDir]/batchRuns)
  --logLevel: "WARN" | "INFO" | "DEBUG" (default: "INFO")
  --force: Ignore errors (DB version mismatch) and execute anyway

Examples:
  $ npx @repeato/cli-testrunner --workspaceDir C:/path/to/your/workspace
  List all batches in your workspace
  
  $ npx @repeato/cli-testrunner --workspaceDir C:/path/to/your/workspace --batchId 278962-W73KOl2-SIUSZ
  Executes all the tests in the given batch

Outputs:

1. A JUnit xml file is created in the root folder of Repeato CLI
2. For each test a testRun object is created in [workspaceDir]/[testId]/test.json
3. For each test step a screenshot is created in [workspaceDir]/[testId]/testRuns/[deviceId]/
4. An html batch report is created in the [workspaceDir]/batchRuns directory. The exported directory can be shared via a webserver to make the report accessible to your QA team.
5. A batchRun.json file containing all the batch run information in [workspaceDir]/batchReports/[your batch report] directory
6. Returns with exit code 0 if batch runner managed to run all tests, even if a test failed. Returns different exit code if some error occured during execution.

Requirements:

Supported Node.js version: 16 (v16.13.0 is tested). You can use nvm to switch to the right Node.js version: nvm use 16.

Supported operating systems: MacOS, Windows, Linux

For Apple Sillicon (M1 / ARM architecture), you need to use Node.js v16

Starting the testrunner

Simply execute npx @repeato/cli-testrunner [args]

Installing emulators or simulators

We have a couple of guides on how to install Android emulators or iOS simulators on your target system here: https://www.repeato.app/documentation/virtual-test-devices/

Install apps programmatically on your target emulator / simulator

On Android you can use ADB to push your APK to the virtual device.

1. Install ADB
2. adb push /path/to/myApp.apk

On iOS you can use IDB

1. Install IDB
2. idb install /path/to/myApp.app

Cloud services for running your tests

We tested our CLI test runner with following services:

• Github Actions
• Bitrise
• Codemagic

You can find some example workflows here: https://www.repeato.app/documentation/continuous-integration/

Troubleshooting

Tests run fine in Repeato Studio, but fail in Repeato CLI

In most cases Repeato should deliver the same results as Repeato Studio. We are actually using the same code base for both.

However, it really also depends on how the tests and the app is designed.

However, there are limits: Here is an example where Studio and CLI could deliver different results: Let's say your app is designed in a way, that a button appears on stage, but it's not clickable for 100ms (because it was coded this way. Maybe there is a server request involved, ...). Let's further assume Repeato-Studio (running on an older workstation) executes slower than the CLI (which is running on a fast server).

Repeato Studio (being slower) would click the button quite late (150ms delay), the test would run through. Repeato CLI (being fast) would click the button quite early (50ms delay), but the button would not be ready and therefore not trigger an action.

Here are some options of what you can try to solve the issue:

• Make sure you are using the same device on Repeato Studio and Repeato CLI
• Slow down the CLI testrunner: There is a --waitDurationBetweenSteps flag that could be used to make it wait between steps.
• Figure out the problematic steps and add a wait step before them
• Send us your batch report ([email protected]) and we can have a look at it