Testlio platform CLI

Command-line interface to interact with Testlio platform, allowing to create and schedule automated tests, and retrieve its results.

Creating automated test run on Testlio platform with Testlio internal provider

If you want to schedule automated tests on Testlio platform with Testlio internal provider, you can use the following commands:

NOTE: Documentation to be added soon.

Creating an Automated Test Run on Testlio Platform with an External Provider

To schedule automated tests on the Testlio platform using an external provider, follow these steps:

Step 0: Install the CLI & Configure the CLI

• Install the CLI by running npm install -g @testlio/cli
• See Configuration section below for more details on how to configure the CLI.

Step 1: Set up Automated Browser/Device

• Either create a new automatedBrowser/automatedDevice or locate an existing one that you intend to use.

Step 2: Create Test Run (External)

• Create a new automated test run. Ensure you include the --externalResults true flag while creating the run.

Step 3: Submit Test Results

• Submit the automated test results. Currently, we only support test results formatted using the Allure framework.

Step 4: Submit Additional Assets (Optional)

• You may also submit any additional assets, such as videos, screenshots, or logs to support your test results.

Step 5: Finalize Results

• Once you've completed the above steps, finalize the test results to conclude the test run.

Step 6: Read Results

• After test is finalized on step 5 and results are processed, you can read the results from the Testlio platform.

NOTE: The results won't be finalized until an async job in result-service processes them. For local environment purposes, you may need to trigger that process manually before reading the results.

Utility Commands

• find-device-result: Find a device result based on various parameters.

NOTE: For any of the commands you can use --h or --help param to get more details about the command

Step 0: Configuration

All the below configuration steps are required.

1. Make sure to set the RUN_API_TOKEN environment variable. (You can get this token from the Testlio platform)

Example: export RUN_API_TOKEN=your_token

2. Add project file project-config.json to the root of your project folder. Project config file contains the static configuration related to the project which will not change for a workspace:

    {
        "baseURI": "https://api.testlio.com",
        "platformURI": "https://app.testlio.com/tmt/project/",
        "projectId": "project id, this can be found while generating token token on Testlio Platform. Please check the screenshot below",
        "automatedRunCollectionGuid": "this can be found while generating token on Testlio Platform. Please check the screenshot below",
        "testRunCollectionGuid": "this can be found while generating token on Testlio Platform. Please check the screenshot below",
        "resultCollectionGuid": "this can be found while generating token on Testlio Platform. Please check the screenshot below",
        "workspaceName": "this can be found while generating token on Testlio Platform. Please check the screenshot below"
    }

3. Add testConfig file test-config.json to the root of your project folder. Test config is of two variants, config section is optional.

WITHOUT ADDITIONAL CONFIGS

{
  "automatedTestNamePrefix": "Prefix you want to append with the automated test (e.g Circle CI:)"
}

WITH ADDITIONAL CONFIGS

{
  "automatedTestNamePrefix": "Prefix you want to append with the automated test (e.g Circle CI:)",
  "config": {
    "deviceTestType": "OPTIONAL IF YOU WANT TO SPECIFY TEST TYPE (e.g APPIUM_JAVA_JUNIT, APPIUM_NODE, APPIUM_RUBY, APPIUM_PYTHON, SELENIUM_JAVA_TESTNG, SELENIUM_JAVA_JUNIT, SELENIUM_NODE, SELENIUM_PYTHON)"
  }
}

Step 1: Set up Automated Browser/Device

Either create a new automatedBrowser/automatedDevice or locate an existing one that you intend to use. This section has two subsections below

Step 1.1: Create a new automatedBrowser or search for an existing one

The create-automated-browser command provides a versatile way to manage automated browsers. Whether you're looking to create a new automated browser or search for existing ones based on various parameters, this command has you covered.

Usage

testlio create-automated-browser [Options]

Use --help to see the list of available options.

--checkExisting:
   When set to true, this flag will initiate a search for existing browsers based on the parameters provided.
   If browsers are found, a list will be displayed. Otherwise, a message indicating no browsers were found will be shown.
   
   When --checkExisting is true, then following parameters become optional, and can be use to narrow down the search:
             --externalId: External identifier for the browser (e.g., windows-chrome-latest)
             --browserName: Name of the browser (e.g., Google Chrome, Microsoft Edge (Chromium), Mozilla Firefox)
             --platformName: Name of the platform (must be from these values ['Microsoft Windows', 'Macintosh', 'Linux', 'Android', 'iOS'])            
             --provider: Name of the provider (must be from these values ['aws', 'browserstack', 'local', 'sauce-labs', 'azure-playwright', 'bitbar'])
             
   When --checkExisting is false or not set, then following parameters become required, this is used to create a custom browser:
            --externalId: To create a new automated browser, provide an external identifier (e.g., windows-MicrosoftEdge-latest)
            --browserName: Name of the browser you wish to create (e.g., Turbo Browser, Ruko Browser)
            --platformName: Platform where the browser will run (must be from these values ['Microsoft Windows', 'Macintosh', 'Linux', 'Android', 'iOS'])
            --provider: Name of the provider (must be from these values ['browserstack', 'local', 'sauce-labs', 'azure-playwright', 'bitbar'])

  --projectConfig [path]: (Optional) Path to project config (default: project-config.json)

NOTE: When creating a new automated-browser you can't select provider as 'aws'.

Example Usage

An example of a command to create a new automated browser:
testlio create-automated-browser --externalId windows-MicrosoftEdge-latest --browserName "Microsoft Edge (Chromium)" --platformName "Microsoft Windows" --provider local

An example of a command to search for existing browsers:
testlio create-automated-browser --checkExisting true --provider local

Step 1.2: Create a new automatedDevice or search for an existing one

The create-automated-device command provides a versatile way to manage automated devices. Whether you're looking to create a new automated device or search for existing ones based on various parameters, this command has you covered.

Usage

Usage: testlio create-automated-device [Options]

Use --help to see the list of available options.

Options:
    --checkExisting   (optional) If set, will return list of devices found. You can additionally narrow down the search by passing [externalId, model, manufactureName, typeName, operatingSystemName].

    When --checkExisting is set to true: (You can search existing devices on basis of any of the following parameters, if no parameters are passed, it will return all the devices)
        --externalId      (optional) (e.g unique-provider-identifier)
        --model           (optional) (e.g Galaxy S21, Samsung Galaxy S23 Ultra)
        --manufactureName (optional) (e.g Nokia, HTC, ZTE, Samsung, Sony, Apple)
        --typeName        (optional) (e.g 'Mobile', 'Tablet', 'Connected devices', 'Gaming', 'Wearable', 'Desktop (PC)')
        --operatingSystemName (optional) (e.g 'Symbian', 'Android', 'Windows Phone', 'BlackBerry', 'Windows', 'Roku OS', 'Chrome OS', 'webOS', 'Brew Mobile Platform', 'Xbox system software', 'Android TV', 'Fire OS', 'Palm OS', 'iOS', 'Tizen', 'Firefox', 'watchOS', 'PlayStation system software', 'Mac OS', 'tvOS', 'Android Wear', 'Linux', 'Danger', 'Windows CE', 'Orsay (Legacy)', 'Nintendo Switch OS', 'SmartCast TV', 'Amazon Echo OS', 'Garmin OS', 'Xfinity Flex OS', 'Polar OS', 'Suunto OS', 'Google Stadia', 'Steam OS', 'Jabra Firmware', 'Plantronics Firmware', 'Sennheiser Firmware', 'Cast Firmware', 'NetCast', 'Vidaa')
        --provider       (optional) (e.g ['aws', 'browserstack', 'local', 'sauce-labs', 'azure-playwright', 'bitbar'])
        
    When --checkExisting is not set or set to false:
        --externalId      (required) To create a new automated-device please provide externalId (this-can-be-unique-provider-identifier, if you do not know about it please generate a random string device-name-someGuid)
        --provider        (required) To create a new automated-device please provide provider (e.g ['browserstack', 'local', 'sauce-labs', 'azure-playwright', 'bitbar'])
        --model           (required) To create a new automated-device please provide model, model is full name of the device (e.g Galaxy S21)
        --manufactureName (required) To create a new automated-device please provide manufactureName (e.g Nokia, Samsung, Apple)
        --typeName        (required) To create a new automated-device please provide typeName (e.g ['Mobile', 'Tablet', 'Connected devices', 'Gaming', 'Wearable', 'Desktop (PC)'])
        --operatingSystemName (required) To create a new automated-device please provide operatingSystemName (e.g 'Symbian', 'Android', 'Windows Phone', 'BlackBerry', 'Windows', 'Roku OS', 'Chrome OS', 'webOS', 'Brew Mobile Platform', 'Xbox system software', 'Android TV', 'Fire OS', 'Palm OS', 'iOS', 'Tizen', 'Firefox', 'watchOS', 'PlayStation system software', 'Mac OS', 'tvOS', 'Android Wear', 'Linux', 'Danger', 'Windows CE', 'Orsay (Legacy)', 'Nintendo Switch OS', 'SmartCast TV', 'Amazon Echo OS', 'Garmin OS', 'Xfinity Flex OS', 'Polar OS', 'Suunto OS', 'Google Stadia', 'Steam OS', 'Jabra Firmware', 'Plantronics Firmware', 'Sennheiser Firmware', 'Cast Firmware', 'NetCast', 'Vidaa')
        --operatingSystemVersion   (required) To create a new automated-device please provide operatingSystemVersion (e.g 11)
    --projectConfig [path]     (optional) path to project config (default: project-config.json)   

NOTE: When creating a new automated-device you can't select provider as 'aws'.

Example Usage

An example of a command to search for existing devices:
testlio create-automated-device --checkExisting true --provider local

An example of a command to create a new automated device:
testlio create-automated-device --externalId device-name-someGuid --provider local --model "Galaxy S21" --manufactureName Samsung --typeName Mobile --operatingSystemName Android --operatingSystemVersion 14

Step 2: Create Test Run (External)

Create a new automated test run. Ensure you include the --externalResults true flag while creating the run.

• Automated Browser/Device: Ensure that you've already created or located the automatedBrowser or automatedDevice you intend to use for this run. (See above sections for more details on how to create or search for automatedBrowser/automatedDevice)
• Flag for External Results: Use the --externalResults true flag to indicate that the execution of scripts is taking place on the Testlio platform.
  • Specify Automated Browser or Automated Device: Add either the --automatedBrowserIds or --automatedDeviceIds flag to indicate which automated browser or device to attach to the run.
    • You can pass a single ID or a comma-separated list of IDs.

      --automatedBrowserIds 71fd1448-4bac-424d-b5e8-504df996001d,71fd1448-4bac-424d-b5e8-504df996001d

• Result Provider: Use the --resultProvider flag to specify where the test results are coming from. The possible values are [aws, browserstack, local, sauce-labs, azure-playwright, bitbar].
  • aws: Allows usage of devices/browsers registered on AWS Device Farm and currently also registered in the Testlio Platform.
    • NOTE: This doesn't mean that your run will be scheduled on AWS, or we are reserving above AWS devices
  • browserstack: Allows usage of devices/browsers registered on Browserstack and currently also registered in the Testlio Platform.
    • NOTE: This doesn't mean that your run will be scheduled on Browserstack, or we are reserving above Browserstack devices
  • local: Allows usage of devices/browsers registered on Testlio Platform only. If your custom/browser is not registered on Testlio platform, use the above section to create a new automatedBrowser/automatedDevice.
    • NOTE: If you are using --resultProvider local, please make sure that you have created automatedBrowser/automatedDevice on Testlio Platform
  • sauce-labs: Allows usage of devices/browsers registered on Testlio Platform only. If your custom/browser is not registered on Testlio platform, use the above section to create a new automatedBrowser/automatedDevice.
    • NOTE: If you are using --resultProvider sauce-labs, please make sure that you have created automatedBrowser/automatedDevice on Testlio Platform
  • azure-playwright: Allows usage of devices/browsers registered on Testlio Platform only. If your custom/browser is not registered on Testlio platform, use the above section to create a new automatedBrowser/automatedDevice.
    • NOTE: If you are using --resultProvider azure-playwright, please make sure that you have created automatedBrowser/automatedDevice on Testlio Platform
  • bitbar: Allows usage of devices/browsers registered on Testlio Platform only. If your custom/browser is not registered on Testlio platform, use the above section to create a new automatedBrowser/automatedDevice.
    • NOTE: If you are using --resultProvider bitbar, please make sure that you have created automatedBrowser/automatedDevice on Testlio Platform
• Test Config and Project Config: Use the --testConfig and --projectConfig flags to specify the path to the test config and project config files, respectively. If these flags are not provided, the CLI will look for these files in the current directory.
  • NOTE: See last section for more details on how to create these files

Example usage

NOTE: These automatedDeviceIds/automatedBrowserIds values are just an example

# Too add multiple automatedDeviceIds/automatedBrowserIds, please use comma separated values, make sure the values are not space separated, and values are unique
testlio create-run --externalResults true --resultProvider local --automatedDeviceIds <REPLACE_ID_FROM_STEP_0_DEVICE_1_ID>,<REPLACE_ID_FROM_STEP_0_DEVICE_2_ID>
testlio create-run --externalResults true --resultProvider local --automatedBrowserIds <REPLACE_ID_FROM_STEP_0>

NOTE: After running the above command it will generate .env file which will have RESULT_ID value. This file is auto generated at the root of your directory

Step 3: Submit Test Results

This command allows you to submit test results to the Testlio platform. We currently only support Allure framework. Please zip your Allure results, the folder should look like this:

main-zip-file.zip // file name can be anything
    -- allure-results // folder name should be allure-results
        -- content from allure-results folder. // this is the actual allure results generated by allure framework

NOTE: MAX FILE LIMIT FOR ZIP FILE IS 500 MB

Usage:

  testlio parse-run-results [options]
  
  Use --help to see the list of available options.

Options:

  --deviceResultId           ID of the specific device result you want to submit results, if you do not know the deviceResult Id you can use find-device-result command or else you can pass automatedDeviceId/automatedBrowserId.
                             (Mutually exclusive with --automatedDeviceId and --automatedBrowserId; provide only one of these options.)
                             
  --automatedDeviceId        ID of the automated device for which you want to submit results.
                             (Mutually exclusive with --deviceResultId and --automatedBrowserId; provide only one of these options.)
                             
  --automatedBrowserId       ID of the automated browser for which you want to submit results.
                             (Mutually exclusive with --deviceResultId and --automatedDeviceId; provide only one of these options.)
                             
  --browserVersion           REQUIRED. In case your device is browser, please pass on the exact browser version on which the tests were executed (e.g., 116.0.5845.110)                           
                             
  --externalSource           Optional. Specifies the external source URL or identifier where the test execution took place. 
                              This is useful for linking back to the original test execution platform (e.g., AWS Device Farm, BrowserStack, etc).

Examples:
- AWS Device Farm: "https://us-west-2.console.aws.amazon.com/devicefarm/home?region=us-east-1#/mobile/projects/<REPLACE_PROJECT_ID>/runs/<REPLACE_RUN_ID>"
- BrowserStack: "https://automate.browserstack.com/builds/abcd1234/sessions/xyz789"
- Azure DevOps: "https://dev.azure.com/your-org/your-project/_build/results?buildId=123"
  
  --path                     REQUIRED. Specifies the local path to the zip package containing Allure results.
  
  --projectConfig [path]     Specifies the path to the project configuration file.
                             (Optional; default is project-config.json)

Note:

  You must provide exactly one of the following: --deviceResultId, --automatedDeviceId, or --automatedBrowserId.
  The --path option is mandatory and should point to a valid zip package containing Allure results.
  The --browserVersion is mandatory in case your device is browser. Please pass on exact browser version on which the test were executed. 

Example usage

testlio parse-run-results --path ./allure-results-demo-package.zip --automatedDeviceId <REPLACE_ID_FROM_STEP_0>

testlio parse-run-results \
  --path ./allure-results-demo-package.zip \
  --automatedBrowserId <REPLACE_ID_FROM_STEP_0> \
  --browserVersion 116.0.5845.110 \
  --externalSource "https://us-west-2.console.aws.amazon.com/devicefarm/home?region=us-east-1#/mobile/projects/<REPLACE_PROJECT_ID>/runs/<REPLACE_RUN_ID>"

Step 4: Submit Additional Assets (Optional)

You may also submit any additional assets, such as videos, screenshots, or logs to support your test results. These assets will be displayed at device level, and they will not be linked to any test, or step.

Usage

testlio upload-asset [options]

Use --help to see the list of available options.

Options:

  --deviceResultId          (Mutually exclusive) Device result id to which you want to upload assets.
  --automatedBrowserId      (Mutually exclusive) Automated Browser id to which you want to upload assets.
  --automatedDeviceId       (Mutually exclusive) Automated Device id to which you want to upload assets.
  --path                    (Required) Local path to the asset that you want to upload.
  --type                    (Required) Type of the asset. Possible values: "video", "screenshot", "log", "other".
  --source                  (Required) Source of the asset. Possible values: "allure", "unknown".
  --projectConfig [path]     (Optional) Path to project config. Defaults to 'project-config.json'.

Note: One of the following options must be specified: --deviceResultId, --automatedBrowserId, --automatedDeviceId.

NOTE: Max file limit for each asset is 50 MB ONLY

Example usage

testlio upload-asset --automatedDeviceId <REPLACE_ID_FROM_STEP_0> --path ./screenshot.png --type screenshot --source unknown

Step 5: Finalize Results

Once you've completed the above steps, finalize the test results to conclude the test run.

Usage

testlio finalize-results [options]

  --projectConfig [path]     (Optional) path to project config (default: project-config.json)

Example usage

testlio finalize-results

Step 6: Get Results

Once you've completed the above steps, you can get the test results printed in the console.

Usage

testlio get-results [options]

  --projectConfig [path]     (Optional) path to project config (default: project-config.json)

Utility Commands

These are some additional commands that you may find useful.

find-device-result

This command allows you to search for a device result based on various parameters.

Usage

testlio find-device-result [OPTIONS]

Options:

  --automatedBrowserId      AutomatedBrowserId, pass on to find corresponding device-result-id.
  --automatedDeviceId       AutomatedDeviceId, pass on to find corresponding device-result-id.
  --projectConfig [path]     (Optional) Path to project config. Defaults to 'project-config.json'.

Note: One of the following options must be specified:  --automatedBrowserId, --automatedDeviceId.