Superquick installation via npm install -g bash-universal-tester

:white_check_mark: bash-universal-tester             

Universal testing script for bash

General purpose awesome testing-script

What?

Currently this script supports:

• Testing native executables with text input/out files (.in/.out)

• Testing native executables with .in/.out files packed to a zip file

• Using custom testing scripts

• Some nice types of output formatting

Requirements

This is stand-alone script.

• Works on Linux with bash shell

• Works on Windows (BUT YOU NEED BASH EMULATION LIKE CYGWIN)

Installation

• Install via npm:

npm install -g bash-universal-tester

• Download through provided link     

• Alternatively on bash type:

wget https://raw.githubusercontent.com/styczynski/bash-universal-tester/master/utest.sh && chmod u+x ./utest.sh && mv ./utest.sh utest && PATH=$PATH:$PWD

Basic usage

Basic usage: utest <prog>

The script tries to autodetect folder with input/output test files. And sometimes gives suggestions what program you may want to test.

Basic in + out + non empty err

To test input folder (only .in and .out):

utest <prog> <folder>

Basic in + out + ignore err (err is not checked anytime)

To test input folder (only .in and .out):

utest --tierr <prog> <folder>

Basic in + out + err (missing .err file cause error)

To test input folder (.in, .out, and .err files):

utest --tnerr <prog> <folder>

Basic in + out + err (missing .err files are ignored)

To test input folder (.in, .out, and .err files):

utest --tgerr <folder> <prog> <folder>

Globbing input files

Let's suppose we have the following file structure:

tests
|- test1 
|  |- input.in
|  \- output.out
|
|- test2
   |- input.in
   \- output.out

We want to feed utest with input files nested in subdirectories. For that purpose just use:

utest <prog> "./tests/**/*.in"

Note that globbing must be provided in quotes otherwise it will be parsed by shell and won't work!

Custom input layout

Let's suppose we have the following file structure (even more unfriendly!):

tests
|- test1 
|  |- input.txt
|  \- out
|
|- test2
   |- input.txt
   \- out

We want to feed utest with input files nested in subdirectories. And the input files have custom extensions. We must tell utest where to find output files. We use --tgout flag that utilizes dynamic variable to generate output path. You can read more about dynamic variables in variables section.

utest <prog> --tgout "%input_file_folder/out" "./tests/**/input.txt"

Note that globbing must be provided in quotes otherwise it will be parsed by shell and won't work!

Advanced usage

utest [test_flags] <prog> <dir> [prog_flags]

• <prog> is path to the executable, you want to test

• <dir> is the path to folder containing .in/.out files

• [prog_flags] are optional conmmand line argument passed to program <prog>

• [test_flags] are optional flags for test script

| Switch | Parameters | Description | |-----------------------------------|------------|--------------| | --tdebug | | Disables removing any files (so the utest probably won't do any harm) and enables --tlog flag. | | --tlog | | Enable logging into utest.log file in the current directory. | | --ttools | [tools] | Sets additional debug tools.[tools] is the coma-separated array of tools names.Tools names can be as the following:size - prints the size of input file in bytes.time - prints time statistic using Unix time command.stime - measures time using bash date command (not as precise as time tool).vmemcheck - uses valgrind memcheck tools to search for application leaks.vmassif - uses valgrind massif and prints peak memory usage. | | --tscript | [script] | Sets output testing command as [script]Script path is path to the executed script/program.There exists few built-in testing scripts:Use --tscript ignore to always assume output is OK. | | --tscript-err | [script] | Sets stderr output testing command as [script]Script path is path to the executed script/program.There exists few built-in testing scripts:Use --tscript-err ignore to always assume stderr is OK. | | --tflags | | Enables --t(...) flags interpreting at any place among command line arguments(by default flags after dir are expected to be program flags) | | --tsty-format | | Make tester use !error!, !info! etc. output format | | --tterm-format | | Make tester use (default) term color formatting | | --tc--tnone-format | | Make tester use clean (only-text) formatting | | --tpipe-in |[command] | Use preprocessing of input. See Piping section | | --tpipe-out |[command] | Use postprocessing of output. See Piping section | | --tpipe-out-err |[command] | Use postprocessing of output error stream. See Piping section | | --ts | | Skips always oks | | --tierr | | Always ignore stderr output | | --tgout | [dir] | Sets (good) .out input directory(default is the same as dir/inputs will be still found in dir location/use when .out and .in are in separate locations) | | --tgerr | [dir] | Same as --tgout but says where to find good .err files(by default nonempty .err file means error) | | --terr | [dir] | Sets .err output directory (default is /out) | | --tout | [dir] | Set output .out file directory (default is /out) | | --tf | | Proceeds even if directories do not exists etc. | | --tneed-err--tnerr | | Always need .err files (by default missing good .err files are ignored)If --tnerr flag is used and --tgerr not specified the good .err files are beeing searched in [dir] folder. | | --te--tdefault-no-err | | If the .err file not exists (ignored by default) require stderr to be empty | | --tt | | Automatically create missing .out files using program output | | --tn | | Skips after-testing summary | | --ta | | Aborts after +5 errors | | -help--help | | Displays help info | | --tm | | Use minimalistic mode (less output) | | --tmm | | Use very minimalistic mode (even less output) | | --tmmm | | Use the most minimialistic mode (only file names are shown) |

Wherever -help, --help flags are placed the script always displays its help info.

About minimalistic modes:

• In --tm mode OKs are not printed / erors are atill full with diff

• In --tmm mode errors are only generally descripted / OK at output on success

• In --tmmm only names of error files are printed / Nothing at output on success

Variables

In <prog>, --tgerr <dir>, --tgout <dir> and config files you can use special dynamic variables. These are the following:

| name | description | |--------------------------|------------------------------------------------------------------| | %input_file | Current input file name along with extension | | %input_file_name | Current input file without .in or .out extension | | %input_file_folder | Directory of current input file | | %input_file_path | Full input path | | %input_file_list | List of all input files (separated by space) that will be loaded | | %file_count | Number of all input files that will be loaded | | %file_index | Number of the current input file starting from 1 | | %ok_index | Current number of test that succeeded | | %warn_index | Current number of test that generated warnnings | | %not_exists_index | Current number of test that had problems with non existing files | | %param_prog | Currently tested command | | %input_prog_flag_acc | Currently tested command's arguments |

Example usage:

utest "echo %input_file" <folder>

Moreover you can use formatting variables (that are set via formatting switches). Please use them instead of hard-coded values, because it's easy and improves customizability of your output.

| formatting variable name | description | |--------------------------|------------------------------------------------| | %bdebug | Begins DEBUG text section | | %edebug | Ends DEBUG text section | | %berr | Begins ERROR text section | | %eerr | Ends ERROR text section | | %binfo | Begins INFORMATION text section | | %einfo | Ends INFORMATION text section | | %bwarn | Begins WARNNING text section | | %ewarn | Ends WARNNING text section | | %bbold | Begins NOTICE text section | | %ebold | Ends NOTICE text section | | %bok | Begins OK STATUS text section | | %eok | Ends OK STATUS text section |

Example usage:

input: ./test
executions:
    - prog
hooks:
    test_case_start:
        - @echo %{bwarn}Hello%{ewarn} %input_file %{bok} %ok_index %{eok}
prog:
    command: echo Something
    

Piping

Utest provides easy way to preprocess your input file or postprocess program outputs.

All you have to do is to use --tpipe-in <command>, --tpipe-out <command> or --tpipe-out-err <command>.

Pipes are provided with additional variables:

| name | description | |------------------------|----------------------------------------------| | %input | Pipe program input file path | | %output | Pipe program output file path |

For example let's sort program output alphabetically:

utest.sh --tpipe-out "cat %input | sort > %output" <prog> <folder>

Advantage of pipes are that you do not modify in/out files directly. And you can test programs that may potentailly give not exactly the same answers but which are still correct.

Configuration file

Global configuration

Instead of passing all parameters by command line we offer the ability to put everything into single YAML file!

Utest seek for utest.yaml file in current directory. It can contain all configuration available via command line switches and flags!

All config options are listed there:

input: test/*.in
silent: false
good_output: test/%input_file_name.out
good_err: test/%input_file_name.err
need_error_files: false
testing_script_out: ignore
testing_script_err: ignore
executions:
    - prog1
    - prog2
prog1:
    cwd: ./somefolder
    command: ./test/totest.sh
    args: %input_file_name
    pipes_out:
        - echo 123 > %output
    pipes_in:
        - echo 123 > %output
    pipes_out_err:
        - echo 123 > %output
        - cat %input | echo 123 > %output
prog2:
    command: echo 99

Single test configuration

You can also configure environment for single test case! Just put xyz.config.yaml file next to your input file xyz.in.

All options of config file for single test are listed below:

prog2:
    cwd: ./somefolder
    args: %input_file_name some additional args
    input: override_input_file.in

You must identify program by the command it calls.

Hooks

You can provide hooks commands for any of testing lifecycle state.

All available hooks are:

• init - called when testing begins
• deinit - called when testing ends
• test_case_start - called when new test file is tested
• test_case_finish - called when the test file was tested
• test_case_fail - called when test fails
• test_case_fail_out - called when test fails on std output (launched after test_case_fail)
• test_case_fail_err - called when test fails on error output (launched after test_case_fail)
• test_case_fail_success - called when test succeeded

Please note that: You can add mutliple commands that are executed in order from up to the bottom. If the command begins with @ character then it's output is directly displayed. If not then utest can change it to be more readable to the user!

input: test/*.in
good_output: test/%input_file_name.out
need_error_files: false
executions:
    - prog1
    - prog2
hooks:
    init:
        - @echo Testing
        - @echo Prepare input...
    deinit:
        - @echo Goodbye
    test_case_fail:
        - @echo [%{input_file}]  Test case failed for %{param_prog}
    test_case_success:
        - @echo [%{input_file}]  Test case successed for %{param_prog}
    test_case_fail_out:
        - @echo [%{input_file}]  FAILED ON OUTPUT
    test_case_fail_err:
        - @echo [%{input_file}]  FAILED ON ERR
        - @echo Whats a shame
    test_case_start:
        - @echo New test case jsut started %{input_file}
    test_case_finish:
        - @echo The test case was finished
prog1:
    command: ./test/totest.sh
    args: %input_file_name
    pipes_out:
        - echo 15 > %output
prog2:
    command: echo 159

Custom output format

Using --tsilent flags allows only hooks to write output. So if you use @ sign along with hooks (see Hooks section) you can make utest output any format of the output you want!

Example of outputing ERR <file> only on errors.

hooks:
    test_case_fail:
        - @echo ERR %{input_file}

Simple enough, right?