Revision as of 06:16, 29 December 2010

Testrunner-lite

Testrunner-lite is a lightweight tool for test execution, which reads Test Plan XML files as input, and produces a common-format test result file which can be published at qa-reports.meego.com. With testrunner-lite, you can

• Execute automatic, semi-automatic and manual test cases
• Execute test cases locally or in host-based mode
• Use filters to select the test cases to be executed
• Validate the used test plan file automatically

See our demo videos at Youtube:

• Demo of version 1.3.17
• Demo of version 1.3.11
• Demo of version 1.3.10
• MeeGo QA tools Youtube channel

There is also a graphical user interface version of the tool called Testrunner available.

Installation

Testrunner-lite can be installed on MeeGo Netbook UX, Ubuntu 10.04 (and newer) and Fedora 13. First, set up the repository. Then, install testrunner-lite:

MeeGo Netbook 1.1:

 sudo zypper install testrunner-lite

Ubuntu:

sudo apt-get install testrunner-lite

Fedora 13:

# yum install testrunner-lite

Usage

The program is executed from command line

/usr/bin/testrunner-lite [options]
Options

-h, --help
Display short help and exit

-V, --version
Display version information and exit

-f FILE, --file FILE
 Input file with test definitions in XML (required)

-o FILE, --output FILE
 Output file format; FORMAT can be xml or text (Default xml)

-v, -vv, --verbose[={INFO|DEBUG}]
 Enable verbosity mode; -v and --verbose=INFO are equivalent outputting INFO, ERROR and WARNING messages. Similarly -vv and --verbose=DEBUG are equivalent, outputting also debug messages. Default behaviour is silent mode.

-L URL, --logger=URL
 Remote HTTP logger. Log messages are sent to given URL in a HTTP POST message. URL format is [http://]host[:port][/path/], where host may be a hostname or an IPv4 address.

-a, --automatic
 Enable only automatic tests to be executed

-S, --syslog
 Enable logging to syslog

-P,  --print-step-output
 Output standard streams from test steps.

-m, --manual
 Enable only manual tests to be executed

-c, --ci
 Disable validation of test definition against schema

-s, --semantic
 Enable validation of test definition against stricter (semantics) schema

-A, --validate-only
 Do only input xml validation, do not execute tests

-H, --no-hwinfo
 Do not try to obtain hardware information

-t [USER@]ADDRESS, --target=[USER@]ADDRESS
 Enable host-based testing. If given, commands are executed from test control PC (host) side. ADDRESS is the ipv4 adress of the system under test.

Filtering options

Filtering options allow selecting which tests are to be executed. Filtering options are given as a string. String may contain one or more filter entries. Format of each entry is 'filtername=values' where 'filtername' typically corresponds to an attribute in Test Definition XML and 'values' is a string to which the filter should match. Multiple values can be specified if separated by comma (without extra space). A value containing space must be enclosed in double quotes (""). Current list of forbidden characters in values is as follows: Single quote ('), double quote ("), is-equal-to sign (=), comma (,).

How filtering is carried out: Before processing the filter options, all tests are active by default. Filters can only deactivate tests: If filter value does not match with the value of the corresponding attribute, the test is disabled. Each filter entry is applied for all the active tests in the test package, one after another, in the given order. Filtering is always carried out at the lowest level of the "suite-set-case-step" hierarchy where the corresponding attribute can be defined. Note that some attributes may inherit the value from the upper level. Please refer to Test Definition XML for details.

If 'filtername' is prefixed with dash (-), the filter behaviour is reversed: those tests for which the filter value does match, are disabled.

'filtername' can be any of the following:

   * feature
   * requirement
   * testset
   * type
   * testcase

The following example does the following: First it disables all test cases except those with attribute type='Integration'. Next, a test case named as Bad Test is disabled. The tests left active will be executed.

testrunner-lite -f tests.xml -o ~/results -l 'type=Integration -testcase="Bad Test"'

The following example executes test cases that specify the requirement attribute with a value containing at least one of the following strings: '50001', '50002', '50003'.

testrunner-lite -f tests.xml -o ~/results -l 'requirement=50001,50002,50003'

Note that each key=values is handled as a separate filter, when checking whether a test case should be filtered. The following example will filter all the test cases,

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1 testset=set2'

whereas the following will accept tests from test sets "set1" and "set2".

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1,set2'

Manual Test Cases

It is also possible to execute manual test cases using testrunner-lite, as defined in Test Definition XML.

In case manual test case is encountered during execution, testrunner-lite will go through the defined test steps and ask user whether the step is passed or failed. The test case will terminate at the first failure, otherwise every step defined will be executed. After the test case is done, user has the option to enter additional comments.

Example output when running manual case:

[INFO] 15:15:53 Starting test case: ExampleTestCase
--- Execute test step ---
Description:
Open calculator. Expected result: calculator opens up.

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Execute test step ---
Description:
Stop calculator

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Test steps executed, case is PASSED ---
Please enter additional comments (ENTER to finish):
Execution was slow.

[INFO] 15:16:41 Finished test case. Result: PASS

About process control

Each test step is executed in a separate shell. Testrunner-lite spawns new process for the execution, and waits for the step to finish (or timeout). In case test step contains command that is started to background, the step returns immediately. After test case has finnished a cleanup routine is executed, where testrunner-lite tries to kill all processes that may have been left running by the test steps. Cleanup for pre-steps and post-steps is done after the post steps are executed (i.e. when test set has been executed).

About host based execution

Testrunner-lite supports host based execution, where testrunner-lite is executed on a PC, and the test steps over ssh on hardware. This requires that key based ssh authentication is enabled between device and host.

Remember that each single step in pre_steps, post_steps or inside testcases is executed in a separate SSH session, so you'll have to make sure that if your steps leave some processes running in the background which inherits its pipes straight from the parent shell, the SSH connection will hang until those processes are terminated (or until they close their pipes). Therefore, you should always redirect the stdout, stderr and stdin streams of your background processes, if you don't want your test steps / pre- / post steps to time out in host-based execution.

See http://www.snailbook.com/faq/background-jobs.auto.html for more information.

Testrunner-lite state machine

Below is a simplified picture describing the operation of testrunner-lite. The program flow can be considered to consist of three phases: initializing, executing and clean up. The execution is done set by set, so that the tool needs to maintain only one set at a time in its memory.

xfig file: File:Testrunnerlitesm.fig

Further development

We're actively making testrunner-lite a better tool. See discussion about architecture changes and suggestions in following page: Testrunner-lite development