Latest revision as of 12:35, 2 December 2010

MeeGo Core Test Suite Quality Standards

Introduction and Importance

While the importance of testing needs little explaining, the relevance of maintaining quality standards may be less obvious. The standards of test documentation, packaging etc. are not as important when used for internal testing but when external users need to make use of the tests, ease of use and consistent packaging become important. MCTS will be used in hardware acceptance testing by companies not related to the test development therefore, it is important that the tests are easy to install, run and understand.

Packaging

Packages must all conform to these standards and be self contained. This way, the user of the tests won't need to search for information/files.

In package names, words should be separated with a dash ('-').

For MCTS, adding a '-tests' suffix is not required as MCTS is a dedicated test suite. For test packages not contained in a dedicated test suite, the '-tests' suffix must be added.

Files

• README (See Documentation section below)
• src/ : contain source files
• utils/: contain utility and tools if any
• Makefile (or Makefile.am, configure.ac, autogen.sh, qmake)
• tests.xml (alternatively <packagename>.xml, should the tests all be thrown into the same folder, this needs to be documented in the README)
• data/: contain small data files (For big data such as media content, need separate package)
• pack.sh: the script to generate rpm package
• test.spec: spec files to create rpm package

File Naming Structure

• Use descriptive names (ie. ftp_file_send)
• Do not use numbers as test names (Numbers really don't tell the user what the test is supposed to test)
• Name length should be less than 30 characters long
• Use '_' to connect words in file names (except in package names)

Test Data

Small data files (a few KB) should be included with the tests. Larger files should be made available separately. Instructions for obtaining the data files must be placed in the README file.

Test data must be publicly available (MeeGo git, public media server etc.).

Documentation

README File

A basic template for a README file can be found here. It is important to remember that the user who will be running the tests in the package may not have had previous experience with tests. Thus, the instructions should be as simple and explicit as possible.

The README file should also contain a list of tests and a description of them. If such a list would be too long, make it a separate file and mention this file in the README.

Description of Environment

While the description of the environment can be included in the README file, for complex environments, it may be worthwhile to include a separate file. The environment description needs to be very explicit, *do not* assume that the user knows anything about your test environment.

Where possible, tests should not depend on proprietary components (hardware or software). All environmental needs should be such that they are available to anyone in the community.

The environment includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

For example: "A wireless network called 'test_net' must be accessible by the device and this network needs to be connected to the internet" This is not adequate. Several questions are left unanswered, such as: is a DHCP server expected on the network? Are specific ports required to be opened? Descriptions of environments should be explicit and parameters (such as network name) should be modifiable in a configuration file or passed as a parameter. *Do not* make the tester dig through source code to find the requirements for the testing environment. It is acceptable to point a user to a settings or configuration file to obtain environment information, provided that the file is commented and easy to read.

Further Documentation

With the exception of a list of tests and their description, any further information that is not critical to the running od the tests should be placed in a separate file or on the internet (with a link in the README file).

Source Code

Header

Must contain:

• Copyright
• License
• Author
• Contact details (email address) for maintainer
• Changelog (optional)

A template can be found [[Mcts-header|here].

Coding Style

A common coding-style rule of thumb is provided here. Bear in mind that consistency and readability are the primary concerns. If a library that the tests are heavily tied to breaks these common guidelines, then it is acceptable to follow the coding standard of the library. More detailed code style instructions for specific languages can be found here:

• C/C++
• Python
• Shell

Note on Commenting

The code style guide suggests commenting blocks of code rather than individual lines, but any line of code that is not self-explanatory should be commented individually (code readability is the key, so keep that in mind when deciding what, and how much, to comment).

Note on Relative References

Relative file references should always be relative to the location of the script. For example, in Python, instead of using:

to refer to a directory 2 levels up from the location of the python script itself, use:

While this requires the additional import of the 'os' module and is definitely a 'quick-and-dirty' solution, it will give the directory of the currently running script regardless of where it is run from (e.g. if the script is run from another directory).

Running the Tests

The method used to run the tests must be described in detail in the README file. For inexperienced users, provide a sample command line: e.g. To run the tests use: './mcts_package_test_name.py' Several other points should be kept in mind about the tests while they run:

Environment Handling

Tests should set up their own environment and clean the environment again at the end.

• Test must set up its own environment, any requested parameters should be set by the test itself.
• Test runs in desired environment
• Test cleans up and restores environment to original state or to a default state.

NOTE: The environment includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

Test Granularity

• Tests must be able to run in *any order* i.e. no test should depend on another test.
• Running one test must not prevent another test from running afterwards (i.e. do not damage the environment to the point that no other tests can run)
• Tests must be able to run individually. i.e. be able to run individual tests, not just whole packages.
• Thus test cases can be executed at any order, test cases can be skipped and one or certain test cases can be repeated multiple times.

Log and Result Files

Tests cases should always generate log and data files. This allows for easier analysis of results and problems, especially if the running of test cases is automated.

• Result files
  • Must contain a Pass/Fail result or a measurement result etc.
  • Must contain test names, so that the user can tell which result corresponds to which test
  • If the test fails, major failure cause should be written to the result file
  • Should contain passed steps: e.g.: Checked power on, turned power off, turned power on data transfer, turned power off
  • Do not rely on what the API returns, check everything (see Result Verification Section for more information)
• Log files
  • Log files must be detailed in the README file so that they can be found.
  • Log files must contain test results.
  • Reason for test failure should be evident from the log and result files, the developer should not need to run tests to find the cause of failure.
  • It is suggested that log files be verbose and contain (at least) the following:
    • Functions Visited
    • Parameters passed (and their values) - (for tests that generate very long log files, there should be an option to disable this)
    • When a critical function or API call is passed, this should be noted

Test Definition

• XML format can be found here.
• One test package corresponds one test case definition file
• Test definition file should be moved to "/usr/share/<testset>-tests/" during the install process.
• If the file containing all test definitions is called something other than "tests.xml", this should be mentioned in the README.
• All tests must be included in the tests.xml file.
• The tests.xml file should always be up to date.

Makefile (or autoconf/automake)

• make : build test suite.
• make install : install whole test suite (binary, script, utils, helper, data) into /opt/{PACKAGE}, and tests.xml into /usr/share/<testpackagename>/tests.xml
• make clean : remove all object file and temporary files
• make uninstall : remove test binary, scripts, and tests.xml out of target installation directory.

Test Design

The Objective of Testing

While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This should be avoided.

• Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
• Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
• Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
• The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

Rules to Remember

Following rules shall be followed by test developer. Deviation from these rules are allowed only with very good grounds:

• Clear result: Test case should return clear Pass/Fail result
• Self explanatory: Test case should be easy to understand
• Avoid duplicated code: Use common functions or classes as much as possible
• Automate test under condition of stability
• Main function should kept simple and structure kept neat.
• For across-test common function, create separated lib file.
• No dependency on other test case.
• No dependency on UX or vertical specific apps.
• Use relative path to access dependency files inside test package such as helper, utilities and data (see notes on relative referencing above)
• Clean environment after test complete (see Test Granularity section).
• Enough comments (see comments section above).
• Use common Linux functions, avoid rewriting functionality that already exists.
• Avoid complicated code logic (if you have to use it, comment it!)
• Remove useless code.
• Use Timeouts to prevent test-case stall.

Automation

Automation is of great importance when testing is time sensetive and resources are limited. Where possible, tests should be automated as much as possible.

NOTE: A test is considered a manual when the 'manual="true"' field is set in the case definition.

Result Verification

Parameters under test should be verified before and after the test.

Example: BT power-on test. Check that the BT device is off, if it is already on, then the test case may pass (because the device is on at the the end) but the test has not actually turned the device off. If the power is already on before the test starts, turn it off, then have the test turn it on. Do not accept an API call executing with no errors as a 'pass' result, check that the BT device is on after the test. At the end, return environment to default configuration.

Test case complexity

In order to meet the environment setup/tear-down requirements, a test case cannot be overly simple: e.g. a wifi data transfer case must enable a wifi device, detect a network, connect to it, transfer data and then close the connection. In this case, there is no need to write a specific network detection, connect to network or disconnect from network test as all of these functions are tested by the data transfer test.

Rather write fewer test-cases that are more flexible and thorough. The objective is not to have many tests but to test the relevant API to the required extent.

Dealing with Parameters

Parameters should be read from a configuration file wherever possible instead of being hard-coded. The configuration file should be commented (to explain what each parameter does) so that it is easy to edit. The location of the configuration file and the tests that it applies to should be described in the README file.

For example, a test that uses a GSM network to dial number and then hang up should have any network parameters and phone number (and perhaps the time before hanging up) described in a configuration file.

For tests where 1 or 2 parameters change frequently (and tests need to be re-run with the new parameters), it is acceptable to pass these as arguments to the test command line. It may be beneficial to have these parameters in the configuration file anyway, and simply have the values that are passed as arguments take priority (overwrite) the ones in the file while the test is running.

Test Coverage

For a detailed description of how to measure coverage, and what coverage actually means, please see here.

Test coverage comes in 3 levels, a description of the levels follows below:

• Thorough
  • Cover all APIs.
  • Test case targets for usage (API combination) instead of single API.
  • With Parm-pairs of equivalence partitioning and boundary condition (All-pairs optional)
• Average
  • High risk APIs(e.g. new functionality relevant APIs) should be covered with thorough test rules
  • Legacy or low risk APIs are covered only with basic tests
• Light
  • Test most critical functionality