m (Fixed the MeeGo wiki links) |
(→Test Definition) |
||
| Line 123: | Line 123: | ||
* XML format can be found [http://gitorious.org/qa-tools/test-definition here]. | * XML format can be found [http://gitorious.org/qa-tools/test-definition here]. | ||
* One test package corresponds one test case definition file | * One test package corresponds one test case definition file | ||
| + | * Test definition file should be moved to "/usr/share/<testset>-tests/" during the install process. | ||
| + | * If the file containing all test definitions is called something other than "tests.xml", this should be mentioned in the README. | ||
== Makefile (or autoconf/automake) == | == Makefile (or autoconf/automake) == | ||
Contents |
While the importance of testing needs little explaining, the relevance of maintaining quality standards may be less obvious. The standards of test documentation, packaging etc. are not as important when used for internal testing but when external users need to make use of the tests, ease of use and consistent packaging become important. MCTS will be used in hardware acceptance testing by companies not related to the test development therefore, it is important that the tests are easy to install, run and understand.
Packages should all conform to the same standards and be self contained. This way, the user of the tests will not have to search for information/files.
Test package names should have a "–tests" suffix if they are not part of a dedicated test suite.
Small data files (a few KB) should be included with the tests. Larger files should be made available separately. Instructions for obtaining the data files should be placed in the README file.
Test data should be publicly available (MeeGo git, public media server etc.).
A basic template for a README file can be found here. It is important to remember that the user who will be running the tests in the package may not have had previous experience with tests. Thus, the instructions should be as simple and explicit as possible.
The README file should also contain a list of tests and a description of them. If such a list would be too long, make it a separate file and mention this file in the README.
While the description of the environment can be included in the README file, for complex environments, it may be worthwhile to include a separate file. The environment description needs to be very explicit, *do not* assume that the user knows anything about your test environment.
Where possible, tests should not depend on proprietary components (hardware or software). All environmental needs should be such that they are available to anyone in the community.
The environment includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.
For example:
"A wireless network called 'test_net' must be accessible by the device and this network needs to be connected to the internet"
This is not adequate. Several questions are left unanswered, such as: is a DHCP server expected on the network? Are specific ports required to be opened?
Descriptions of environments should be explicit and parameters (such as network name) should be modifiable in a configuration file or passed as a parameter. *Do not* make the tester dig through source code to find the requirements for the testing environment.
It is acceptable to point a user to a settings or configuration file to obtain environment information, provided that the file is commented and easy to read.
With the exception of a list of tests and their description, any further information that is not critical to the running od the tests should be placed in a separate file or on the internet (with a link in the README file).
Must contain:
A template can be found [[Mcts-header|here].
A common coding-style rule of thumb is provided here. Bear in mind that consistency and readability are the primary concerns. If a library that the tests are heavily tied to breaks these common guidelines, then it is acceptable to follow the coding standard of the library. More detailed code style instructions for specific languages can be found here:
While the style guide suggests commenting blocks of code rather than individual lines, but any line of code that is not self-explanatory should be commented individually (code readability is key, so keep that in mind when deciding what, and how much, to comment).
Relative file references should always be relative to the location of the script. For example, in Python, instead of using:
to refer to a directory 2 levels up from the location of the python script itself, use:
While this requires the additional import of the 'os' module and is definitely a 'quick-and-dirty' solution, it will give the directory of the currently running script regardless of where it is run from (e.g. if the script is run from another directory).
The method used to run the tests must be described in detail in the README file. To allow for inexperienced users, provide a sample command line: e.g. To run the tests use: './mcts_package_test_name.py'
several other points should be kept in mind about the tests while they run:
Tests should set up their own environment and clean the environment again at the end.
NOTE: The environment includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.
Tests cases should always generate log and data files. This allows for easier analysis of results and problems, especially if the running of test cases is automated.
make : build test suite.
make install : install whole test suite (binary, script, utils, helper, data) into /opt/{PACKAGE}, and tests.xml into /usr/share/<testpackagename>/tests.xml
make clean : remove all object file and temporary files
make uninstall : remove test binary, scripts, and tests.xml out of target installation directory.
While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This should be avoided.
Following rules shall be followed by test developer. Deviation from these rules are allowed only with very good grounds:
Parameters under test should be verified before and after the test.
Example: BT power-on test. Check that the BT device is off, if it is already on, then the test case may pass (because the device is on at the the end) but the test has not actually turned the device off. If the power is already on before the test starts, turn it off, then have the test turn it on. Do not accept an API call executing with no errors as a 'pass' result, check that the BT device is on after the test. At the end, return environment to default configuration.
In order to meet the environment setup/tear-down requirements, a test case cannot be overly simple: e.g. a wifi data transfer case must enable a wifi device, detect a network, connect to it, transfer data and then close the connection. In this case, there is no need to write a specific network detection, connect to network or disconnect from network test as all of these functions are tested by the data transfer test.
Rather write fewer test-cases that are more flexible and thorough. The objective is not to have many tests but to test the relevant API to the required extent.
Parameters should be read from a configuration file wherever possible instead of being hard-coded. The configuration file should be commented (to explain what each parameter does) so that it is easy to edit. The location of the configuration file and the tests that it applies to should be described in the README file.
For example, a test that uses a GSM network to dial number and then hang up should have any network parameters and phone number (and perhaps the time before hanging up) described in a configuration file.
For tests where 1 or 2 parameters change frequently (and tests need to be re-run with the new parameters), it is acceptable to pass these as arguments to the test command line. It may be beneficial to have these parameters in the configuration file anyway, and simply have the values that are passed as arguments take priority (overwrite) the ones in the file while the test is running.
For a detailed description of how to measure coverage, and what coverage actually means, please see here.
Test coverage comes in 3 levels, a description of the levels follows below: