Testkit Howto

Latest revision as of 13:40, 20 June 2011

How to Use Testkit

Overview

Testkit is one testing execution environment, which is composed by testkit-manager and testkit-lite. Testkit-manager represents a user interface front-end in browser, with the webUI interface, it manages the auto tests execution remotely, and provides unified webUI to help the manual tests execution. Testkit-lite is the default core test runner driving test execution in testkit-manager and it also provides more comprehensive options for your testing in command line mode.

Source Code Git

testit-lite git

testkit-manager git

Installation

1. Download testkit-lite and testkit-manager packages(rpm) from the download page, or source code from gitorious testit-lite git testkit-manager git

2. Install the packages.

rpm -ivh testkit-lite-<version>.<architecture>.rpm
rpm -ivh testkit-manager-<version>.<architecture>.rpm

Before that, you might need to install the following dependency packages:

perl-XML-Simple, perl-XML-SAX, perl-JSON, man, python-lxml

3. Startup the testkit-manager web server

/opt/testkit/manager/webapps/bin/testkit_start.pl [port-number]

4. Visit the testing page: http://test-machine-name:8890

Test Package Requirements:

Test package should follow the guideline: http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline

• The test packages for testkit should have one tests.xml which aligns with schema file testdefinition-syntax.xsd.
• The tests.xml should be settled down to /usr/share/<packagename>.
• The test materials (script, binary, data …) is always under /opt/<packagename>.

Using Web UI

1. In the browser you will find the tests packages overview in page “VIEW TEST”. Click each package

2. To run the target tests packages, click page “SELECT TEST”. Then tick target test packages and click “Run Tests”.

3. To save time to select the same test packages the next time, save the selection set as one profile.

4. After “Run Tests” is clicked in page “SELECT TEST”, it jumps to page “EXECUTE TEST” and start the auto tests execution.

5. In page “EXECUTE TEST”, you can also select the existed profile directly and click “start” to start the execution, during the execution, could “stop” the current execution.

6. After execution ends from page “SELECT TEST”, it jumps to page “DETAIL TEST REPORT”. In this page, you can check the execution brief summary.

7. Click “VIEW TEST LOG” to view the detail execution log.

8. Click “VIEW HISTORY” to see all the test history entry link. This page could be also redirected by “TEST REPORT” tab.

9. If the selected tests package include some manual tests, click “RUN MANUAL TEST” in page “DETAIL TEST REPORT” in step 6.

Click each test from the left frame, view the detail steps and then select PASS, FAIL or N/A, “SAVE RESULT” to save this test’s result.

After all manual tests are executed, click “FINISH TEST” to end up the manual tests execution. Note that the once the “FINISH TEST” is clicked, it means one round of test cycle is finished that no chance to modify the test results any more. The “RUN MANUAL TEST” button in “DETAIL TEST REPORT” will be removed.

Using Command Line UI

Testkit-lite is the default core test runner in testkit-manager and it provides more comprehensive options for your testing in command line mode.

How to execute

1. You can simply conduct one or more tests.xml:

testkit-lite -f <somewhere1>/tests.xml <somewhere2>/tests.xml

2. You can also arrange several tests.xml to one testsxmlconfig, and run:

testkit-lite --testxmlconfig <somewhere1>/testsxmlconfig

Also, it supports multi testsxmlconfig:

testkit-lite --testxmlconfig <somewhere1>/testsxmlconfig <somewhere2>/testsxmlconfig

The testsxmlconfig is like this:

/opt/browser/tests.xml
/opt/ofono/tests.xml
/opt/gfx/tests.xml
/opt/pulseaudio/tests.xml

3. If you want to validate one new tests.xml:

testkit-lite -f tests.xml -V

4. If you just want to get the statistic (such as the testcases number or the structure), dryrun could help:

testkit-lite -f tests.xml -D

5. If you want to execute auto tests:

testkit-lite -f tests.xml

6. If you just want to execute manual tests:

testkit-lite -f tests.xml -M

7. If you want to execute all the auto and manual tests:

testkit-lite -f tests.xml -A

8. If you just want to execute the significant tests:

testkit-lite -f tests.xml -S

9. If you want to choose some filters:

testkit-lite -f tests1.xml tests2.xml --level level1 level2 --type type1 type2 --Ntestcase gfx1 gfx2...

Note that the test cases will be filtered out by both black rules and white rules.

10. At last, you can freely compose the above paramters together:

testkit-lite -f tests1.xml tests2.xml --testxmlconfig config1 config2 -D -A -S -C --level level1 level2 --type type1 type2 ...

11. One more parameter is prepared for customized compatible mode for test report:

testkit-lite -f tests.xml -C

12. Notes:

testkit-lite enables only automatic tests by default. Obviously -A and -M are conflict options.

How to get result

1. testkit-lite's TestLog is stored to /opt/testkit/lite/latest after execution, you can also check the history results in /opt/testkit/lite/yyyy-mm-dd-HH:MM:SS.NNNNNN.

2. Two kinds of test report will be generated:

1)<tests xml name>.textresult, e.g.

2)<tests xml name>.xmlresult

xml result files aligned with schema files: /opt/testkit-lite/xsd/testdefinition-results.xsd, e.g.

Detail Options of testkit-lite: