OTS is a Distributed Test System that takes a Data Driven Approach using XML Files to allow parallel testing of Applications on evolving Hardware platforms.
Design has been driven by the need for a language neutral test platform that allows testing under different device architectures in parallel.
OTS allows a Test Sequence to be fanned out from a hub known as a Distributor to a remote machine known as a Worker. Each Worker has a Device Group associated with it, this is defined in the configuration. The Tests are routed to the Worker on the basis of the Device Group.
An XML Test Definition File describes the Tests that are run. The results are returned to the Distributor in an XML Results File.
The Distributor takes a Device Group and Timeout as inputs, as well as the Command for running Tasks. Tests are run within a Task which is currently run as a process. The Command will typically take a path to the Test Definition File as an input. If the process exceeds the Timeout the Task is stopped (the process is killed).
A number of Tasks can be added to a Testrun. The Testrun will normally wait for all Tasks to finish before completion.
Data (status, error and results) are communicated back from the Task to the Worker. The Results Object is a container for the files outputted: Test Definition File, Results File and any other files produced by the test as well as metadata relating to the run.
OTS is written in Python 2.6.
AMQP protocol is used for communication.
The XML file formats provide the datum for the system.
You need to install RabbitMQ, this is AMQP message server.
If you need to delete or empty the queues there are utilities to help with this.
The following python packages are needed
The following steps show how to get a Development Environment up and running
From the root directory
./setup.sh source ./paths.sh
Run the unittests from the root directory
The source contains a simple demonstration of the way the key elements work.
cd $SERVER cd hub/demos python demo_hub.py
<snip> 20XX-XX-XX XX:XX:XX,XXX - ots.server.distributor.taskrunner - DEBUG - Sending command '['echo', 'hello world']' with key 'foo'
cd $WORKER python worker.py -c config.ini
20XX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world
It will raise a PackageException. This is normal as no Tasks that are run return Packages to the Server
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - DEBUG - Task in state: 'finish' XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - INFO - Recommence consume on queue: foo
It is intended that the Distributor and each Worker should be installed to separate machines.
The format of the configuration file is as follows.
The `host` is the name of the RabbitMQ server
[Client] host = localhost vhost = / port = 5672 username = guest password = guest queue = FIXME consumer_tag = worker services_exchange = services
Edit your config.ini so that the
* `queue` * `routing_key` * `services_exchange`
Are set for your Device Group.
And the `host` is the name of your RabbitMQ server.
[Worker] host = localhost vhost = / port = 5672 username = guest password = guest queue = foo routing_key = foo services_exchange = foo
To start your Worker:
python worker.py -c config.ini
OTS worker starts conductor (part of OTS) for each test run. Conductor starts testrunner-lite, a tool for running tests on remote devices according to Test Plan XML. More information in testrunner-lite wiki: http://wiki.meego.com/Quality/QA-tools/Testrunner-lite.
testrunner-lite --file=tests.xml --output=~/results.xml --verbose
For a complete list of options run
Conductor starts testrunner-lite using command line interface (CLI) inside the same shell. Conductor runs a base testrunner-lite command with some extra options resolved from the test run options.
Base command example without extra options:
/usr/bin/testrunner-lite --semantic --verbose --automatic --ci --file=/path/to/some-package-tests.xml --output=/path/to/some-package-results.xml --logid=1
Example with some extra options currently supported by OTS:
/usr/bin/testrunner-lite --semantic --verbose --automatic --ci --file=/path/to/some-package-tests.xml --output=/path/to/some-package-results.xml --logid=1 --filter="some-test" --logger=/logger/ots/%s/ --firstname.lastname@example.org --rich-core-dumps="/path/to/richcore-dumps" --resume=exit
There are currently three cases where conductor and testrunner-lite communicate using POSIX signaling.
Automated system tests are available in
ots.system_tests/log_tests/log_tests.py. They trigger real testruns with xmlrpc and check return value and log messages.
ots-system-testsconfigured so that it defaults to the worker.
ots.plugin.test.log_publisherinstalled on server
ots.system_tests/log_testsdirectory. The values in
log_tests.local.confoverride values in
python log_tests.py TestErrorConditions.test_connection_failure_without_resume
ots.system_tests/log_tests/run-all-tests.shfor further instructions.
Reusable scripts exists for running unit and integration tests on Jenkins CI. They can be found in
OTS is distributed under an LGPL license.
OTS welcomes contributions.
The following areas would benefit greatly from your help:
MeeGo QA mailing list
The #meego-qa irc channel in freenode
The Meego bugs page
OTS is tested to run on Linux only.
Tested on Ubuntu (8.04, 10.04, 10.10), Fedora 13, MeeGo and Redhat.
Binary support for Ubuntu, Fedora and MeeGo.
The top level api for external clients utilises parameters in a dictionary.
Documentation for experimental branches, spikes, WIPs etc. can be found in Quality/QA-tools/OTS/DeveloperDocs/Experimental.
OTS releases are done when a new release makes sense from OTS point of view. There are no scheduled releases.
The QA Tools Release Practices gives more details.