Revision as of 09:29, 19 December 2011

OTS - Open Test System

Related Documentation

• Development Guidelines
• Packages
• Error Codes
• Quality/QA-tools/OTS/Glossary
• Making transition from 0.1 to 0.8

Synopsis

OTS is a Distributed Test System that takes a Data Driven Approach using XML Files to allow parallel testing of Applications on evolving Hardware platforms.

Design has been driven by the need for a language neutral test platform that allows testing under different device architectures in parallel.

High Level Overview

OTS allows a Test Sequence to be fanned out from a hub known as a Distributor to a remote machine known as a Worker. Each Worker has a Device Group associated with it, this is defined in the configuration. The Tests are routed to the Worker on the basis of the Device Group.

An XML Test Definition File describes the Tests that are run. The results are returned to the Distributor in an XML Results File.

The Distributor takes a Device Group and Timeout as inputs, as well as the Command for running Tasks. Tests are run within a Task which is currently run as a process. The Command will typically take a path to the Test Definition File as an input. If the process exceeds the Timeout the Task is stopped (the process is killed).

A number of Tasks can be added to a Testrun. The Testrun will normally wait for all Tasks to finish before completion.

Data (status, error and results) are communicated back from the Task to the Worker. The Results Object is a container for the files outputted: Test Definition File, Results File and any other files produced by the test as well as metadata relating to the run.

Technology

OTS is written in Python 2.6.

AMQP protocol is used for communication.

XmlFileFormats

The XML file formats provide the datum for the system.

Download them and add the path to the results_xsd parameter in your server.conf file

Third Party Dependencies

RabbitMQ

You need to install RabbitMQ, this is AMQP message server.

Managing the Queues

If you need to delete or empty the queues there are utilities to help with this.

Python Packages

The following python packages are needed

• setuptools

• configobj

• multiprocessing

• pyamqplib (version 0.6.1 or newer)

• minixsv

• django

• nose

Diving In

The following steps show how to get a Development Environment up and running

1. Install Dependencies

• Install the Third Party Dependencies

• Get the XML File Formats and set the OTS_TESTDEFINITION environment variable

2. Build the Eggs

From the root directory

./setup.sh
source ./paths.sh

3. Unittests

Run the unittests from the root directory

./nose.sh

4. Hello OTS World

The source contains a simple demonstration of the way the key elements work.

1. Open two terminal windows.

2. In the first window run the server

cd $SERVER
cd hub/demos
python demo_hub.py

3. You should see this:

<snip>
20XX-XX-XX XX:XX:XX,XXX - ots.server.distributor.taskrunner - DEBUG - Sending command '['echo', 'hello world']' with key 'foo'

4. In the second window run the worker

cd $WORKER
python worker.py -c config.ini

5. Now in the Server terminal the command should run to completion. You should see the following line in the logs:

 20XX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world

It will raise a PackageException. This is normal as no Tasks that are run return Packages to the Server

6. And on the logs on the Worker side should contain:

XXXX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - DEBUG - Task in state: 'finish'
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - INFO - Recommence consume on queue: foo

Installation

It is intended that the Distributor and each Worker should be installed to separate machines.

Configuration

Distributor

The format of the configuration file is as follows.

The `host` is the name of the RabbitMQ server

[Client]
host =  localhost
vhost = / 
port = 5672
username = guest
password = guest
queue = FIXME
consumer_tag = worker
services_exchange = services

Worker

Edit your config.ini so that the

* `queue`
* `routing_key`
* `services_exchange` 

Are set for your Device Group.

And the `host` is the name of your RabbitMQ server.

[Worker]
host =  localhost
vhost = / 
port = 5672
username = guest
password = guest
queue = foo
routing_key = foo
services_exchange = foo

Starting the Worker

To start your Worker:

python worker.py -c config.ini

Co-operation with testrunner-lite

OTS worker starts conductor (part of OTS) for each test run. Conductor starts testrunner-lite, a tool for running tests on remote devices according to Test Plan XML. More information in testrunner-lite wiki: http://wiki.meego.com/Quality/QA-tools/Testrunner-lite.

Usage:

testrunner-lite [options]

Example:

testrunner-lite --file=tests.xml --output=~/results.xml --verbose

For a complete list of options run

testrunner-lite --help

How OTS conductor starts testrunner-lite

Conductor starts testrunner-lite using command line interface (CLI) inside the same shell. Conductor runs a base testrunner-lite command with some extra options resolved from the test run options.

Base command example without extra options:

/usr/bin/testrunner-lite --semantic --verbose --automatic --ci --file=/path/to/some-package-tests.xml --output=/path/to/some-package-results.xml --logid=1

Example with some extra options currently supported by OTS:

/usr/bin/testrunner-lite --semantic --verbose --automatic --ci --file=/path/to/some-package-tests.xml --output=/path/to/some-package-results.xml --logid=1 --filter="some-test" --logger=/logger/ots/%s/ --target=root@some.host --rich-core-dumps="/path/to/richcore-dumps" --resume=exit

Signaling between OTS conductor and testrunner-lite

There are currently three cases where conductor and testrunner-lite communicate using POSIX signaling.

Connection failure

Edit page for the sequence diagram

Automated System Tests

Automated system tests are available in ots.system_tests/log_tests/log_tests.py. They trigger real testruns with xmlrpc and check return value and log messages.

Requirements

• Fully functional OTS system with one worker.
• A SW Product ots-system-tests configured so that it defaults to the worker.
• A meego image with test-definition-tests and testrunner-lite-regression-tests.
• Django based advanced OTS setup.
• BeautifulSoup python library.
• ots.plugin.test.log_publisher installed on server

Configuration

• make the system test image available for the workers. (for example copy it to your apache static content directory /var/www/static/)
• Create a log_tests.local.conf in ots.system_tests/log_tests directory. The values in log_tests.local.conf override values in log_tests.conf.
• Make sure all configured urls work.

Notes

• System tests execute over 20 full testruns. Running all tests can take 24 hours with N900 worker, mainly because of flashing the device. Without flashing the test should run in about 3 hours.
• Some error handling tests expect certain devicegroups and sw products _not_ to be available. If these test cases fail you can remove those queues with ots tools.
• You can run a single test like this:

python log_tests.py TestErrorConditions.test_connection_failure_without_resume

• To run all the tests and in parallel check ots.system_tests/log_tests/run-all-tests.sh for further instructions.

Running tests on CI

Reusable scripts exists for running unit and integration tests on Jenkins CI. They can be found in ci/jenkins directory.

License

OTS is distributed under an LGPL license.

Contributions

OTS welcomes contributions.

The following areas would benefit greatly from your help:

• Adding to the list of freely available Publisher Plugins
• Examples of OTS in use with different hardware architectures
• Improving the ease of deployment

Please check out the Guidelines and Review process.

Wiki

OTS Wiki Home

Source Code Documentation

OTS docstrings are in epytext format which enables the use of epydoc in API documentation generation.

Mailing List

MeeGo QA mailing list

IRC Channel

The #meego-qa irc channel in freenode

Bugs

The Meego bugs page

Platform Dependencies

OTS is tested to run on Linux only.

Tested on Ubuntu (8.04, 10.04, 10.10), Fedora 13, MeeGo and Redhat.

Binary support for Ubuntu, Fedora and MeeGo.

Architecture

• Components

• Server

• Plugins

• DataFlow

• Behaviour

APIs

Server

The top level api for external clients utilises parameters in a dictionary.

The parameters are explicitly defined in options.py. The conversion from the loosely defined dictionary to a clear API is undertaken by the options_factory.py

Distributor

Results

Worker

Extending OTS

You can extend OTS with your own Publisher and with custom package distribution models.

Experimental

Documentation for experimental branches, spikes, WIPs etc. can be found in Quality/QA-tools/OTS/DeveloperDocs/Experimental.

Analysis

OTS 0.1 Error Situations

Typical OTS users and usage scenarios

Project Planning

• Roadmap
• Meetings
• ReleaseChecklist

Release Practices

OTS releases are done when a new release makes sense from OTS point of view. There are no scheduled releases.

The QA Tools Release Practices gives more details.