Views

Quality/QA-tools/OTS/DeveloperDocs

From MeeGo wiki

< Quality | QA-tools | OTS

Revision as of 14:22, 18 March 2011 by Tgalvin (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Jump to: navigation, search

1 OTS - Open Testing Service

OTS - Open Testing Service

Synopsis

OTS is a Distributed Test System that takes a Data Driven Approach using XML Files to allow parallel testing of Applications on evolving Hardware platforms.

Design has been driven by the need for a language neutral test platform that allows testing under different device architectures in parallel.

High Level Overview

OTS allows a Test Sequence to be fanned out from a hub known as a Distributor to a remote machine known as a Worker. Each Worker has a Device Group associated with it, this is defined in the configuration. The Tests are routed to the Worker on the basis of the Device Group.

An XML Test Definition File describes the Tests that are run. The results are returned to the Distributor in an XML Results File.

The Distributor takes a Device Group and Timeout as inputs, as well as the Command for running Tasks. Tests are run within a Task which is currently run as a process. The Command will typically take a path to the Test Definition File as an input. If the process exceeds the Timeout the Task is stopped (the process is killed).

A number of Tasks can be added to a Testrun. The Testrun will normally wait for all Tasks to finish before completion.

Data (status, error and results) are communicated back from the Task to the Worker. The Results Object is a container for the files outputted: Test Definition File, Results File and any other files produced by the test as well as metadata relating to the run.

Technology

OTS is written in Python 2.6.

AMQP protocol is used for communication.

XmlFileFormats

The XML file formats provide the datum for the system.

Download them and add the path to the results_xsd parameter in your ots_server.ini file

Third Party Dependencies

RabbitMQ

You need to install RabbitMQ, this is AMQP message server.

Managing the Queues

If you need to delete or empty the queues there are utilities to help with this.

Python Packages

The following python packages are needed

setuptools

configobj

multiprocessing

pyamqplib (version 0.6.1 or newer)

minixsv

django

nose

Diving In

The following steps show how to get a Development Environment up and running

1. Install Dependencies

Install the Third Party Dependencies

Get the XML File Formats and set the OTS_TESTDEFINITION environment variable

2. Build the Eggs

From the root directory


./setup.sh
source ./paths.sh

3. Unittests

Run the unittests from the root directory


./nose.sh

4. Component Tests

Component Tests run high level tests on an OTS Component using Mocks of the dependencies.

You can run these with the component_tests shell script in the root directory.


./component_tests.sh

5. Hello OTS World

The source contains a simple demonstration of the way the key elements work.

1. Open two terminal windows.

2. In the first window run the server


cd $SERVER
cd hub/demos
python demo_hub.py

3. You should see this:


<snip>
20XX-XX-XX XX:XX:XX,XXX - ots.server.distributor.taskrunner - DEBUG - Sending command '['echo', 'hello world']' with key 'foo'

4. In the second window run the worker


cd $WORKER
python worker.py -c config.ini

5. Now in the Server terminal the command should run to completion. You should see the following line in the logs:


 20XX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world

It will raise a PackageException. This is normal as no Tasks that are run return Packages to the Server

6. And on the logs on the Worker side should contain:


XXXX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - DEBUG - Task in state: 'finish'
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - INFO - Recommence consume on queue: foo

Installation

It is intended that the Distributor and each Worker should be installed to separate machines.

Configuration

Distributor

The format of the configuration file is as follows.

The `host` is the name of the RabbitMQ server


[Client]
host =  localhost
vhost = / 
port = 5672
username = guest
password = guest
queue = FIXME
consumer_tag = worker
services_exchange = services

Worker

Edit your config.ini so that the

* `queue`
* `routing_key`
* `services_exchange`

Are set for your Device Group.

And the `host` is the name of your RabbitMQ server.


[Worker]
host =  localhost
vhost = / 
port = 5672
username = guest
password = guest
queue = foo
routing_key = foo
services_exchange = foo

Starting the Worker

To start your Worker:


python worker.py -c config.ini

Automated System Tests

Automated system tests are available in ots.system_tests. They trigger various testruns with xmlrpc and check return value and log messages.

Requirements

Fully functional OTS system with one worker.
A SW Product "ots-system-tests" configured so that it defaults to the worker.
A meego image with test-definition-tests and testrunner-lite-regression-tests.
Django based advanced OTS setup.
BeautifulSoup python library.

Configuration

make the system test image available for the workers. (for example copy it to your apache static content directory /var/www/static/)
Modify system_tests.conf in ots.system_tests directory to match your system.
Make sure all the urls in system_tests.conf work.

Notes

System tests execute multiple full testruns. Running all tests can easily take more than 1.5 hours with N900 worker.
Make sure there's no other activities ongoing in the system while running the tests.
Some error handling tests expect certain devicegroups and sw products _not_ to be available. If these test cases fail you can remove those queues with ots tools.

License

OTS is distributed under an LGPL license.

Contributions

OTS welcomes contributions.

The following areas would benefit greatly from your help:

   * Adding to the list of freely available Publisher Plugins
   * Examples of OTS in use with different hardware architectures
   * Improving the ease of deployment