MeeGo wiki - User contributions [en]

Quality/QA-tools/OTS

2011-02-14T08:35:33Z

Tvainio: /* Introduction */

Quality/QA-tools/OTS/DeveloperDocs/Experimental

2011-02-07T13:37:58Z

Tvainio: /* How to monitor ots? */

== Experimental Area ==

Documentation for experimental branches, spikes, WIPs etc. You can also add documentation related to your merge requests or patches here.

The docs should be moved to proper place in OTS wiki after the feature gets integrated to master branch.

'''This area is relevant for OTS developers only! Information under this area does not apply to master or released OTS versions.'''

=== 0.1 ===

* <s>[[Quality/QA-tools/OTS/DeveloperDocs/Experimental/Django|Advanced OTS server using django]] (Relates to branch dev_branch_0.1_django_server) </s>
**Released in 0.1.1 version. Documentation moved to [[Quality/QA-tools/OTS/UserDocumentation/Installation#Django_Based_Server]]

=== 0.8.2 ===

=== How to monitor ots? ===

We have monitor DTO:s. Probably some kind of hierarchy of dto monitor objects.

* Testrun Events for all testrun states (flashing, bootup, testing, test result processing...):
** Test pkg execution event (start/stop) (minimum requirement for [FEA] 9036)
** Flashing Event (start/stop) (n:th try)
** Bootup event (n:th try)
** Testsuite/set/case/step event? (What level of detail we want to monitor?) Remember that logging still exists...
** Testrun overall result

* Worker events:
** Worker state (Start/stop/busy(testrun id)/free) (information about queues)
** Heartbeat?

* Failure event dto? (Flashing failed, device failure...)?

What about statistics monitoring? What do we need?

* Number of testruns?
* Number of workers?
* Active testruns?
* PASS/FAIL/ERROR ratios?
* Error statistics?
** Most common errors
** Some kind of grouping of errors and statistics for them (user fault vs catastrophic ots failure vs device/image error vs...)

Notes:

* One host can have multiple worker instances running. Dto:s need to have information about the instance, not just hostname.
*

==== Who receives monitor dto:s? ====

Current implementation uses the same testrun response queue and process for monitor dto:s. This is simple to implement for testrun related events but very limited for generic monitoring of the whole system. For example most of the worker events cannot be sent to testrun queue because they are not related to a specific testrun and we don't know if there even is any testrun response queues around.

Having a separate permanent queue and a separate listener process for monitoring would allow monitoring also outside a testrun.

Pros:

* Can be deployed as a completely separate component
** even into a different machine
** Maintenance, update etc. does not affect test execution in any way

Cons:

* One more process/daemon to maintain
** More complicated install
* Might cause errors.
** Not running => Lots of messages pile up in the queue
* Dependencies
** What if for example the distribution model relies on monitor component to provide up to date data?
* How to provide testrun specific monitor info to Publishers?

==== TODO ====

* Current server side implementation does not send monitor dtos to Publishers in real time. They are sent only after the testrun is done.

=== Bug 9036 - [FEA] Test package distribution based on history (last execution time of a package) ===

Proposal:

* Data provided by monitor dto:s
* Data stored to a DB
** A separate DB for this plugin only or dependency to a more common monitor DB?
* A custom distribution model that reads data from DB and creates tasks based on that

Notes:

* We cannot limit to testpackage level. Implementation for set level distribution is already ongoing.

Quality/QA-tools/OTS/DeveloperDocs/Experimental

2011-02-07T13:32:54Z

Tvainio: /* How to monitor ots? */

== Experimental Area ==

Documentation for experimental branches, spikes, WIPs etc. You can also add documentation related to your merge requests or patches here.

The docs should be moved to proper place in OTS wiki after the feature gets integrated to master branch.

'''This area is relevant for OTS developers only! Information under this area does not apply to master or released OTS versions.'''

=== 0.1 ===

* <s>[[Quality/QA-tools/OTS/DeveloperDocs/Experimental/Django|Advanced OTS server using django]] (Relates to branch dev_branch_0.1_django_server) </s>
**Released in 0.1.1 version. Documentation moved to [[Quality/QA-tools/OTS/UserDocumentation/Installation#Django_Based_Server]]

=== 0.8.2 ===

=== How to monitor ots? ===

We have monitor DTO:s. Probably some kind of hierarchy of dto monitor objects.

* Testrun Events for all testrun states (flashing, bootup, testing, test result processing...):
** Test pkg execution event (start/stop) (minimum requirement for [FEA] 9036)
** Flashing Event (start/stop) (n:th try)
** Bootup event (n:th try)
** Testsuite/set/case/step event? (What level of detail we want to monitor?) Remember that logging still exists...
** Testrun overall result

* Worker events:
** Worker state (Start/stop/busy(testrun id)/free) (What about queue info?)
** Heartbeat?

* Failure event dto? (Flashing failed, device failure...)?

What about statistics monitoring? What do we need?

* Number of testruns?
* Number of workers?
* Active testruns?
* PASS/FAIL/ERROR ratios?
* Error statistics?
** Most common errors
** Some kind of grouping of errors and statistics for them (user fault vs catastrophic ots failure vs device/image error vs...)

Notes:

* One host can have multiple worker instances running. Dto:s need to have information about the instance, not just hostname.
*

==== Who receives monitor dto:s? ====

Current implementation uses the same testrun response queue and process for monitor dto:s. This is simple to implement for testrun related events but very limited for generic monitoring of the whole system. For example most of the worker events cannot be sent to testrun queue because they are not related to a specific testrun and we don't know if there even is any testrun response queues around.

Having a separate permanent queue and a separate listener process for monitoring would allow monitoring also outside a testrun.

Pros:

* Can be deployed as a completely separate component
** even into a different machine
** Maintenance, update etc. does not affect test execution in any way

Cons:

* One more process/daemon to maintain
** More complicated install
* Might cause errors.
** Not running => Lots of messages pile up in the queue
* Dependencies
** What if for example the distribution model relies on monitor component to provide up to date data?
* How to provide testrun specific monitor info to Publishers?

==== TODO ====

* Current server side implementation does not send monitor dtos to Publishers in real time. They are sent only after the testrun is done.

=== Bug 9036 - [FEA] Test package distribution based on history (last execution time of a package) ===

Proposal:

* Data provided by monitor dto:s
* Data stored to a DB
** A separate DB for this plugin only or dependency to a more common monitor DB?
* A custom distribution model that reads data from DB and creates tasks based on that

Notes:

* We cannot limit to testpackage level. Implementation for set level distribution is already ongoing.

Quality/QA-tools/OTS/DeveloperDocs/Experimental

2011-02-07T13:30:30Z

Tvainio: /* How to monitor ots? */

== Experimental Area ==

Documentation for experimental branches, spikes, WIPs etc. You can also add documentation related to your merge requests or patches here.

The docs should be moved to proper place in OTS wiki after the feature gets integrated to master branch.

'''This area is relevant for OTS developers only! Information under this area does not apply to master or released OTS versions.'''

=== 0.1 ===

* <s>[[Quality/QA-tools/OTS/DeveloperDocs/Experimental/Django|Advanced OTS server using django]] (Relates to branch dev_branch_0.1_django_server) </s>
**Released in 0.1.1 version. Documentation moved to [[Quality/QA-tools/OTS/UserDocumentation/Installation#Django_Based_Server]]

=== 0.8.2 ===

=== How to monitor ots? ===

We have monitor DTO:s. Probably some kind of hierarchy of dto monitor objects.

* Testrun Events:
** Test pkg execution event (start/stop) (minimum requirement for [FEA] 9036)
** Flashing Event (start/stop) (n:th try)
** Bootup event (n:th try)
** Testsuite/set/case/step event? (What level of detail we want to monitor?) Remember that logging still exists...
** Testrun overall result

* Worker events:
** Worker state (Start/stop/busy(testrun id)/free) (What about queue info?)
** Heartbeat?

* Failure event dto? (Flashing failed, device failure...)?

What about statistics monitoring? What do we need?

* Number of testruns?
* Number of workers?
* Active testruns?
* PASS/FAIL/ERROR ratios?
* Error statistics?
** Most common errors
** Some kind of grouping of errors and statistics for them (user fault vs catastrophic ots failure vs device/image error vs...)

Notes:

* One host can have multiple worker instances running. Dto:s need to have information about the instance, not just hostname.
*

==== Who receives monitor dto:s? ====

Current implementation uses the same testrun response queue and process for monitor dto:s. This is simple to implement for testrun related events but very limited for generic monitoring of the whole system. For example most of the worker events cannot be sent to testrun queue because they are not related to a specific testrun and we don't know if there even is any testrun response queues around.

Having a separate permanent queue and a separate listener process for monitoring would allow monitoring also outside a testrun.

Pros:

* Can be deployed as a completely separate component
** even into a different machine
** Maintenance, update etc. does not affect test execution in any way

Cons:

* One more process/daemon to maintain
** More complicated install
* Might cause errors.
** Not running => Lots of messages pile up in the queue
* Dependencies
** What if for example the distribution model relies on monitor component to provide up to date data?
* How to provide testrun specific monitor info to Publishers?

==== TODO ====

* Current server side implementation does not send monitor dtos to Publishers in real time. They are sent only after the testrun is done.

=== Bug 9036 - [FEA] Test package distribution based on history (last execution time of a package) ===

Proposal:

* Data provided by monitor dto:s
* Data stored to a DB
** A separate DB for this plugin only or dependency to a more common monitor DB?
* A custom distribution model that reads data from DB and creates tasks based on that

Notes:

* We cannot limit to testpackage level. Implementation for set level distribution is already ongoing.

Quality/QA-tools/OTS/DeveloperDocs/Experimental

2011-02-07T13:18:26Z

Tvainio: /* How to monitor ots? */

== Experimental Area ==

Documentation for experimental branches, spikes, WIPs etc. You can also add documentation related to your merge requests or patches here.

The docs should be moved to proper place in OTS wiki after the feature gets integrated to master branch.

'''This area is relevant for OTS developers only! Information under this area does not apply to master or released OTS versions.'''

=== 0.1 ===

* <s>[[Quality/QA-tools/OTS/DeveloperDocs/Experimental/Django|Advanced OTS server using django]] (Relates to branch dev_branch_0.1_django_server) </s>
**Released in 0.1.1 version. Documentation moved to [[Quality/QA-tools/OTS/UserDocumentation/Installation#Django_Based_Server]]

=== 0.8.2 ===

=== How to monitor ots? ===

We have monitor DTO:s. Probably some kind of hierarchy of dto monitor objects.

* Testrun Events:
** Test pkg execution event (start/stop) (minimum requirement for [FEA] 9036)
** Flashing Event (start/stop) (n:th try)
** Bootup event (n:th try)
** Testsuite/set/case/step event? (What level of detail we want to monitor?) Remember that logging still exists...
** Testrun overall result

* Worker events:
** Worker state (Start/stop/busy/free) (What about queue info?)
** Heartbeat?

* Failure event dto? (Flashing failed, device failure...)?

What about statistics monitoring? What do we need?

* Number of testruns?
* Number of workers?
* Active testruns?
* PASS/FAIL/ERROR ratios?
* Error statistics?
** Most common errors
** Some kind of grouping of errors and statistics for them (user fault vs catastrophic ots failure vs device/image error vs...)

Notes:

* One host can have multiple worker instances running. Dto:s need to have information about the instance, not just hostname.
*

==== Who receives monitor dto:s? ====

Current implementation uses the same testrun response queue and process for monitor dto:s. This is simple to implement for testrun related events but very limited for generic monitoring of the whole system. For example most of the worker events cannot be sent to testrun queue because they are not related to a specific testrun and we don't know if there even is any testrun response queues around.

Having a separate permanent queue and a separate listener process for monitoring would allow monitoring also outside a testrun.

Pros:

* Can be deployed as a completely separate component
** even into a different machine
** Maintenance, update etc. does not affect test execution in any way

Cons:

* One more process/daemon to maintain
** More complicated install
* Might cause errors.
** Not running => Lots of messages pile up in the queue
* Dependencies
** What if for example the distribution model relies on monitor component to provide up to date data?
* How to provide testrun specific monitor info to Publishers?

==== TODO ====

* Current server side implementation does not send monitor dtos to Publishers in real time. They are sent only after the testrun is done.

=== Bug 9036 - [FEA] Test package distribution based on history (last execution time of a package) ===

Proposal:

* Data provided by monitor dto:s
* Data stored to a DB
** A separate DB for this plugin only or dependency to a more common monitor DB?
* A custom distribution model that reads data from DB and creates tasks based on that

Notes:

* We cannot limit to testpackage level. Implementation for set level distribution is already ongoing.

Quality/QA-tools/OTS/DeveloperDocs/Experimental

2011-02-07T12:37:49Z

Tvainio: /* Bug 9036 - [FEA] Test package distribution based on history (last execution time of a package) */

== Experimental Area ==

Documentation for experimental branches, spikes, WIPs etc. You can also add documentation related to your merge requests or patches here.

The docs should be moved to proper place in OTS wiki after the feature gets integrated to master branch.

'''This area is relevant for OTS developers only! Information under this area does not apply to master or released OTS versions.'''

=== 0.1 ===

* <s>[[Quality/QA-tools/OTS/DeveloperDocs/Experimental/Django|Advanced OTS server using django]] (Relates to branch dev_branch_0.1_django_server) </s>
**Released in 0.1.1 version. Documentation moved to [[Quality/QA-tools/OTS/UserDocumentation/Installation#Django_Based_Server]]

=== 0.8.2 ===

=== How to monitor ots? ===

We have monitor DTO:s. Probably some kind of hierarchy of dto monitor objects.

* Testrun Events:
** Test pkg execution event (start/stop) (minimum requirement for [FEA] 9036)
** Flashing Event (start/stop)
** Testsuite/set/case/step event? (What level of detail we want to monitor?) Remember that logging still exists...
** Testrun overall result

* Worker events:
** Worker state (Start/stop/busy/free) (What about queue info?)
** Heartbeat?

* Failure event dto? (Flashing failed, device failure...)?

What about statistics monitoring? What do we need?

* Number of testruns?
* Number of workers?
* Active testruns?
* PASS/FAIL/ERROR ratios?
* Error statistics?
** Most common errors
** Some kind of grouping of errors and statistics for them (user fault vs catastrophic ots failure vs device/image error vs...)

==== Who receives monitor dto:s? ====

Current implementation uses the same testrun response queue and process for monitor dto:s. This is simple to implement for testrun related events but very limited for generic monitoring of the whole system. For example most of the worker events cannot be sent to testrun queue because they are not related to a specific testrun and we don't know if there even is any testrun response queues around.

Having a separate permanent queue and a separate listener process for monitoring would allow monitoring also outside a testrun.

Pros:

* Can be deployed as a completely separate component
** even into a different machine
** Maintenance, update etc. does not affect test execution in any way

Cons:

* One more process/daemon to maintain
** More complicated install
* Might cause errors.
** Not running => Lots of messages pile up in the queue
* Dependencies
** What if for example the distribution model relies on monitor component to provide up to date data?
* How to provide testrun specific monitor info to Publishers?

==== TODO ====

* Current server side implementation does not send monitor dtos to Publishers in real time. They are sent only after the testrun is done.

=== Bug 9036 - [FEA] Test package distribution based on history (last execution time of a package) ===

Proposal:

* Data provided by monitor dto:s
* Data stored to a DB
** A separate DB for this plugin only or dependency to a more common monitor DB?
* A custom distribution model that reads data from DB and creates tasks based on that

Notes:

* We cannot limit to testpackage level. Implementation for set level distribution is already ongoing.

Quality/QA-tools/OTS/DeveloperDocs/Experimental

2011-02-07T12:31:57Z

Tvainio: /* Experimental Area */

== Experimental Area ==

Documentation for experimental branches, spikes, WIPs etc. You can also add documentation related to your merge requests or patches here.

The docs should be moved to proper place in OTS wiki after the feature gets integrated to master branch.

'''This area is relevant for OTS developers only! Information under this area does not apply to master or released OTS versions.'''

=== 0.1 ===

* <s>[[Quality/QA-tools/OTS/DeveloperDocs/Experimental/Django|Advanced OTS server using django]] (Relates to branch dev_branch_0.1_django_server) </s>
**Released in 0.1.1 version. Documentation moved to [[Quality/QA-tools/OTS/UserDocumentation/Installation#Django_Based_Server]]

=== 0.8.2 ===

=== How to monitor ots? ===

We have monitor DTO:s. Probably some kind of hierarchy of dto monitor objects.

* Testrun Events:
** Test pkg execution event (start/stop) (minimum requirement for [FEA] 9036)
** Flashing Event (start/stop)
** Testsuite/set/case/step event? (What level of detail we want to monitor?) Remember that logging still exists...
** Testrun overall result

* Worker events:
** Worker state (Start/stop/busy/free) (What about queue info?)
** Heartbeat?

* Failure event dto? (Flashing failed, device failure...)?

What about statistics monitoring? What do we need?

* Number of testruns?
* Number of workers?
* Active testruns?
* PASS/FAIL/ERROR ratios?
* Error statistics?
** Most common errors
** Some kind of grouping of errors and statistics for them (user fault vs catastrophic ots failure vs device/image error vs...)

==== Who receives monitor dto:s? ====

Current implementation uses the same testrun response queue and process for monitor dto:s. This is simple to implement for testrun related events but very limited for generic monitoring of the whole system. For example most of the worker events cannot be sent to testrun queue because they are not related to a specific testrun and we don't know if there even is any testrun response queues around.

Having a separate permanent queue and a separate listener process for monitoring would allow monitoring also outside a testrun.

Pros:

* Can be deployed as a completely separate component
** even into a different machine
** Maintenance, update etc. does not affect test execution in any way

Cons:

* One more process/daemon to maintain
** More complicated install
* Might cause errors.
** Not running => Lots of messages pile up in the queue
* Dependencies
** What if for example the distribution model relies on monitor component to provide up to date data?
* How to provide testrun specific monitor info to Publishers?

==== TODO ====

* Current server side implementation does not send monitor dtos to Publishers in real time. They are sent only after the testrun is done.

=== Bug 9036 - [FEA] Test package distribution based on history (last execution time of a package) ===

Steps:

* Data provided by monitor dto:s
* Data stored to a DB
* A custom distribution model that reads data from DB and creates tasks based on that

Notes:

* We cannot limit to testpackage level. Implementation for set level distribution is already ongoing.

Quality/QA-tools/OTS

2011-02-04T13:25:14Z

Tvainio: /* Releases */

Quality/QA-tools/OTS/DeveloperDocs/CreatingACustomDistributionModel

2011-02-04T10:47:53Z

Tvainio: /* How to Customize Test Package Distribution in OTS */

== How to Customize Test Package Distribution in OTS ==

OTS supports 2 distribution models. By default all packages are executed in one task. In practice this means that one worker will take care of the whole testrun by flashing the device and running all test packages one after another. If "perpackage" distribution model is used, a separate task will be created for each package. If enough workers are free, all packages will be tested on a separate worker in parallel. This is in theory the fastest possible way to execute the packages.

In reality the device preparation takes a lot of time and there usually is not enough workers free so some kind of optimized distribution model between these two extremes might make the system more useful. To make this possible starting from version 0.8.1 OTS supports custom distribution models.

More background information about the distribution optimization can be found in [[Quality/QA-tools/OTS/OptimisingThroughput|Optimising throughput.]]

=== Creating a Custom Distribution Model ===

Distribution models use setuptools entry point mechanism just like PublisherPlugins do. An example can be found in [[http://meego.gitorious.org/meego-quality-assurance/ots/trees/master/examples/ots.plugin.example_distribution_model examples/ots/plugin/example_distribution_model/]]

==== Defining the Distribution Model in setup.py ====

An entry point named "ots_distribution_model" needs to be defined in the setup.py to make OTS find the custom distribution model:
<pre>
entry_points={"ots_distribution_model":
["example_model = ots.plugin.example_distribution_model.example_model:get_model"]
},
</pre>

The name "example_model" is the name of the custom model. It maps directly to the OTS testrun parameter distribution_model and ots_trigger parameter <code>-c DISTMODEL, --distribution=DISTMODEL</code>

==== Implementing the Distribution Model ====

The actual distribution model is loaded with a factory method that gets all testrun options as parameters. It should return a callable that takes test package list as input and returns conductor commands as a list.

A plugin example is available in [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/master/examples/ots.plugin.example_distribution_model/ots/plugin/example_distribution_model/example_model.py examples/ots/plugin/example_distribution_model/example_model.py]

Fully implemented distribution models can be found in [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/master/ots.server/ots/server/allocator/default_distribution_models.py ots/server/allocator/default_distribution_models.py]

Quality/QA-tools/OTS/DeveloperDocs/CreatingACustomDistributionModel

2011-02-04T10:45:03Z

Tvainio: /* Creating a Custom Distribution Model */

== How to Customize Test Package Distribution in OTS ==

OTS supports 2 distribution models. By default all packages are executed in one task. In practice this means that one worker will take care of the whole testrun by flashing the device and running all test packages one after another. If "perpackage" distribution model is used, a separate task will be created for each package. If enough workers are free, all packages will be tested on a separate worker in parallel. This is in theory the fastest possible way to execute the packages.

In reality the device preparation takes a lot of time and there usually is not enough workers free so some kind of optimized distribution model between these two extremes might make the system more useful. To make this possible OTS supports custom distribution models.

More background information about the distribution optimization can be found in [[Quality/QA-tools/OTS/OptimisingThroughput|Optimising throughput.]]

=== Creating a Custom Distribution Model ===

Distribution models use setuptools entry point mechanism just like PublisherPlugins do. An example can be found in [[http://meego.gitorious.org/meego-quality-assurance/ots/trees/master/examples/ots.plugin.example_distribution_model examples/ots/plugin/example_distribution_model/]]

==== Defining the Distribution Model in setup.py ====

An entry point named "ots_distribution_model" needs to be defined in the setup.py to make OTS find the custom distribution model:
<pre>
entry_points={"ots_distribution_model":
["example_model = ots.plugin.example_distribution_model.example_model:get_model"]
},
</pre>

The name "example_model" is the name of the custom model. It maps directly to the OTS testrun parameter distribution_model and ots_trigger parameter <code>-c DISTMODEL, --distribution=DISTMODEL</code>

==== Implementing the Distribution Model ====

The actual distribution model is loaded with a factory method that gets all testrun options as parameters. It should return a callable that takes test package list as input and returns conductor commands as a list.

A plugin example is available in [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/master/examples/ots.plugin.example_distribution_model/ots/plugin/example_distribution_model/example_model.py examples/ots/plugin/example_distribution_model/example_model.py]

Fully implemented distribution models can be found in [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/master/ots.server/ots/server/allocator/default_distribution_models.py ots/server/allocator/default_distribution_models.py]

Quality/QA-tools/OTS/DeveloperDocs/CreatingACustomDistributionModel

2011-02-04T10:44:08Z

Tvainio: Created page with "== How to Customize Test Package Distribution in OTS == OTS supports 2 distribution models. By default all packages are executed in one task. In practice this means that one wor…"

== How to Customize Test Package Distribution in OTS ==

OTS supports 2 distribution models. By default all packages are executed in one task. In practice this means that one worker will take care of the whole testrun by flashing the device and running all test packages one after another. If "perpackage" distribution model is used, a separate task will be created for each package. If enough workers are free, all packages will be tested on a separate worker in parallel. This is in theory the fastest possible way to execute the packages.

In reality the device preparation takes a lot of time and there usually is not enough workers free so some kind of optimized distribution model between these two extremes might make the system more useful. To make this possible OTS supports custom distribution models.

More background information about the distribution optimization can be found in [[Quality/QA-tools/OTS/OptimisingThroughput|Optimising throughput.]]

=== Creating a Custom Distribution Model ===

Distribution models use setuptools entry point mechanism just like PublisherPlugins do. An example can be found in [[http://meego.gitorious.org/meego-quality-assurance/ots/trees/master/examples/ots.plugin.example_distribution_model ots.plugin.example_distribution_model]]

==== Defining the Distribution Model in setup.py ====

An entry point named "ots_distribution_model" needs to be defined in the setup.py to make OTS find the custom distribution model:
<pre>
entry_points={"ots_distribution_model":
["example_model = ots.plugin.example_distribution_model.example_model:get_model"]
},
</pre>

The name "example_model" is the name of the custom model. It maps directly to the OTS testrun parameter distribution_model and ots_trigger parameter <code>-c DISTMODEL, --distribution=DISTMODEL</code>

==== Implementing the Distribution Model ====

The actual distribution model is loaded with a factory method that gets all testrun options as parameters. It should return a callable that takes test package list as input and returns conductor commands as a list.

A plugin example is available in [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/master/examples/ots.plugin.example_distribution_model/ots/plugin/example_distribution_model/example_model.py examples/ots/plugin/example_distribution_model/example_model.py]

Fully implemented distribution models can be found in [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/master/ots.server/ots/server/allocator/default_distribution_models.py ots/server/allocator/default_distribution_models.py]

Quality/QA-tools/OTS/DeveloperDocs

2011-02-04T10:21:00Z

Tvainio: /* Extending OTS */

= OTS - Open Testing Service =

== Related Documentation ==

* [[Quality/QA-tools/OTS/DevelopmentGuidelines | Development Guidelines]]
* [[Quality/QA-tools/OTS/Contributions | Contributions]]
* [[Quality/QA-tools/OTS/Architecture]]
* [[Quality/QA-tools/OTS/Packages | Packages]]
* [[Quality/QA-tools/OTS/ErrorCodes | Error Codes]]
* [[Quality/QA-tools/OTS/Glossary]]
* [[Quality/QA-tools/OTS/DeveloperDocs/Transition0.1-0.8|Making transition from 0.1 to 0.8]]

== Synopsis ==

OTS is a Distributed Test System that takes a Data Driven Approach using XML Files to allow parallel testing of Applications on evolving Hardware platforms.

Design has been driven by the need for a language neutral test platform that allows testing under different device architectures in parallel.

== High Level Overview ==

OTS allows a Test Sequence to be fanned out from a hub known as a '''Distributor''' to a remote machine known as a '''Worker'''.
Each '''Worker''' has a '''Device Group''' associated with it, this is defined in the configuration. The Tests are routed to the '''Worker'''
on the basis of the '''Device Group'''.

An XML '''Test Definition File''' describes the Tests that are run. The results are returned to the '''Distributor''' in an XML '''Results File'''.

The '''Distributor''' takes a '''Device Group''' and '''Timeout''' as inputs, as well as the '''Command''' for running '''Tasks'''. Tests are run within a '''Task''' which
is currently run as a process. The '''Command''' will typically take a path to the '''Test Definition File''' as an input.
If the process exceeds the '''Timeout''' the '''Task''' is stopped (the process is killed).

A number of '''Tasks''' can be added to a '''Testrun'''. The '''Testrun''' will normally wait for all '''Tasks''' to finish before completion.

Data (status, error and results) are communicated back from the '''Task''' to the '''Worker'''. The '''Results Object''' is a container for the files outputted: '''Test Definition File''', '''Results File'''
and any other files produced by the test as well as metadata relating to the run.

== Technology ==

OTS is written in [http://www.python.org/ Python 2.6].

[http://www.amqp.org/confluence/display/AMQP/Advanced+Message+Queuing+Protocol AMQP] protocol is used for communication.

=== XmlFileFormats ===

The XML file formats provide the datum for the system.

Download [http://meego.gitorious.org/meego-quality-assurance/test-definition/trees/master them] and add the path to the '''results_xsd''' parameter in your [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/dev_branch_0.8/ots.server/ots/server/ots_server.ini ots_server.ini] file

=== Third Party Dependencies ===

==== RabbitMQ ====

You need to install [http://www.rabbitmq.com/server.html RabbitMQ], this is AMQP message server.

===== Managing the Queues =====

If you need to delete or empty the queues there are [http://meego.gitorious.org/meego-quality-assurance/ots/trees/master/ots.server/ots/server/distributor/dev_utils utilities] to help with this.

==== Python Packages ====

The following python packages are needed

* [http://pypi.python.org/pypi/setuptools setuptools]

* [http://www.voidspace.org.uk/python/configobj.html configobj]

* [http://pypi.python.org/pypi/multiprocessing multiprocessing]

* [http://code.google.com/p/py-amqplib/ pyamqplib] (version 0.6.1 or newer)

* [http://www.familieleuthe.de/DownloadMiniXsv.html minixsv]

* [http://www.djangoproject.com/download/1.1.2/tarball/ django]

* [http://code.google.com/p/python-nose/ nose]

== Diving In ==

The following steps show how to get a Development Environment up and running

=== 1. Install Dependencies ===

* Install the Third Party Dependencies

* Get the XML File Formats and set the OTS_TESTDEFINITION environment variable

=== 2. Build the Eggs ===

From the root directory

<code>
./setup.sh
source ./paths.sh
</code>

=== 3. Unittests ===

Run the unittests from the root directory

<code>
./nose.sh
</code>

=== 4. Component Tests ===

Component Tests run high level tests on an OTS Component using Mocks of the dependencies.

You can run these with the component_tests shell script in the root directory.

<code>
./component_tests.sh
</code>

=== 5. Hello OTS World ===

The source contains a simple demonstration of the way the key elements work.

==== 1. Open two terminal windows. ====

==== 2. In the first window run the server ====

<code>
cd $SERVER
cd hub/demos
python demo_hub.py
</code>

==== 3. You should see this: ====

<code>
<snip>
20XX-XX-XX XX:XX:XX,XXX - ots.server.distributor.taskrunner - DEBUG - Sending command '['echo', 'hello world']' with key 'foo'
</code>

==== 4. In the second window run the worker ====

<code>
cd $WORKER
python worker.py -c config.ini
</code>

==== 5. Now in the Server terminal the command should run to completion. You should see the following line in the logs: ====

<code>
20XX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world
</code>

It will raise a PackageException. This is normal as no Tasks that are run return Packages to the Server

==== 6. And on the logs on the Worker side should contain: ====

<code>
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - DEBUG - Task in state: 'finish'
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - INFO - Recommence consume on queue: foo
</code>

== Installation ==

It is intended that the '''Distributor''' and each '''Worker''' should be installed to separate machines.

== Configuration ==

=== Distributor ===

The format of the configuration file is as follows.

The `host` is the name of the RabbitMQ server

<code>

[Client]
host = localhost
vhost = /
port = 5672
username = guest
password = guest
queue = FIXME
consumer_tag = worker
services_exchange = services

</code>

=== Worker ===

Edit your config.ini so that the

* `queue`
* `routing_key`
* `services_exchange`

Are set for your '''Device Group'''.

And the `host` is the name of your RabbitMQ server.

<code>
[Worker]
host = localhost
vhost = /
port = 5672
username = guest
password = guest
queue = foo
routing_key = foo
services_exchange = foo
</code>

==== Starting the Worker ====

To start your Worker:

<code>
python worker.py -c config.ini
</code>

== Automated System Tests ==

Automated system tests are available in ots.system_tests. They trigger various testruns with xmlrpc and check return value and log messages.

=== Requirements ===

* Fully functional OTS system with one worker.
* A SW Product "ots-system-tests" configured so that it defaults to the worker.
* A meego image with test-definition-tests and testrunner-lite-regression-tests.
* Django based advanced OTS setup.
* BeautifulSoup python library.

=== Configuration ===

* make the system test image available for the workers. (for example copy it to your apache static content directory /var/www/static/)
* Modify system_tests.conf in ots.system_tests directory to match your system.
* Make sure all the urls in system_tests.conf work.

=== Notes ===

* System tests execute multiple full testruns. Running all tests can easily take more than 1.5 hours with N900 worker.
* Make sure there's no other activities ongoing in the system while running the tests.
* Some error handling tests expect certain devicegroups and sw products _not_ to be available. If these test cases fail you can remove those queues with ots tools.

== License ==

OTS is distributed under an LGPL license.

== Contributions ==

* [[Quality/QA-tools/OTS/Contributions | Contributions]]
* [[Quality/QA-tools/OTS/DevelopmentGuidelines | Guidelines]]
* [[Quality/QA-tools/OTS/DevelopmentGuidelines/Review | Review process]]

== Wiki ==

[[Quality/QA-tools/OTS | OTS Wiki Home]]

== Source Code Documentation ==

FIXME

== Mailing List ==

[http://lists.meego.com/listinfo/meego-qa MeeGo QA] mailing list

== IRC Channel ==

The #meego-qa-tools irc channel in freenode

== Bugs ==

The [http://bugs.meego.com/ Meego bugs page]

== Platform Dependencies ==

OTS is tested to run on Linux only.

Tested on Ubuntu (8.04, 10.04, 10.10), Fedora 13, MeeGo and Redhat.

Binary support for Ubuntu, Fedora and MeeGo.

== Architecture ==

* [[Quality/QA-tools/OTS/DeveloperDocs/Components | Components]]

* [[Quality/QA-tools/OTS/DeveloperDocs/Server | Server]]

* [[Quality/QA-tools/OTS/DeveloperDocs/Plugins | Plugins]]

* [[Quality/QA-tools/OTS/DeveloperDocs/DataFlow | DataFlow]]

* [[Quality/QA-tools/OTS/DeveloperDocs/Behaviour | Behaviour]]

== APIs ==

* [[Quality/QA-tools/OTS/DeveloperDocs/OTS_Server_API | Server]]

* [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/dev_branch_0.8/ots.server/ots/server/distributor/api.py Distributor]

* [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/dev_branch_0.8/ots.results/ots/results/api.py Results]

* [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/dev_branch_0.8/ots.worker/ots/worker/api.py Worker]

== Extending OTS ==

You can extend OTS with your own [[Quality/QA-tools/OTS/CreatingAPublisher|Publisher]]
and with [[Quality/QA-tools/OTS/DeveloperDocs/CreatingACustomDistributionModel|custom package distribution models.]]

== Experimental ==

Documentation for experimental branches, spikes, WIPs etc. can be found in [[Quality/QA-tools/OTS/DeveloperDocs/Experimental]].

== Analysis ==

[[Quality/QA-tools/OTS/DeveloperDocs/ErrorSituations| OTS 0.1 Error Situations]]

[[Quality/QA-tools/OTS/DeveloperDocs/TypicalUsage| Typical OTS users and usage scenarios]]

== Project Planning ==

* [[Quality/QA-tools/OTS/Roadmap | Roadmap]]

* [[Quality/QA-tools/OTS/DevelopmentStatus | DevelopmentStatus]]

* [[Quality/QA-tools/OTS/Meetings | Meetings]]

* [[Quality/QA-tools/OTS/ReleaseChecklist | ReleaseChecklist]]

* [[Quality/QA-tools/OTS/ReleasePractices | ReleasePractices]]

Quality/QA-tools/OTS/UserDocumentation/Migration 0.1 0.8

2011-02-03T13:43:17Z

Tvainio: /* OTS Server */

= OTS migration from version 0.1.x to 0.8.x =

== OTS Server ==

* First backup all old configurations
/etc/ots-server.ini
/opt/ots.sqlite
/etc/ots_qareports_plugin.conf
Your extension configurations

* Remove 0.1.x version
sudo apt-get purge python-ots-common python-ots-server python-server python-ots-qareports-plugin python-ots-tools
or
sudo rm -fr /usr/local/lib/python2.6/dist-packages/ots*
or
sudo rm -fr /usr/lib/python2.6/site-packages/ots*

* Follow [[Quality/QA-tools/OTS/UserDocumentation/Installation| 0.8 installation instructions]] how to setup the server

=== Configuring 0.8 to match 0.1 ===

==== Email ====

* In 0.8 the email feature is implemented as [[Quality/QA-tools/OTS/Plugins/Email| OTS server's plug-in]]

* It uses the server's configuration file
/etc/ots_server.ini

* In 0.1 these settings were defined in your extension config. Copy values from there.

* If you want to use your 0.1 message_body, remove line break marks '\n' from the string and put all text between the """ -marks.

==== Default SW configuration and SW products ====

* In 0.8 default SW configurations and SW products are defined in the server's configuration.
/etc/ots_server.ini

* In 0.1 these settings were defined in your extension config.

* Add one or several SW products under the 'swproducts' section.

<pre>
[swproducts]
[[meego_n900]]
timeout = 60
distribution_model = default
email = on
email-attachments = off
emmc = ""
input_plugin = "default"
[[[device]]]
devicegroup = n900

[[meego_aava]]
timeout = 120
distribution_model = default
email = on
email-attachments = on
emmc = ""
input_plugin = "default"
[[[device]]]
devicegroup = aava
</pre>

==== Django Based server ====

* Follow the [[Quality/QA-tools/OTS/Plugins/HTTP_logger| logger plug-in]] instructions how to setup new Django server.

* Old database file /opt/ots.sqlite, is '''not''' compatible with 0.8 version.

* '''No''' custom ots_config or extension points used anymore. If want to use testrun_db, create new plug-in to OTS server.

* HTTP logging is automatically on if plug-in is installed.

==== QA-Reports ====

* No changes in 0.8, just install [[Quality/QA-tools/OTS/Plugins/QAReports| python-ots-plugin-qareports]] to your OTS server.

* Configuration file is the same
/etc/ots_qareports_plugin.conf

==== Other configurations ====

* OTS 0.8 uses mainly the server's configuration. Update values from the 0.1 configuration (/etc/ots-server.ini) to 0.8 (/etc/ots_server.ini).

* '''No''' extension points used in 0.8

==== Input Plugins ====

OTS 0.1 supports feeding information back to ots client(for example the build machine) with input plugins. The key features of of OTS 0.1 input plugins are:

* Publish testrun log url to the client in the beginning of the testrun
* Publish result urls to the client
* Publish result files to the client
* Runtime selection of input plugin with testrun options

OTS 0.8 does not support input plugins but is able to do the same things with Publisher plugins. The needed steps are described below. The publisher plugin replacing an input plugin is referred as ClientPublisher.

* Publish URLs to clients:
# All publishers can share their report urls by implementing the get_this_publisher_uris() method. For example ots.plugins.logger shares the the testrun log url by doing this. All reporting tool plugins should do the same.
# ClientPublisher needs to implement set_all_publisher_uris(self, uris_dict) method. The uris_dict contains all URLs from all publishers so it's easy to send them to the client in this method. The plugin can also filter the URLs if it does not want to send all of them to the client.
# The URL processing is done right after all publishers are loaded so log url gets sent in the beginning of the testrun as expected.

* Result file Storage
# ClientPublisher gets all result files by implementing set_results(self, results). It's easy to send them to client in that method..

* Dynamic runtime selection of input plugins
# All plugin constructors get the additional testrun options as keyword arguments. Disabling and enabling a specific ClientPublisher can be done in the constructor by checking the input parameters.

* get_changed_packages(self, build_id)
# This should be implemented in the particular reporting tool that requires this information. OTS does not use this information in test execution so there's no need to handle this in OTS.

== OTS Worker ==

* First backup all old configurations
/etc/conductor.conf
/etc/conductor/
/etc/ots.ini

* Remove 0.1.x version
sudo apt-get purge python-ots-common python-ots-worker
or
sudo rm -fr /usr/local/lib/python2.6/dist-packages/ots*
or
sudo rm -fr /usr/lib/python2.6/site-packages/ots*

* Follow [[Quality/QA-tools/OTS/UserDocumentation/Installation| 0.8 installation instructions]] how to setup the worker

* No changes in the configuration files, just copy old ots.ini and conductor.conf to /etc.

* No changes to customflasher.

Quality/QA-tools/OTS/DevelopmentStatus

2011-02-02T10:37:06Z

Tvainio:

''' This page is no longer maintained. Development is managed in bugs.meego.org '''

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| Done
|-
| Exceptions
| An understandable exception handling model. Documented
| OTS 0.9
|-
| Logging
| A coherent distributed logging system
| Done
|-
| Upgrade Capability
| Worker version management
| OTS 0.9
|-
| Function call
| Conductor should be function call not a process
| OTS 0.9
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| Done
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Done
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Done
|-
| Hub
| High level procedural code
| Done
|-
| Disambiguation Datamodel
| Clear persistence and data management
| Done
|-
| Distributor
| Opaque Data Transfer
| Done
|-
| Default Options
| Mapping of Default Options to SW product
| Done
|-
| XMLRPC
| Add XMLRPC interface
| Done
|-
| Allocator
| API for custom Task allocation
| OTS 0.8.1
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Done
|-
| Input Plugin
| Incorporate and mark as deprecated
| OTS 0.8.1
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| OTS 0.8.1
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| Done
|-
| Email Plugin
| Email Plugin for Publisher API
| Done
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| Done

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

== Other Problems ==

* '''<s>A testrun with invalid sw_product executes a testrun with sandbox-values. An error should be raised and no testrun should be executed.</s>'''
* '''<s>Exceptions are not handled and they hit the xmlrpc. request_sync() should always return PASS/FAIL/ERROR. nothing else.</s>'''
* '''Incompatible worker version leaves worker in an unknown state and does not report the problem clearly.'''

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2011-02-02T10:35:20Z

Tvainio: /* Other Problems */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| Done
|-
| Exceptions
| An understandable exception handling model. Documented
| OTS 0.9
|-
| Logging
| A coherent distributed logging system
| Done
|-
| Upgrade Capability
| Worker version management
| OTS 0.9
|-
| Function call
| Conductor should be function call not a process
| OTS 0.9
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| Done
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Done
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Done
|-
| Hub
| High level procedural code
| Done
|-
| Disambiguation Datamodel
| Clear persistence and data management
| Done
|-
| Distributor
| Opaque Data Transfer
| Done
|-
| Default Options
| Mapping of Default Options to SW product
| Done
|-
| XMLRPC
| Add XMLRPC interface
| Done
|-
| Allocator
| API for custom Task allocation
| OTS 0.8.1
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Done
|-
| Input Plugin
| Incorporate and mark as deprecated
| OTS 0.8.1
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| OTS 0.8.1
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| Done
|-
| Email Plugin
| Email Plugin for Publisher API
| Done
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| Done

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

== Other Problems ==

* '''<s>A testrun with invalid sw_product executes a testrun with sandbox-values. An error should be raised and no testrun should be executed.</s>'''
* '''<s>Exceptions are not handled and they hit the xmlrpc. request_sync() should always return PASS/FAIL/ERROR. nothing else.</s>'''
* '''Incompatible worker version leaves worker in an unknown state and does not report the problem clearly.'''

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2011-02-02T10:34:50Z

Tvainio: /* Plugins */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| Done
|-
| Exceptions
| An understandable exception handling model. Documented
| OTS 0.9
|-
| Logging
| A coherent distributed logging system
| Done
|-
| Upgrade Capability
| Worker version management
| OTS 0.9
|-
| Function call
| Conductor should be function call not a process
| OTS 0.9
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| Done
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Done
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Done
|-
| Hub
| High level procedural code
| Done
|-
| Disambiguation Datamodel
| Clear persistence and data management
| Done
|-
| Distributor
| Opaque Data Transfer
| Done
|-
| Default Options
| Mapping of Default Options to SW product
| Done
|-
| XMLRPC
| Add XMLRPC interface
| Done
|-
| Allocator
| API for custom Task allocation
| OTS 0.8.1
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Done
|-
| Input Plugin
| Incorporate and mark as deprecated
| OTS 0.8.1
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| OTS 0.8.1
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| Done
|-
| Email Plugin
| Email Plugin for Publisher API
| Done
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| Done

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

== Other Problems ==

* '''<s>A testrun with invalid sw_product executes a testrun with sandbox-values. An error should be raised and no testrun should be executed.</s>'''
* '''Exceptions are not handled and they hit the xmlrpc. request_sync() should always return PASS/FAIL/ERROR. nothing else.'''
* '''Incompatible worker version leaves worker in an unknown state and does not report the problem clearly.'''

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2011-02-02T10:33:47Z

Tvainio: /* Server */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| Done
|-
| Exceptions
| An understandable exception handling model. Documented
| OTS 0.9
|-
| Logging
| A coherent distributed logging system
| Done
|-
| Upgrade Capability
| Worker version management
| OTS 0.9
|-
| Function call
| Conductor should be function call not a process
| OTS 0.9
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| Done
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Done
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Done
|-
| Hub
| High level procedural code
| Done
|-
| Disambiguation Datamodel
| Clear persistence and data management
| Done
|-
| Distributor
| Opaque Data Transfer
| Done
|-
| Default Options
| Mapping of Default Options to SW product
| Done
|-
| XMLRPC
| Add XMLRPC interface
| Done
|-
| Allocator
| API for custom Task allocation
| OTS 0.8.1
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Done
|-
| Input Plugin
| Incorporate and mark as deprecated
| OTS 0.8.1
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| OTS 0.8.1
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| done

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

== Other Problems ==

* '''<s>A testrun with invalid sw_product executes a testrun with sandbox-values. An error should be raised and no testrun should be executed.</s>'''
* '''Exceptions are not handled and they hit the xmlrpc. request_sync() should always return PASS/FAIL/ERROR. nothing else.'''
* '''Incompatible worker version leaves worker in an unknown state and does not report the problem clearly.'''

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2011-02-02T10:32:12Z

Tvainio: /* Common */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| Done
|-
| Exceptions
| An understandable exception handling model. Documented
| OTS 0.9
|-
| Logging
| A coherent distributed logging system
| Done
|-
| Upgrade Capability
| Worker version management
| OTS 0.9
|-
| Function call
| Conductor should be function call not a process
| OTS 0.9
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| Done
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Done
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| '''BROKEN. Default device options are not handled correctly. see options_factory unit tests'''
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started''' It should be possible to implement this as a publisher. Reporting tools can handle getting target package information.
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| In Progress
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| done

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

== Other Problems ==

* '''<s>A testrun with invalid sw_product executes a testrun with sandbox-values. An error should be raised and no testrun should be executed.</s>'''
* '''Exceptions are not handled and they hit the xmlrpc. request_sync() should always return PASS/FAIL/ERROR. nothing else.'''
* '''Incompatible worker version leaves worker in an unknown state and does not report the problem clearly.'''

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2011-02-02T10:31:53Z

Tvainio: /* Application Framework */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| Done
|-
| Exceptions
| An understandable exception handling model. Documented
| OTS 0.9
|-
| Logging
| A coherent distributed logging system
| Done
|-
| Upgrade Capability
| Worker version management
| OTS 0.9
|-
| Function call
| Conductor should be function call not a process
| OTS 0.9
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| Done
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Completed (Awaiting review)
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| '''BROKEN. Default device options are not handled correctly. see options_factory unit tests'''
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started''' It should be possible to implement this as a publisher. Reporting tools can handle getting target package information.
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| In Progress
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| done

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

== Other Problems ==

* '''<s>A testrun with invalid sw_product executes a testrun with sandbox-values. An error should be raised and no testrun should be executed.</s>'''
* '''Exceptions are not handled and they hit the xmlrpc. request_sync() should always return PASS/FAIL/ERROR. nothing else.'''
* '''Incompatible worker version leaves worker in an unknown state and does not report the problem clearly.'''

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2011-02-02T10:31:31Z

Tvainio: /* Application Framework */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| Done
|-
| Exceptions
| An understandable exception handling model. Documented
| OTS 0.9
|-
| Logging
| A coherent distributed logging system
| Done
|-
| Upgrade Capability
| Worker version management
| Awaiting review
|-
| Function call
| Conductor should be function call not a process
| OTS 0.9
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| Done
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Completed (Awaiting review)
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| '''BROKEN. Default device options are not handled correctly. see options_factory unit tests'''
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started''' It should be possible to implement this as a publisher. Reporting tools can handle getting target package information.
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| In Progress
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| done

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

== Other Problems ==

* '''<s>A testrun with invalid sw_product executes a testrun with sandbox-values. An error should be raised and no testrun should be executed.</s>'''
* '''Exceptions are not handled and they hit the xmlrpc. request_sync() should always return PASS/FAIL/ERROR. nothing else.'''
* '''Incompatible worker version leaves worker in an unknown state and does not report the problem clearly.'''

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DeveloperDocs

2011-01-28T13:49:39Z

Tvainio: /* Technology */

= OTS - Open Testing Service =

== Related Documentation ==

* [[Quality/QA-tools/OTS/Architecture]]
* [[Quality/QA-tools/OTS/Glossary]]
* [http://wiki.meego.com/Quality/QA-tools/OTS/DeveloperDocs/Transition0.1-0.8 Making transition from 0.1 to 0.8]

== Synopsis ==

OTS is a Distributed Test System that takes a Data Driven Approach using XML Files to allow parallel testing of Applications on evolving Hardware platforms.

Design has been driven by the need for a language neutral test platform that allows testing under different device architectures in parallel.

== High Level Overview ==

OTS allows a Test Sequence to be fanned out from a hub known as a '''Distributor''' to a remote machine known as a '''Worker'''.
Each '''Worker''' has a '''Device Group''' associated with it, this is defined in the configuration. The Tests are routed to the '''Worker'''
on the basis of the '''Device Group'''.

An XML '''Test Definition File''' describes the Tests that are run. The results are returned to the '''Distributor''' in an XML '''Results File'''.

The '''Distributor''' takes a '''Device Group''' and '''Timeout''' as inputs, as well as the '''Command''' for running '''Tasks'''. Tests are run within a '''Task''' which
is currently run as a process. The '''Command''' will typically take a path to the '''Test Definition File''' as an input.
If the process exceeds the '''Timeout''' the '''Task''' is stopped (the process is killed).

A number of '''Tasks''' can be added to a '''Testrun'''. The '''Testrun''' will normally wait for all '''Tasks''' to finish before completion.

Data (status, error and results) are communicated back from the '''Task''' to the '''Worker'''. The '''Results Object''' is a container for the files outputted: '''Test Definition File''', '''Results File'''
and any other files produced by the test as well as metadata relating to the run.

== Technology ==

OTS is written in Python 2.6.

OTS has the following FIXME (provided as separate PDF)

=== XmlFileFormats ===

The XML file formats provide the datum for the system.

Download [http://meego.gitorious.org/meego-quality-assurance/test-definition/trees/master them] and add the path to the '''results_xsd''' parameter in your [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/dev_branch_0.8/ots.server/ots/server/ots_server.ini ots_server.ini] file

=== Third Party Dependencies ===

==== RabbitMQ ====

You need to install [http://www.rabbitmq.com/server.html RabbitMQ].

===== Managing the Queues =====

If you need to delete or empty the queues there are [http://meego.gitorious.org/meego-quality-assurance/ots/trees/master/ots.server/ots/server/distributor/dev_utils utilities] to help with this.

==== Python Packages ====

The following python packages are needed

* [http://pypi.python.org/pypi/setuptools setuptools]

* [http://www.voidspace.org.uk/python/configobj.html configobj]

* [http://pypi.python.org/pypi/multiprocessing multiprocessing]

* [http://code.google.com/p/py-amqplib/ pyamqplib] (version 0.6.1 or newer)

* [http://www.familieleuthe.de/DownloadMiniXsv.html minixsv]

* [http://www.djangoproject.com/download/1.1.2/tarball/ django]

* [http://code.google.com/p/python-nose/ nose]

== Diving In ==

The following steps show how to get a Development Environment up and running

=== 1. Install Dependencies ===

* Install the Third Party Dependencies

* Get the XML File Formats and set the OTS_TESTDEFINITION environment variable

=== 2. Build the Eggs ===

From the root directory

<code>
./setup.sh
source ./paths.sh
</code>

=== 3. Unittests ===

Run the unittests from the root directory

<code>
./nose.sh
</code>

=== 4. Component Tests ===

Component Tests run high level tests on an OTS Component using Mocks of the dependencies.

You can run these with the component_tests shell script in the root directory.

<code>
./component_tests.sh
</code>

=== 5. Hello OTS World ===

The source contains a simple demonstration of the way the key elements work.

==== 1. Open two terminal windows. ====

==== 2. In the first window run the server ====

<code>
cd $SERVER
cd hub/demos
python demo_hub.py
</code>

==== 3. You should see this: ====

<code>
<snip>
20XX-XX-XX XX:XX:XX,XXX - ots.server.distributor.taskrunner - DEBUG - Sending command '['echo', 'hello world']' with key 'foo'
</code>

==== 4. In the second window run the worker ====

<code>
cd $WORKER
python worker.py -c config.ini
</code>

==== 5. Now in the Server terminal the command should run to completion. You should see the following line in the logs: ====

<code>
20XX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world
</code>

It will raise a PackageException. This is normal as no Tasks that are run return Packages to the Server

==== 6. And on the logs on the Worker side should contain: ====

<code>
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - DEBUG - Task in state: 'finish'
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - INFO - Recommence consume on queue: foo
</code>

== Installation ==

It is intended that the '''Distributor''' and each '''Worker''' should be installed to separate machines.

== Configuration ==

=== Distributor ===

The format of the configuration file is as follows.

The `host` is the name of the RabbitMQ server

<code>

[Client]
host = localhost
vhost = /
port = 5672
username = guest
password = guest
queue = FIXME
consumer_tag = worker
services_exchange = services

</code>

=== Worker ===

Edit your config.ini so that the

* `queue`
* `routing_key`
* `services_exchange`

Are set for your '''Device Group'''.

And the `host` is the name of your RabbitMQ server.

<code>
[Worker]
host = localhost
vhost = /
port = 5672
username = guest
password = guest
queue = foo
routing_key = foo
services_exchange = foo
</code>

==== Starting the Worker ====

To start your Worker:

<code>
python worker.py -c config.ini
</code>

== Automated System Tests ==

Automated system tests are available in ots.system_tests. They trigger various testruns with xmlrpc and check return value and log messages.

=== Requirements ===

* Fully functional OTS system with one worker.
* A SW Product "ots-system-tests" configured so that it defaults to the worker.
* A meego image with test-definition-tests and testrunner-lite-regression-tests.
* Django based advanced OTS setup.
* BeautifulSoup python library.

=== Configuration ===

* make the system test image available for the workers. (for example copy it to your apache static content directory /var/www/static/)
* Modify system_tests.conf in ots.system_tests directory to match your system.
* Make sure all the urls in system_tests.conf work.

=== Notes ===

* System tests execute multiple full testruns. Running all tests can easily take more than 1.5 hours with N900 worker.
* Make sure there's no other activities ongoing in the system while running the tests.
* Some error handling tests expect certain devicegroups and sw products _not_ to be available. If these test cases fail you can remove those queues with ots tools.

== License ==

OTS is distributed under an LGPL license.

== Contributions ==

From [http://meego.gitorious.org/meego-quality-assurance/ qa-tools]

== Wiki ==

[[Quality/QA-tools/OTS | OTS Wiki Home]]

== Source Code Documentation ==

FIXME

== Mailing List ==

FIXME

== IRC Channel ==

The #meego-qa-tools irc channel in freenode

== Bugs ==

The [http://bugs.meego.com/ Meego bugs page]

== Platform Dependencies ==

OTS is tested to run on Linux only. Productization environment is Ubuntu 8.04 LTS (Long Term Support).

== Architecture ==

* [[Quality/QA-tools/OTS/DeveloperDocs/Components | Components]]

* [[Quality/QA-tools/OTS/DeveloperDocs/Server | Server]]

* [[Quality/QA-tools/OTS/DeveloperDocs/Plugins | Plugins]]

* [[Quality/QA-tools/OTS/DeveloperDocs/DataFlow | DataFlow]]

* [[Quality/QA-tools/OTS/DeveloperDocs/Behaviour | Behaviour]]

== APIs ==

* [[Quality/QA-tools/OTS/DeveloperDocs/OTS_Server_API | Server]]

* [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/dev_branch_0.8/ots.server/ots/server/distributor/api.py Distributor]

* [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/dev_branch_0.8/ots.results/ots/results/api.py Results]

* [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/dev_branch_0.8/ots.worker/ots/worker/api.py Worker]

== Extending OTS ==

You can extend OTS with your own [http://wiki.meego.com/Quality/QA-tools/OTS/CreatingAPublisher Publisher]

== Experimental ==

Documentation for experimental branches, spikes, WIPs etc. can be found in [[Quality/QA-tools/OTS/DeveloperDocs/Experimental]].

== Analysis ==

[[Quality/QA-tools/OTS/DeveloperDocs/ErrorSituations| OTS 0.1 Error Situations]]

[[Quality/QA-tools/OTS/DeveloperDocs/TypicalUsage| Typical OTS users and usage scenarios]]

== Project Planning ==

* [[Quality/QA-tools/OTS/Roadmap | Roadmap]]

* [[Quality/QA-tools/OTS/DevelopmentStatus | DevelopmentStatus]]

* [[Quality/QA-tools/OTS/Meetings | Meetings]]

* [[Quality/QA-tools/OTS/ReleaseChecklist | ReleaseChecklist]]

* [[Quality/QA-tools/OTS/ReleasePractices | ReleasePractices]]

Quality/QA-tools/OTS

2011-01-28T13:24:02Z

Tvainio: /* OTS Index */

= OTS Index =

Welcome to the Open Test System (OTS) Wiki Index.

== Introduction ==

OTS is the '''Open Test Service''' for testing software systems on evolving hardware architectures.

It was born out of the test framework incrementally developed for use on [http://maemo.org/ Maemo] project and is in use for 24/7 testing service by [http://www.nokia.com/ Nokia].

=== What it does ===

'''OTS has the following primary functions:'''

* Distribution
* Load balancing
* Remote command running
* Hardware control
* Data Acquisition
* Data Analysis

'''OTS offers the following services:'''

* Fully Open Source
* Web server
* Extensible Architecture
* Interoperability

=== Why it does it ===

Quite simply: to increase efficiency and quality of on-device software testing.

=== How it does it ===

The [[Quality/QA-tools/OTS/Architecture | System Overview]] gives an outline of how OTS works

=== Installing ===

Follow the [[Quality/QA-tools/OTS/UserDocumentation/Installation | installation instructions]]

== Index ==

=== Developer Documentation ===

* [[Quality/QA-tools/OTS/Architecture | Overview]]

* [[Quality/QA-tools/OTS/DeveloperDocs | Developers Start Here ]]

* [[Quality/QA-tools/OTS/DevelopmentGuidelines | Development Guidelines]]

* [[Quality/QA-tools/OTS/Contributions | Contributions]]

=== Terminology ===

* [[Quality/QA-tools/OTS/Glossary | Glossary]]

=== System Administration ===

* [[Quality/QA-tools/OTS/Routing | Testrun Routing]]

* [[Quality/QA-tools/OTS/Protocols | Ports and Protocols]]

* [[Quality/QA-tools/OTS/Configuration | Configuration]]

* [[Quality/QA-tools/OTS/DeveloperDocs/OTS_Timeouts | Timeouts]]

* [[Quality/QA-tools/OTS/OptimisingThroughput | Optimising Throughput]]

=== Standards and Conventions ===

* [[Quality/QA-tools/OTS/Packages | Packages]]

* [[Quality/QA-tools/OTS/ErrorCodes | Error Codes]]

=== User Documentation ===

* [[Quality/QA-tools/OTS/UserDocumentation/Installation | Installation]]

* [[Quality/QA-tools/OTS/UserDocumentation | User Documentation]]

* [[Quality/QA-tools/OTS/Flashing | Flashing]]

Quality/QA-tools/OTS/DeveloperDocs

2011-01-13T11:33:59Z

Tvainio: /* Python Packages */

= OTS - Open Testing Service =

== Related Documentation ==

* [[Quality/QA-tools/OTS/Architecture]]
* [[Quality/QA-tools/OTS/Glossary]]
* [http://wiki.meego.com/Quality/QA-tools/OTS/DeveloperDocs/Transition0.1-0.8 Making transition from 0.1 to 0.8]

== Synopsis ==

OTS is a Distributed Test System that takes a Data Driven Approach using XML Files to allow parallel testing of Applications on evolving Hardware platforms.

Design has been driven by the need for a language neutral test platform that allows testing under different device architectures in parallel.

== High Level Overview ==

OTS allows a Test Sequence to be fanned out from a hub known as a '''Distributor''' to a remote machine known as a '''Worker'''.
Each '''Worker''' has a '''Device Group''' associated with it, this is defined in the configuration. The Tests are routed to the '''Worker'''
on the basis of the '''Device Group'''.

An XML '''Test Definition File''' describes the Tests that are run. The results are returned to the '''Distributor''' in an XML '''Results File'''.

The '''Distributor''' takes a '''Device Group''' and '''Timeout''' as inputs, as well as the '''Command''' for running '''Tasks'''. Tests are run within a '''Task''' which
is currently run as a process. The '''Command''' will typically take a path to the '''Test Definition File''' as an input.
If the process exceeds the '''Timeout''' the '''Task''' is stopped (the process is killed).

A number of '''Tasks''' can be added to a '''Testrun'''. The '''Testrun''' will normally wait for all '''Tasks''' to finish before completion.

Data (status, error and results) are communicated back from the '''Task''' to the '''Worker'''. The '''Results Object''' is a container for the files outputted: '''Test Definition File''', '''Results File'''
and any other files produced by the test as well as metadata relating to the run.

== Technology ==

OTS is written in Python 2.5.

OTS has the following FIXME (provided as separate PDF)

=== XmlFileFormats ===

The XML file formats provide the datum for the system.

Download [http://meego.gitorious.org/meego-quality-assurance/test-definition/trees/master them] and add the path to the '''results_xsd''' parameter in your [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/dev_branch_0.8/ots.server/ots/server/ots_server.ini ots_server.ini] file

=== Third Party Dependencies ===

==== RabbitMQ ====

You need to install [http://www.rabbitmq.com/server.html RabbitMQ].

===== Managing the Queues =====

If you need to delete or empty the queues there are [http://meego.gitorious.org/meego-quality-assurance/ots/trees/master/ots.server/ots/server/distributor/dev_utils utilities] to help with this.

==== Python Packages ====

The following python packages are needed

* [http://pypi.python.org/pypi/setuptools setuptools]

* [http://www.voidspace.org.uk/python/configobj.html configobj]

* [http://pypi.python.org/pypi/multiprocessing multiprocessing]

* [http://code.google.com/p/py-amqplib/ pyamqplib] (version 0.6.1 or newer)

* [http://www.familieleuthe.de/DownloadMiniXsv.html minixsv]

* [http://www.djangoproject.com/download/1.1.2/tarball/ django]

* [http://code.google.com/p/python-nose/ nose]

== Diving In ==

The following steps show how to get a Development Environment up and running

=== 1. Install Dependencies ===

* Install the Third Party Dependencies

* Get the XML File Formats and set the OTS_TESTDEFINITION environment variable

=== 2. Build the Eggs ===

From the root directory

<code>
./setup.sh
source ./paths.sh
</code>

=== 3. Unittests ===

Run the unittests from the root directory

<code>
./nose.sh
</code>

=== 4. Component Tests ===

Component Tests run high level tests on an OTS Component using Mocks of the dependencies.

You can run these with the component_tests shell script in the root directory.

<code>
./component_tests.sh
</code>

=== 5. Hello OTS World ===

The source contains a simple demonstration of the way the key elements work.

==== 1. Open two terminal windows. ====

==== 2. In the first window run the server ====

<code>
cd $SERVER
cd hub/demos
python demo_hub.py
</code>

==== 3. You should see this: ====

<code>
<snip>
20XX-XX-XX XX:XX:XX,XXX - ots.server.distributor.taskrunner - DEBUG - Sending command '['echo', 'hello world']' with key 'foo'
</code>

==== 4. In the second window run the worker ====

<code>
cd $WORKER
python worker.py -c config.ini
</code>

==== 5. Now in the Server terminal the command should run to completion. You should see the following line in the logs: ====

<code>
20XX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world
</code>

It will raise a PackageException. This is normal as no Tasks that are run return Packages to the Server

==== 6. And on the logs on the Worker side should contain: ====

<code>
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - DEBUG - Task in state: 'finish'
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - INFO - Recommence consume on queue: foo
</code>

== Installation ==

It is intended that the '''Distributor''' and each '''Worker''' should be installed to separate machines.

== Configuration ==

=== Distributor ===

The format of the configuration file is as follows.

The `host` is the name of the RabbitMQ server

<code>

[Client]
host = localhost
vhost = /
port = 5672
username = guest
password = guest
queue = FIXME
consumer_tag = worker
services_exchange = services

</code>

=== Worker ===

Edit your config.ini so that the

* `queue`
* `routing_key`
* `services_exchange`

Are set for your '''Device Group'''.

And the `host` is the name of your RabbitMQ server.

<code>
[Worker]
host = localhost
vhost = /
port = 5672
username = guest
password = guest
queue = foo
routing_key = foo
services_exchange = foo
</code>

==== Starting the Worker ====

To start your Worker:

<code>
python worker.py -c config.ini
</code>

== Automated System Tests ==

''''' Not yet merged to dev_branch_0.8 '''''

Automated system tests are available in ots.system_tests. They trigger various testruns with xmlrpc and check return value and log messages.

=== Requirements ===

* Fully functional OTS system with one worker.
* A SW Product "ots-system-tests" configured so that it defaults to the worker.
* A meego image with test-definition-tests and testrunner-lite-regression-tests.
* Django based advanced OTS setup.
* BeautifulSoup python library.

=== Configuration ===

* make the system test image available for the workers. (for example copy it to your apache static content directory /var/www/static/)
* Modify system_tests.conf in ots.system_tests directory to match your system.
* Make sure all the urls in system_tests.conf work.

=== Notes ===

* System tests execute multiple full testruns. Running all tests can easily take more than 1.5 hours with N900 worker.
* Make sure there's no other activities ongoing in the system while running the tests.
* Some error handling tests expect certain devicegroups and sw products _not_ to be available. If these test cases fail you can remove those queues with ots tools.

== License ==

OTS is distributed under an LGPL license.

== Contributions ==

From [http://meego.gitorious.org/meego-quality-assurance/ qa-tools]

== Wiki ==

[[Quality/QA-tools/OTS | OTS Wiki Home]]

== Source Code Documentation ==

FIXME

== Mailing List ==

FIXME

== IRC Channel ==

The #meego-qa-tools irc channel in freenode

== Bugs ==

The [http://bugs.meego.com/ Meego bugs page]

== Platform Dependencies ==

OTS is tested to run on Linux only. Productization environment is Ubuntu 8.04 LTS (Long Term Support).

== Architecture ==

* [[Quality/QA-tools/OTS/DeveloperDocs/Components | Components]]

* [[Quality/QA-tools/OTS/DeveloperDocs/Server | Server]]

* [[Quality/QA-tools/OTS/DeveloperDocs/Plugins | Plugins]]

* [[Quality/QA-tools/OTS/DeveloperDocs/DataFlow | DataFlow]]

* [[Quality/QA-tools/OTS/DeveloperDocs/Behaviour | Behaviour]]

== APIs ==

* [[Quality/QA-tools/OTS/DeveloperDocs/OTS_Server_API | Server]]

* [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/dev_branch_0.8/ots.server/ots/server/distributor/api.py Distributor]

* [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/dev_branch_0.8/ots.results/ots/results/api.py Results]

* [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/dev_branch_0.8/ots.worker/ots/worker/api.py Worker]

== Extending OTS ==

You can extend OTS with your own [http://wiki.meego.com/Quality/QA-tools/OTS/CreatingAPublisher Publisher]

== Experimental ==

Documentation for experimental branches, spikes, WIPs etc. can be found in [[Quality/QA-tools/OTS/DeveloperDocs/Experimental]].

== Analysis ==

[[Quality/QA-tools/OTS/DeveloperDocs/ErrorSituations| OTS 0.1 Error Situations]]

[[Quality/QA-tools/OTS/DeveloperDocs/TypicalUsage| Typical OTS users and usage scenarios]]

== Project Planning ==

* [[Quality/QA-tools/OTS/Roadmap | Roadmap]]

* [[Quality/QA-tools/OTS/DevelopmentStatus | DevelopmentStatus]]

* [[Quality/QA-tools/OTS/Meetings | Meetings]]

* [[Quality/QA-tools/OTS/ReleaseChecklist | ReleaseChecklist]]

* [[Quality/QA-tools/OTS/ReleasePractices | ReleasePractices]]

Quality/QA-tools/OTS/DevelopmentStatus

2011-01-12T14:55:13Z

Tvainio: /* Other Problems */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| In Progress (Spike completed)
|-
| Exceptions
| An understandable exception handling model. Documented
| In Progress (Awaiting review)
|-
| Logging
| A coherent distributed logging system
| Done
|-
| Upgrade Capability
| Worker version management
| Awaiting review
|-
| Function call
| Conductor should be function call not a process
| Postponed until next release
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| '''Not started'''
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Completed (Awaiting review)
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| '''BROKEN. Default device options are not handled correctly. see options_factory unit tests'''
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started''' It should be possible to implement this as a publisher. Reporting tools can handle getting target package information.
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| In Progress
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| done

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

== Other Problems ==

* '''<s>A testrun with invalid sw_product executes a testrun with sandbox-values. An error should be raised and no testrun should be executed.</s>'''
* '''Exceptions are not handled and they hit the xmlrpc. request_sync() should always return PASS/FAIL/ERROR. nothing else.'''
* '''Incompatible worker version leaves worker in an unknown state and does not report the problem clearly.'''

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2011-01-12T14:52:42Z

Tvainio: /* Application Framework */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| In Progress (Spike completed)
|-
| Exceptions
| An understandable exception handling model. Documented
| In Progress (Awaiting review)
|-
| Logging
| A coherent distributed logging system
| Done
|-
| Upgrade Capability
| Worker version management
| Awaiting review
|-
| Function call
| Conductor should be function call not a process
| Postponed until next release
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| '''Not started'''
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Completed (Awaiting review)
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| '''BROKEN. Default device options are not handled correctly. see options_factory unit tests'''
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started''' It should be possible to implement this as a publisher. Reporting tools can handle getting target package information.
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| In Progress
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| done

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

== Other Problems ==

* '''A testrun with invalid sw_product executes a testrun with sandbox-values. An error should be raised and no testrun should be executed.'''
* '''Exceptions are not handled and they hit the xmlrpc. request_sync() should always return PASS/FAIL/ERROR. nothing else.'''
* '''Incompatible worker version leaves worker in an unknown state and does not report the problem clearly.'''

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/UserDocumentation/Installation

2011-01-12T10:22:14Z

Tvainio: fixed example config file to avoid copypaste errors

== Installation Instructions for OTS 0.1 ==

These instructions have been tested with Ubuntu 10.04.

=== Network Setup ===

Using static IP addresses on all OTS machines is strongly recommended. It is possible to setup the system also with dynamic IP addresses but it might cause problems every now and then because IP's and hostnames change.

All OTS machines need to be able to access other OTS machines based on their hostname.

=== Server ===

==== prerequisites ====

* An smtp server is needed for OTS to be able to send result emails.
** You can use the smtp server provided by your ISP or organization
** Or you can setup your own. for example [http://my.opera.com/Contrid/blog/show.dml/478684 postfix]

==== Basic Setup ====

* Download and install RabbitMQ server from http://www.rabbitmq.com/server.html

* Install dependencies from ubuntu repository:
sudo apt-get update
sudo apt-get install python-setuptools python-amqplib python-django

* Install rest of the dependencies with easy_install:
sudo easy_install minixsv

* Get source code from http://meego.gitorious.org/meego-quality-assurance/ots/ (latest tag)

* Install ots.common:
cd ots/ots.common/
sudo python setup.py install

* Install ots.server:
cd ots/ots.server/
sudo python setup.py install

* Install ots.tools: (Required only for triggering testruns easily from command line over xmlrpc)
cd ots/ots.tools/
sudo python setup.py install

* Download and install test definition schema files from http://meego.gitorious.org/meego-quality-assurance/test-definition

* Create a directory for testrun logs and make sure the user running ots.server has write permissions to it:
sudo mkdir /var/log/ots

* At this point it should be possible to trigger testruns from python shell if you have a worker already set up and configured to use this server. Here's an example ipython session:
<pre>
In [1]: from ots.server.xmlrpc.public import request_sync

In [4]: program = "example_sw_product"

In [5]: request = "5"

In [6]: notify_list = ['tvainio@localhost']

In [7]: options = dict()

In [8]: options['image'] = "http://somewhere/some_image.bin"

In [9]: options['email'] = "off"

In [13]: options["device"] = "devicegroup:default" # Make this match the worker

In [14]: request_sync(program, request, notify_list, options)
Out[14]: 'FAIL'
</pre>

The result will be PASS/FAIL/ERROR. ERROR means that something went wrong in server side and testrun was not even started. FAIL means failed testrun and PASS means that everything went fine and all tests passed. Testrun logs with more information about the possible errors are available in /var/log/ots/

==== Configuring the Server ====

The previous example was not very useful. To get OTS server running as a real server you have two options. A simple xmlrpc server or a more advanced django based server. First thing you need to do is setup a custom config file.

===== Custom Config File =====

Ots server tries to import a configuration module ots_extensions.ots_config. If it does not succeed it uses the default from ots.server.testrun_host.default_ots_config. You can set up your own ots_config by following these instructions:

* Create a directory for your extensions:
mkdir my_ots_extensions

* Add it to your python path
export PYTHONPATH=/full/path/to/my_ots_extensions/:$PYTHONPATH

* Create custom ots_extensions python package:
mkdir my_ots_extensions/ots_extensions
touch my_ots_extensions/ots_extensions/__init__.py

* Copy the default config file from ots/ots.server/ots/server/testrun_host/default_ots_config.py to my_ots_extensions/ots_extensions/ots_config.py

* Modify the file to suit your needs. Check at least the following settings:
** Email settings (The smtp server OTS uses to send testrun results)
** xmlrpc server config (if you are setting the simple xmlrpc server setup. More info below.)
** The default options for your sw product:
You can define default values for all options here. The example_sw_product contains and example how to define options. You cannot use an sw_product if it does not contain the default options in ots_config. Here's a small example. I'm developing a Meego based coffeemaker and creating custom software images for it. To automate the testing I create an sw_product called "my_meego_variant":
'''TODO: Document all options'''
<pre>
#
# Defaults for my_meego_variant
#

# Lets start with the global_defaults
default_options["my_meego_variant"] = global_defaults.copy()

# The images work only on meego coffeemakers so the default devicegroup will be "meego_coffeemaker"
default_options["my_meego_variant"]['device'] = {'devicegroup':'meego_coffeemaker'}

# Coffeemaker needs to be fast so we define a 3 minutes global timeout for the test execution.
default_options["my_meego_variant"]['timeout'] = 3
</pre>

===== Extension Points =====

OTS uses plugins to do various things during the testrun. You can write your own plugins to add more functionality by implementing a module ots_extensions.extension_points. An example file can be found from ots/ots.server/ots/server/testrun_host/example_extension_points.py

==== Simple XMLRPC Server ====

* After you have implemented your custom config file and defined suitable xmlrpc_host and xmlrpc_port values you can start the simple xmlrpc server:
ots_server

* After that you can start testruns on your ots_server by calling the request_sync() method over xmlrpc just like we did locally in the example above. There's also a command line tool called ots_trigger in ots.tools package.

* A testrun can be triggered from command line with ots_trigger:

ots_trigger -s your_xmlrpc_host:your_xmlrpc_port -b 333 -i url_to_dummyimage -p my_meego_variant -e your_email_address

The result should be FAIL and if your email server settings are correct you should also receive an email.

* If we look into the new testrun log file in /var/log/ots/ we should find something like this:

<pre>
2010-09-24 15:48:21,188 conductorengine ERROR Device group 'meego_coffeemaker' does not exist
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/ots.server-0.1dev-py2.6.egg/ots/server/conductorengine/conductorengine.py", line 150, in execute
self._taskrunner.run()
File "/usr/local/lib/python2.6/dist-packages/ots.server-0.1dev-py2.6.egg/ots/server/distributor/taskrunner.py", line 417, in run
(self._routing_key))
OtsQueueDoesNotExistError: No queue for meego_coffeemaker
</pre>

* Next step is to set up an OTS worker into the devicegroup meego_coffeemaker. Instructions can be found below.

==== Django Based Server ====

With Django and Apache it's possible to set up a more advanced ots server.

There's for example a global logger application available in ots.server.logger. By setting up the logger it is possible to get log messages for a specific testrun from all parts of the system (ots server, ots conductor and testrunner-lite). The logs are stored into database and can be accessed with a web UI. It's also easy to implement a result db and web views for the test results.

If you are not familiar with Django you can study the basics from the tutorial: http://docs.djangoproject.com/en/1.2/intro/tutorial01/

These instructions assume that you have already done the setup described in: [[http://wiki.meego.com/Quality/QA-tools/OTS/UserDocumentation/Installation#Server]]

There is an example django project in ots.server.django_ots. It is possible to use that directly but in most cases you should create your own django project to be able to modify the settings.

===== Installation Using the Default Django Project in ots.server.django_ots =====

* Install required packages:
<pre>
sudo apt-get update
sudo apt-get install python-django python-setuptools rabbitmq-server python-amqplib
</pre>

* Install django-xmlrpc from https://launchpad.net/django-xmlrpc to somewhere in your pythonpath

* After that you should be able to execute tests for the ots django applications:
<pre>
python ots/server/django_ots/manage.py test logger testrun_db
</pre>

* Create the database. The default ots django project uses sqlite. The DB file is located in /opt/ots.sqlite.
<pre>
python ots/server/django_ots/manage.py syncdb
</pre>

The script asks about creating a superuser for the Django's auth system. OTS does not use the auth system at the moment but it might be useful to create the account if you start writing custom django applications or want to use django admin interface at some point.

If you get errors like "unable to open database file" make sure your user has write permissions to /opt/

* Install apache and mod-python: (OTS requires the prefork version of apache)

sudo apt-get install apache2-mpm-prefork libapache2-mod-python

* Setup your custom ots_config and extension points
** Instructions are available in [[http://wiki.meego.com/Quality/QA-tools/OTS/UserDocumentation/Installation#Configuring_the_Server]]
** Use ots/server/django_ots/extension_points.py as the basis of your extension points. It takes the testrun_db into use.
** Changes http_loggin_enabled to True in your ots_config.py if you want to have global testrun logs:
http_logging_enabled = True

* Configure your django_ots to /etc/apache2/apache2.conf by adding the following:

<pre>
<Location "/">
SetHandler python-program
PythonHandler django.core.handlers.modpython
SetEnv DJANGO_SETTINGS_MODULE ots.server.django_ots.settings
PythonDebug On
PythonPath "['ADD HERE THE PATH WHERE YOUR ots_extensions directory with CUSTOM OTS ENTRYPOINTS AND CONFIGFILE CAN BE FOUND'] + sys.path"
</Location>
</pre>

* Open http://localhost/logger/view/ in your web browser. You should see an empty logging message page.
** If you get "Unable to open database file" make sure that the user apache is running on has read and write permissions to the db file (/opt/ots.sqlite).
* Try triggering a testrun with ots_trigger (note the /xmlrpc in server url):
ots_trigger -s localhost/xmlrpc -b 333 -i url_to_dummyimage -p my_meego_variant -e your_email_address

If everything went ok you should see a succesful testrun and logs from OTS server, OTS Worker.conductor and testrunner-lite appearing in the http://localhost/logger/view/ and http://localhost/logger/view/ots/<your_testrun_id>/

If you only see log messages from your server most probably yuor hostnames are not set up correctly. The worker machine needs to be able to access the server by it's host name. Or you can try to play around with the "results_storage_config["host"]" setting in ots_config.py. It is the hostname ots_server sends to worker to use for global logging.

===== Installation Using a Custom Django Project =====

If you want to use different django setup you can create a custom django project. You can use ots/server/django_ots/ as an example. Django documentation contains lots of information about the features and settings: http://www.djangoproject.com/

The most important files are:
* settings.py - DB settings, installed django applications, email etc.
* urls.py - Url mappings. For example the "xmlrpc/" url is defined here.

You also need to modify your apache configuration to point to your own django project instead of the default one. Also note that if you change your code, mod_python does not automatically reload the changed code so you need to restart apache to make sure you are using the right version of your code. If you do a full apache restart all ongoing testruns will fail. If you do a graceful restart (<code>"apache2ctl graceful"</code>)the testruns won't fail but updated code is reloaded only after the testruns have fininshed.

==== Uploading Results to Meego QA Reports Tool ====

Starting from version 0.1.6 OTS can automatically upload results to Meego QA Reports tool.

* Make sure ots.qareports_plugin is installed.
* Modify values in /etc/ots_qareports_plugin.conf to match your setup:

<pre>
# More information about the options is available in
# http://wiki.meego.com/Quality/QA-tools/QAReports/API

[ots.qareports_plugin]
host = "localhost"
url = "/api/import"
auth_token = ""
release_version = "1.0"
target = "Core"
testtype = "Acceptance"
hwproduct = "N900"
</pre>

* Modify "get_resultbackends()" in your custom extension_points.py to take QaReportsBackend into use:

<pre>
from ots.qareports_plugin.qareports_backend import QAReportsBackend

def get_resultbackends(testrun):
"""Returns custom ResultBackends as a list"""
return [QAReportsBackend()]
</pre>
===== Dynamic Report Control =====

Starting from OTS 0.1.7 it's possible to control reporting options dynamically when triggering a testrun with "--options" command line parameter in ots_trigger:
<pre>
ots_trigger --options="qa_hwproduct:<HWPRODUCT> qa_testtype:<TESTTYPE> qa_target:<TARGET> qa_release_version:<RELEASE_VERSION>"
</pre>

=== Worker ===

* Install dependencies:
sudo apt-get update
sudo apt-get install python-setuptools python-amqplib

* Get source code from http://meego.gitorious.org/meego-quality-assurance/ots/ (latest tag)

* Install ots.common:
cd ots/ots.common/
sudo python setup.py install

* Install ots.worker:
cd ots/ots.worker/
sudo python setup.py install

* Configure the worker by editing devicegroup and server address in /etc/ots.ini:
<pre>
###########################################
# REPLACE `fix_me` WITH YOUR DEVICEGROUP
###########################################
[Device]
# Change this to the devicegroup this worker belongs to
devicegroup = fix_me

#############################
# Optional device properties
#############################

# Uncomment these if you want more detailed testrun routing
# See http://wiki.meego.com/Quality/QA-tools/OTS/Routing for details

#devicename = name of the device type
#deviceid = Specific HW id of the device

#############################
# ADVANCED CONFIGURATION
#############################
[Worker]
# Change this to your ots server address
host = your_server_address
vhost = /
port = 5672
username = guest
password = guest
log_file = /var/log/ots.log
</pre>

More info about testrun routing is available here: [[Quality/QA-tools/OTS/Routing]]

* Start the worker:

sudo ots_worker

* If everything went ok, you should see something like this:
<pre>
2010-11-08 12:55:10,790 - ots.worker.worker - DEBUG - Initialising the server
2010-11-08 12:55:10,790 - ots.worker.connection - DEBUG - Connecting to RabbitMQ
2010-11-08 12:55:10,894 - amqplib - DEBUG - Start from server, version: 8.0, properties: {u'platform': 'Erlang/OTP', u'product': 'RabbitMQ', u'version': '2.1.0', u'copyright': 'Copyright (C) 2007-2010 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.', u'information': 'Licensed under the MPL. See http://www.rabbitmq.com/'}, mechanisms: ['PLAIN', 'AMQPLAIN'], locales: ['en_US']
2010-11-08 12:55:10,935 - amqplib - DEBUG - Open OK! known_hosts []
2010-11-08 12:55:10,935 - amqplib - DEBUG - using channel_id: 1
2010-11-08 12:55:10,937 - amqplib - DEBUG - Channel open
2010-11-08 12:55:10,938 - ots.worker.worker - DEBUG - Starting the worker. server: 192.168.1.102:5672, device_properties: {'devicegroup': 'meego_coffeemaker'}
2010-11-08 12:55:10,957 - ots.worker.task_broker - INFO - consume on queue: meego_coffeemaker
2010-11-08 12:55:10,958 - ots.worker.task_broker - DEBUG - Starting main loop...
</pre>

* There's also an init script available in ots.worker/debian/python-ots-worker.ots-worker.init to make running the worker easier

* After that the worker should be ready for receiving tasks. To make sure the actual test execution works please see the instructions for ots.worker.conductor: [[Quality/QA-tools/OTS/UserDocumentation/Conductor| Conductor]]

Quality/QA-tools/OTS/DevelopmentStatus

2011-01-12T07:38:03Z

Tvainio: /* Server */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| In Progress (Spike completed)
|-
| Exceptions
| An understandable exception handling model. Documented
| In Progress (Awaiting review)
|-
| Logging
| A coherent distributed logging system
| '''Not working properly. Debug level logging to file is missing and http logger needs to be configured to INFO level.'''
|-
| Upgrade Capability
| Worker version management
| Awaiting review
|-
| Function call
| Conductor should be function call not a process
| Postponed until next release
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| '''Not started'''
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Completed (Awaiting review)
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| '''BROKEN. Default device options are not handled correctly. see options_factory unit tests'''
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started''' It should be possible to implement this as a publisher. Reporting tools can handle getting target package information.
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| In Progress
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| done

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

== Other Problems ==

* '''A testrun with invalid sw_product executes a testrun with sandbox-values. An error should be raised and no testrun should be executed.'''
* '''Exceptions are not handled and they hit the xmlrpc. request_sync() should always return PASS/FAIL/ERROR. nothing else.'''
* '''Incompatible worker version leaves worker in an unknown state and does not report the problem clearly.'''

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/Logging

2010-12-31T10:28:43Z

Tvainio: /* Log Levels */

= Logging =

== Log Levels ==

Different log levels are targeted for different user groups. See [[Quality/QA-tools/OTS/DeveloperDocs/TypicalUsage| Typical OTS users and usage scenarios]] for more information.

=== For Basic Users ===

''These are available in the centralized testrun http log.''

* INFO shows what's happening in high level
** What parameters where used for the request
** What's happening right know. (Downloading image x from http://..., Flashing, Executing package asdf-tests.)

* ERROR shows what went wrong.
** It needs to be explained so that the basic user can understand it
** Always think from the basic user point of view. What makes sense to you as a developer probably does not make sense for the user at all.
** Don't say "_get_file("tests.xml") failed!", say "Testpackage asdf-tests is not in the image".
** Tracebacks are always good if they provide additional information for debugging. Just make sure that the actual error message is still there. Most of the users don't know python and do not want to read tracebacks. Instead of "Error! Traceback follows:", say "Custom flasher was not able to flash the image:". Then the basic user gets a rough idea what's going on and system maintainer/ots developer can solve the issue by looking at the traceback.

* WARNING warns if something is wrong but it's not critical
** For example user gives unknown input parameter. It might be a typo so the user should be warned.

=== For Advanced User / Worker Maintainer and Server Maintainer ===

''These logs are available in log files. Currently server debug log for each testrun is in /var/log/ots/. Conductor logs to ~/conductor.log in the worker machine. OTS worker process logs to /var/log/ots.log.''

* DEBUG on worker components
** In most cases INFO and ERROR level messages should be enough but we expect worker maintainers to be able to follow also DEBUG level messages
** DEBUG messages should give enough information to sort out problems in the worker setup.
** Don't say "calling flasher...". Say "executing command '/usr/bin/flasher -f imagefile'"
** Don't log too much. It makes reading the log difficult. Tracebacks should tell where we were when error happened.
** Input parameters and sub commands (parameters, return values, stdout, stderr) are likely to be valuable information when debugging.
==== How to Log Exceptions ====

Logging record objects have exc_info field and that should be used. No need to format or include the exception in the actual log message. This way it's easy to format every exception properly in the logging handlers or views. Current http logger views use this.
* In exception handlers it's enough to do
<pre>
log.exception("Catastrophic error")
</pre>
or
<pre>
log.debug("Catastrophic error", exc_info=True)
</pre>
If exception is received as an object, it can be logged properly like this:
<pre>
except Exception, err:
log.error("Catastrophic error", exc_info=err)
</pre>
or if it's the common (Type, Value, Traceback) tuple:
<pre>
def __exit__(self, exception_type, value, tb):
LOG.warning("Error in plugin", exc_info=(exception_type, value, tb))
</pre>

== Logs ==

There are a number of ways to view the OTS testrun:

=== Main Log ===

The Main OTS log is written to

<code>
/var/log/ots
</code>

=== Server Log ===

This is [http://docs.python.org/release/2.5.2/lib/logging-config-fileformat.html configurable] by the 'logging.conf' file in the root directory

=== Worker Log ===

The Worker can be configured to log to a file specified by the "log_file" parameter in the Worker config file.

=== Conductor Log ===

Writes to the <ahem> home directory

<code>
/path/to/home/conductor.log
</code>

=== Apache Log ===

If you are running OTS with Apache:

<code>
/var/log/apache2/error.log
</code>

== Http Logger ==

In addition there is an [http://wiki.meego.com/Quality/QA-tools/OTS/Plugins/HTTP_logger http_logger].

Quality/QA-tools/OTS/DevelopmentStatus

2010-12-31T10:20:36Z

Tvainio: /* Other Problems */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| In Progress (Spike completed)
|-
| Exceptions
| An understandable exception handling model. Documented
| In Progress (Awaiting review)
|-
| Logging
| A coherent distributed logging system
| '''Not working properly. Debug level logging to file is missing and http logger needs to be configured to INFO level.'''
|-
| Upgrade Capability
| Worker version management
| Awaiting review
|-
| Function call
| Conductor should be function call not a process
| Postponed until next release
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| '''Not started'''
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Completed (Awaiting review)
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| '''BROKEN. Default device options are not handled correctly. see options_factory unit tests'''
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started''' It should be possible to implement this as a publisher. Reporting tools can handle getting target package information.
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| '''Merge from 0.1'''
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| done

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

== Other Problems ==

* '''A testrun with invalid sw_product executes a testrun with sandbox-values. An error should be raised and no testrun should be executed.'''
* '''Exceptions are not handled and they hit the xmlrpc. request_sync() should always return PASS/FAIL/ERROR. nothing else.'''
* '''Incompatible worker version leaves worker in an unknown state and does not report the problem clearly.'''

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2010-12-31T09:17:07Z

Tvainio: /* Release 0.8 unstable */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| In Progress (Spike completed)
|-
| Exceptions
| An understandable exception handling model. Documented
| In Progress (Awaiting review)
|-
| Logging
| A coherent distributed logging system
| '''Not working properly. Debug level logging to file is missing and http logger needs to be configured to INFO level.'''
|-
| Upgrade Capability
| Worker version management
| Awaiting review
|-
| Function call
| Conductor should be function call not a process
| Postponed until next release
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| '''Not started'''
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Completed (Awaiting review)
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| '''BROKEN. Default device options are not handled correctly. see options_factory unit tests'''
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started''' It should be possible to implement this as a publisher. Reporting tools can handle getting target package information.
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| '''Merge from 0.1'''
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| done

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

== Other Problems ==

* '''A testrun with invalid sw_product executes a testrun with sandbox-values. An error should be raised and no testrun should be executed.'''
* '''Exceptions are not handled and they hit the xmlrpc. request_sync() should always return PASS/FAIL/ERROR. nothing else.'''
* '''Host based testing does not work at all. It complains that no tests were executed.'''

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2010-12-31T09:13:31Z

Tvainio: /* Server */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| In Progress (Spike completed)
|-
| Exceptions
| An understandable exception handling model. Documented
| In Progress (Awaiting review)
|-
| Logging
| A coherent distributed logging system
| '''Not working properly. Debug level logging to file is missing and http logger needs to be configured to INFO level.'''
|-
| Upgrade Capability
| Worker version management
| Awaiting review
|-
| Function call
| Conductor should be function call not a process
| Postponed until next release
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| '''Not started'''
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Completed (Awaiting review)
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| '''BROKEN. Default device options are not handled correctly. see options_factory unit tests'''
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started''' It should be possible to implement this as a publisher. Reporting tools can handle getting target package information.
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| '''Merge from 0.1'''
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| done

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2010-12-31T06:09:55Z

Tvainio: /* Plugins */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| In Progress (Spike completed)
|-
| Exceptions
| An understandable exception handling model. Documented
| In Progress (Awaiting review)
|-
| Logging
| A coherent distributed logging system
| '''Not working properly. Debug level logging to file is missing and http logger needs to be configured to INFO level.'''
|-
| Upgrade Capability
| Worker version management
| Awaiting review
|-
| Function call
| Conductor should be function call not a process
| Postponed until next release
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| '''Not started'''
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Completed (Awaiting review)
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| In Progress (Functionality added config consistency needed)
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started''' It should be possible to implement this as a publisher. Reporting tools can handle getting target package information.
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| '''Merge from 0.1'''
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| done

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2010-12-30T15:23:28Z

Tvainio: /* Plugins */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| In Progress (Spike completed)
|-
| Exceptions
| An understandable exception handling model. Documented
| In Progress (Awaiting review)
|-
| Logging
| A coherent distributed logging system
| '''Not working properly. Debug level logging to file is missing and http logger needs to be configured to INFO level.'''
|-
| Upgrade Capability
| Worker version management
| Awaiting review
|-
| Function call
| Conductor should be function call not a process
| Postponed until next release
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| '''Not started'''
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Completed (Awaiting review)
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| In Progress (Functionality added config consistency needed)
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started''' It should be possible to implement this as a publisher. Reporting tools can handle getting target package information.
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| '''Merge from 0.1'''
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| '''In progress'''

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2010-12-30T15:17:24Z

Tvainio: /* Application Framework */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| In Progress (Spike completed)
|-
| Exceptions
| An understandable exception handling model. Documented
| In Progress (Awaiting review)
|-
| Logging
| A coherent distributed logging system
| '''Not working properly. Debug level logging to file is missing and http logger needs to be configured to INFO level.'''
|-
| Upgrade Capability
| Worker version management
| Awaiting review
|-
| Function call
| Conductor should be function call not a process
| Postponed until next release
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| '''Not started'''
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Completed (Awaiting review)
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| In Progress (Functionality added config consistency needed)
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started''' It should be possible to implement this as a publisher. Reporting tools can handle getting target package information.
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| '''Merge from 0.1'''
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| '''merge from OTS 0.1 and write the plugin wrapper'''

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2010-12-30T08:37:10Z

Tvainio: /* Plugins */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| In Progress (Spike completed)
|-
| Exceptions
| An understandable exception handling model. Documented
| In Progress (Awaiting review)
|-
| Logging
| A coherent distributed logging system
| '''Not working properly. HTTP logger does not receive server side log messages from the components that are imported before publisher plugins are loaded'''
|-
| Upgrade Capability
| Worker version management
| Awaiting review
|-
| Function call
| Conductor should be function call not a process
| Postponed until next release
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| '''Not started'''
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Completed (Awaiting review)
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| In Progress (Functionality added config consistency needed)
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started''' It should be possible to implement this as a publisher. Reporting tools can handle getting target package information.
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| '''Merge from 0.1'''
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| '''merge from OTS 0.1 and write the plugin wrapper'''

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2010-12-30T08:36:08Z

Tvainio: /* Server */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| In Progress (Spike completed)
|-
| Exceptions
| An understandable exception handling model. Documented
| In Progress (Awaiting review)
|-
| Logging
| A coherent distributed logging system
| '''Not working properly. HTTP logger does not receive server side log messages from the components that are imported before publisher plugins are loaded'''
|-
| Upgrade Capability
| Worker version management
| Awaiting review
|-
| Function call
| Conductor should be function call not a process
| Postponed until next release
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| '''Not started'''
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Completed (Awaiting review)
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| In Progress (Functionality added config consistency needed)
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started''' It should be possible to implement this as a publisher. Reporting tools can handle getting target package information.
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| '''Merge from 0.1'''
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| merge from OTS 0.1 and write the plugin wrapper

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2010-12-30T08:32:05Z

Tvainio: /* Application Framework */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| In Progress (Spike completed)
|-
| Exceptions
| An understandable exception handling model. Documented
| In Progress (Awaiting review)
|-
| Logging
| A coherent distributed logging system
| '''Not working properly. HTTP logger does not receive server side log messages from the components that are imported before publisher plugins are loaded'''
|-
| Upgrade Capability
| Worker version management
| Awaiting review
|-
| Function call
| Conductor should be function call not a process
| Postponed until next release
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| '''Not started'''
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Completed (Awaiting review)
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| In Progress (Functionality added config consistency needed)
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started'''
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| '''Merge from 0.1'''
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| merge from OTS 0.1 and write the plugin wrapper

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DevelopmentStatus

2010-12-30T06:37:12Z

Tvainio: /* Plugins */

= Release 0.8 unstable =

== Application Framework ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Configuration
| Single configuration file and format,
| Done
|-
| Plugins
| Runtime discoverable mechanism, with clear usage model
| In Progress (Spike completed)
|-
| Exceptions
| An understandable exception handling model. Documented
| In Progress (Awaiting review)
|-
| Logging
| A coherent distributed logging system
| Done
|-
| Upgrade Capability
| Worker version management
| Awaiting review
|-
| Function call
| Conductor should be function call not a process
| Postponed until next release
|-
| Properties
| Merge from 0.1 branch
| Done
|-
| Exceptions, regression
| Check that all relevant exceptions from 0.1 are in place
| '''Not started'''
|}

== Common ==
{| border="1"
|-
! Feature
! Description
! Status
|-
| Data Transfer Objects
| Taxonomy thereof. Pythonic APIs
| Completed (Awaiting review)
|-
| TestDefinition
| Removed
|
|}

== Server ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| High Level API
| Client facing interface
| Awaiting review
|-
| Hub
| High level procedural code
| In Progress
|-
| Disambiguation Datamodel
| Clear persistence and data management
| In Progress (Persistence parameters separated out)
|-
| Distributor
| Opaque Data Transfer
| Completed
|-
| Default Options
| Mapping of Default Options to SW product
| In Progress (Functionality added config consistency needed)
|-
| XMLRPC
| Add XMLRPC interface
| Completed
|-
| Allocator
| API for custom Task allocation
| Awaiting review
|-
| Publisher Plugins
| API and demos for 3rd party Publisher Plugins
| Completed
|-
| Input Plugin
| Incorporate and mark as deprecated
| '''Not started'''
|-
| Allocator Entry Point
| Add appropriate extension point to allocator to allow custom distribution
| '''Awaiting merge'''
|-
| Parallel Testruns
| Trigger the same testrun on multiple targets
| '''Merge from 0.1'''
|}

== Worker ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Integrate Conductor
|
| Full tool chain running
|}

== Plugins ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| BIFH Plugin
| Client facing interface
| In Progress (Spike completed awaiting review)
|-
| Reporting Plugin
| Clear separation between test tool and reporting
| In Progress (First pass API. Stubbed demo)
|-
| Email Plugin
| Email Plugin for Publisher API
| Requires system tests
|-
| QA Reports Plugin
| Publisher plugin for Meego QA Reports tool
| merge from OTS 0.1 and write the plugin wrapper

|}

== Usability Issues ==

{| border="1"
|-
! Feature
! Description
! Status
|-
| Unittests run as sudo
| Queue size API calls
| Fixed
|-
| Developer Installation out-of-the-box
| Make as easy as possible. Clear instructions.
| Awaiting review
|-
| API Documentation
| All component APIs documented
| In Progress
|-
| You Tube Video
| Instruction Video
| In Progress
|-
|}

= Scheduled for 0.9 Release =

== Organisation, maintenance and upgrades ==

* Component reorg (Assess a better representation of Components in code layout. Check impact on deployment)

* Better unittest harvester (perhaps ''setup.py test'')

* Evaluate status of dependencies (AMQP 1.0, minixsv supports Python 2.5, setuptools versus distribute etc)

== Conductor ==

* Python APIs

* API stability

* 6Pack integration

* Automated reboot

== Server ==

* Remove InputPlugin

* Lower level extension points on Result File Parsing

* 6Pack Integration

= Not scheduled items =

== Security ==

* HTTPS XML-RPC

* AMQP connection using SSL encryption

* OTS server and worker working in user mode

* Define firewall settings for OTS server and worker

* User authentication and access rights

* Test request DDoS filter

Quality/QA-tools/OTS/DeveloperDocs/ErrorSituations

2010-12-29T14:21:12Z

Tvainio: /* /ots/server/testrun_host/testrun_host.py:451: */

= OTS 0.1 Error Situations =

OTS 0.1 has 2 ways of handling error information. The testrun has a specific "set_error_info() and set_error_code()" mechanism for critical errors. This is the main mechanism. In addition to that error messages are printed to http log. The error info mechanism is a legacy thing from the old days we didn't have global http logging or amqp.

In OTS 0.8 we have the same http logging. The set_error_info() stuff is gone but it is replaced by exceptions that can be sent over the wire with amqp.

== Critical Errors ==

These cause the testrun to end with result ERROR

=== /ots/common/results/resultjudge.py:169: ===
<pre>
self.testrun.set_error_info("No test packages defined nor found.")
</pre>
* Raised if user didn't specify test packages and conductor couldn't found any.
* This should practically never happen. Conductor should raise other errors before this if image does not contain packages.

=== /ots/common/results/resultjudge.py:179: ===
<pre>
self.testrun.set_error_info("Results missing from: %s" \
</pre>
* Raised if server didn't receive results.xml for all testpackages this.
* This can happen for example if global timeout happens. (Conductor detects timeouts and raises Timeout error so this shouldn't happen anymore?)

=== /ots/server/testrun_host/testrun_host.py:252: ===
<pre>
try:
return ots_config.default_options[sw_product]
except (KeyError, ValueError):
raise Exception("Unknown sw_product %s" % sw_product)
</pre>
* Raised if user tries to use sw product that is not defined in the config file
* '''System test: log_tests.TestErrorConditions.test_non_existing_devicegroup'''

=== /ots/server/testrun_host/testrun_host.py:392: ===
<pre>
for package in test_packages:
if not (package.endswith("-tests")\
or package.endswith("-test")\
or package.endswith("-benchmark")):
invalid_packages.append(package)
</pre>
* error_msg = "Invalid testpackage(s): %s" % invalid_packages
* This is raised if user input contains test packages that do not end -test, -tests, -benchmark
* '''System test: log_tests.TestErrorConditions.test_bad_testpackage_names

=== /ots/server/testrun_host/testrun_host.py:410: ===
<pre>
if not (self.testrun.get_rootstrap_url() \
or self.testrun.get_image_url()):
</pre>
* error_msg = "No image url or rootstrap url defined."
* Raised if image url is not defined in input. (rootstrap is deprecated!)
* '''System test: log_tests.TestErrorConditions.test_no_image_url

=== /ots/server/testrun_host/testrun_host.py:420: ===
<pre>
if not self.testrun.get_image_url() and \
self.testrun.get_host_testpackages():

error_msg = "Image url missing. Executing host based tests "+\
"requires image url."
</pre>
* Deprecated! Image url is mandatory always so this will never happen. The previous error will be raised first.

=== /ots/server/testrun_host/testrun_host.py:443: ===
<pre>
valid_values = ['default', 'perpackage']
for model in custom_models:
valid_values.append(model[0])
model = self.testrun.get_option('distribution_model')

if model not in valid_values:
</pre>
* error_msg = "Invalid distribution model: %s" % model
* Raised if user gives unknown distribution model parameter. (Extension points can introduce custom distribution models in addition to the default values shown.)
* '''System test: log_tests.TestErrorConditions.test_bad_distribution_model

=== /ots/server/testrun_host/testrun_host.py:451: ===
<pre>
if model == 'perpackage' and len(self.testrun.get_testpackages()) == 0:
error_msg = "Test packages must be defined for specified "\
+"distribution model '%s'" % model

</pre>
* Raised if user tries to use perpackage distribution model but does not explicitly specify testpackages.
* '''System test: log_tests.TestErrorConditions.test_perpackage_distribution_no_packages

=== /ots/server/conductorengine/conductorengine.py:168: ===
<pre>
except OtsQueueDoesNotExistError:
error_info = "Queue '%s' does not exist" \
% (self._routing_key)
</pre>

* Raised if requested worker queue is not available in amqp server. (No suitable workers have ever been added to the system)
* '''System test: log_tests.TestErrorConditions.test_non_existing_devicegroup'''

=== /ots/server/conductorengine/conductorengine.py:176: ===
<pre>

error_info = ("Server side timeout. (Worker went offline during "+\
"testrun or some tasks were not started in time)")
</pre>

* Raised if server side global timeout is hit. See timeouts section in wiki for more details.
* This should be very rare. (but might still happen)

=== /ots/server/conductorengine/conductorengine.py:182: ===
<pre>
except OtsConnectionError:
error_info = "A connectivity issue was encountered"
</pre>
* Raised if amqp connection fails
* Should be very rare

=== /ots/server/conductorengine/conductorengine.py:190: ===
<pre>
except OtsQueueTimeoutError, exception:
error_info = ("The job was not started within %s minutes. "+\
"A worker in that devicegroup may be down or there may be "+\
"exceptional demand") % str(exception.timeout_length / 60)
</pre>
* Queue timeout. Raised if a worker does not take the task from the queue in time.

=== /ots/server/conductorengine/conductorengine.py:196: ===
<pre>
except Exception:
error_info = "A miscellaneous error was encountered"

</pre>
* Generic ots distributor error. This is bad!

=== /ots/worker/conductor/executor.py:277: ===
<pre>
def _test_execution_error_handler(self, error_info, error_code):
"""
Handler for testrun timed out errors and ConductorError exceptions
that are raised during test execution.
"""
self.log.error("Test execution error: %s " % error_info)
if not self.stand_alone:
self.responseclient.set_error(error_info, error_code)
</pre>
* Generic conductor test execution & timeout handler.
* More detailed analysis is part of ots 0.9. For 0.8 just make sure these are sent to server and communicated to user properly.
* '''System test: log_tests.TestErrorConditions.test_timeout'''

=== /ots/worker/conductor/conductor.py:371: ===
<pre>
except ConductorError, exc:
log.error("%s (ots error code: %s)" % (exc.error_info, exc.error_code))
if not stand_alone:
responseclient.set_error(exc.error_info, exc.error_code)
</pre>
* Generic conductor exception handler.
* More detailed analysis is part of ots 0.9. For 0.8 just make sure these are sent to server and communicated to user properly.

=== /ots/worker/conductor/conductor.py:378: ===
<pre>
except:
log.error("Unknown error when trying to execute tests.")
log.error("Traceback follows:", exc_info = True)
if not stand_alone:
responseclient.set_error(str(sys.exc_info()[1]), "999")
</pre>
* Conductor last resport try-except.

=== /ots.worker/ots/worker/task_broker.py:206: ===
<pre>
try:
self._dispatch(command)

except (CommandFailed):
LOGGER.error("Process failed")
error_info = "Task execution failed"
</pre>
* Raised if the task command process (=conductor) returns something else than 0.

== Noncritical Errors ==

These produce an error message to log but do not affect the testrun result. These should be warnings.

=== /ots.server/ots/server/testrun_host/testrun_host.py:185: ===
<pre>
try:
if not ots_config.debug_mode:
for text, url in self.testrun.get_result_links():
self.input_plugin.store_url(url, text)
except:
self.log.error("Publishing result url failed. "\
"Traceback follows:", exc_info = True)
</pre>
* OTS tries to publish the result url with input plugin and fails.

=== /ots.server/ots/server/testrun_host/testrun_host.py:192: ===
<pre>
try:
if self.testrun.email_enabled() and self.emailbackend:
self.emailbackend.send_mail()
self.log.error("Sending result email failed. Traceback follows:",
</pre>
* OTS fails to send result email

=== /ots.server/ots/server/testrun_host/testrun_host.py:269: ===
<pre>
self.log.info("Publishing testrun log link")
try:
if not ots_config.debug_mode:
self.input_plugin.store_url(url, text)
except:
self.log.error("Publishing testrun log link failed. "\
"Traceback follows:", exc_info = True)

</pre>
* Another input plugin warning

=== /ots.server/ots/server/testrun_host/testrun_host.py:461: ===
<pre>
try:
target_packages = self.input_plugin.get_changed_packages(build_id)
except:
self.log.error("Getting target packages failed. "\
"Traceback follows:", exc_info = True)
target_packages = []
</pre>
* OTS tries to fetch information about the target packages changed in the build request and fails.

=== /ots.server/ots/server/email_backend/mailmessage.py:145: ===
<pre>
try:
attach_as_zip_file(msg, result_files, unique_id)
except:
LOG.error("Something went wrong when creating email attachment. "\
"Traceback follows:", exc_info = True)
</pre>
* Creating the attachment zip package failed.

=== /ots.server/ots/server/email_backend/email_backend.py:92: ===
<pre>
if not (type(to_address_list) == type(list()) \
or type(to_address_list) == type("")):
self.log.error("Error in sending mail: expected list or string in "\
"to_address_list: %s " % str(to_address_list))
return
</pre>
* Bad email address input

=== /ots.server/ots/server/email_backend/email_backend.py:121: ===
<pre>
self.log.error("Error in sending mail to following addresses:")
self.log.error(str(failed_addresses))
</pre>

=== /ots.qareports_plugin/ots/qareports_plugin/qareports_client.py:60: ===
<pre>
LOG.error("Upload failed. Server returned: %s" % response)
</pre>
* Failed to send test results to Meego QA Reports reporting tool.

Quality/QA-tools/OTS/DeveloperDocs/ErrorSituations

2010-12-29T14:16:16Z

Tvainio: /* Critical Errors */

= OTS 0.1 Error Situations =

OTS 0.1 has 2 ways of handling error information. The testrun has a specific "set_error_info() and set_error_code()" mechanism for critical errors. This is the main mechanism. In addition to that error messages are printed to http log. The error info mechanism is a legacy thing from the old days we didn't have global http logging or amqp.

In OTS 0.8 we have the same http logging. The set_error_info() stuff is gone but it is replaced by exceptions that can be sent over the wire with amqp.

== Critical Errors ==

These cause the testrun to end with result ERROR

=== /ots/common/results/resultjudge.py:169: ===
<pre>
self.testrun.set_error_info("No test packages defined nor found.")
</pre>
* Raised if user didn't specify test packages and conductor couldn't found any.
* This should practically never happen. Conductor should raise other errors before this if image does not contain packages.

=== /ots/common/results/resultjudge.py:179: ===
<pre>
self.testrun.set_error_info("Results missing from: %s" \
</pre>
* Raised if server didn't receive results.xml for all testpackages this.
* This can happen for example if global timeout happens. (Conductor detects timeouts and raises Timeout error so this shouldn't happen anymore?)

=== /ots/server/testrun_host/testrun_host.py:252: ===
<pre>
try:
return ots_config.default_options[sw_product]
except (KeyError, ValueError):
raise Exception("Unknown sw_product %s" % sw_product)
</pre>
* Raised if user tries to use sw product that is not defined in the config file
* '''System test: log_tests.TestErrorConditions.test_non_existing_devicegroup'''

=== /ots/server/testrun_host/testrun_host.py:392: ===
<pre>
for package in test_packages:
if not (package.endswith("-tests")\
or package.endswith("-test")\
or package.endswith("-benchmark")):
invalid_packages.append(package)
</pre>
* error_msg = "Invalid testpackage(s): %s" % invalid_packages
* This is raised if user input contains test packages that do not end -test, -tests, -benchmark
* '''System test: log_tests.TestErrorConditions.test_bad_testpackage_names

=== /ots/server/testrun_host/testrun_host.py:410: ===
<pre>
if not (self.testrun.get_rootstrap_url() \
or self.testrun.get_image_url()):
</pre>
* error_msg = "No image url or rootstrap url defined."
* Raised if image url is not defined in input. (rootstrap is deprecated!)
* '''System test: log_tests.TestErrorConditions.test_no_image_url

=== /ots/server/testrun_host/testrun_host.py:420: ===
<pre>
if not self.testrun.get_image_url() and \
self.testrun.get_host_testpackages():

error_msg = "Image url missing. Executing host based tests "+\
"requires image url."
</pre>
* Deprecated! Image url is mandatory always so this will never happen. The previous error will be raised first.

=== /ots/server/testrun_host/testrun_host.py:443: ===
<pre>
valid_values = ['default', 'perpackage']
for model in custom_models:
valid_values.append(model[0])
model = self.testrun.get_option('distribution_model')

if model not in valid_values:
</pre>
* error_msg = "Invalid distribution model: %s" % model
* Raised if user gives unknown distribution model parameter. (Extension points can introduce custom distribution models in addition to the default values shown.)
* '''System test: log_tests.TestErrorConditions.test_bad_distribution_model

=== /ots/server/testrun_host/testrun_host.py:451: ===
<pre>
if model == 'perpackage' and len(self.testrun.get_testpackages()) == 0:
error_msg = "Test packages must be defined for specified "\
+"distribution model '%s'" % model

</pre>
* Raised if user tries to use perpackage distribution model but does not explicitly specify testpackages.

=== /ots/server/conductorengine/conductorengine.py:168: ===
<pre>
except OtsQueueDoesNotExistError:
error_info = "Queue '%s' does not exist" \
% (self._routing_key)
</pre>

* Raised if requested worker queue is not available in amqp server. (No suitable workers have ever been added to the system)
* '''System test: log_tests.TestErrorConditions.test_non_existing_devicegroup'''

=== /ots/server/conductorengine/conductorengine.py:176: ===
<pre>

error_info = ("Server side timeout. (Worker went offline during "+\
"testrun or some tasks were not started in time)")
</pre>

* Raised if server side global timeout is hit. See timeouts section in wiki for more details.
* This should be very rare. (but might still happen)

=== /ots/server/conductorengine/conductorengine.py:182: ===
<pre>
except OtsConnectionError:
error_info = "A connectivity issue was encountered"
</pre>
* Raised if amqp connection fails
* Should be very rare

=== /ots/server/conductorengine/conductorengine.py:190: ===
<pre>
except OtsQueueTimeoutError, exception:
error_info = ("The job was not started within %s minutes. "+\
"A worker in that devicegroup may be down or there may be "+\
"exceptional demand") % str(exception.timeout_length / 60)
</pre>
* Queue timeout. Raised if a worker does not take the task from the queue in time.

=== /ots/server/conductorengine/conductorengine.py:196: ===
<pre>
except Exception:
error_info = "A miscellaneous error was encountered"

</pre>
* Generic ots distributor error. This is bad!

=== /ots/worker/conductor/executor.py:277: ===
<pre>
def _test_execution_error_handler(self, error_info, error_code):
"""
Handler for testrun timed out errors and ConductorError exceptions
that are raised during test execution.
"""
self.log.error("Test execution error: %s " % error_info)
if not self.stand_alone:
self.responseclient.set_error(error_info, error_code)
</pre>
* Generic conductor test execution & timeout handler.
* More detailed analysis is part of ots 0.9. For 0.8 just make sure these are sent to server and communicated to user properly.
* '''System test: log_tests.TestErrorConditions.test_timeout'''

=== /ots/worker/conductor/conductor.py:371: ===
<pre>
except ConductorError, exc:
log.error("%s (ots error code: %s)" % (exc.error_info, exc.error_code))
if not stand_alone:
responseclient.set_error(exc.error_info, exc.error_code)
</pre>
* Generic conductor exception handler.
* More detailed analysis is part of ots 0.9. For 0.8 just make sure these are sent to server and communicated to user properly.

=== /ots/worker/conductor/conductor.py:378: ===
<pre>
except:
log.error("Unknown error when trying to execute tests.")
log.error("Traceback follows:", exc_info = True)
if not stand_alone:
responseclient.set_error(str(sys.exc_info()[1]), "999")
</pre>
* Conductor last resport try-except.

=== /ots.worker/ots/worker/task_broker.py:206: ===
<pre>
try:
self._dispatch(command)

except (CommandFailed):
LOGGER.error("Process failed")
error_info = "Task execution failed"
</pre>
* Raised if the task command process (=conductor) returns something else than 0.

== Noncritical Errors ==

These produce an error message to log but do not affect the testrun result. These should be warnings.

=== /ots.server/ots/server/testrun_host/testrun_host.py:185: ===
<pre>
try:
if not ots_config.debug_mode:
for text, url in self.testrun.get_result_links():
self.input_plugin.store_url(url, text)
except:
self.log.error("Publishing result url failed. "\
"Traceback follows:", exc_info = True)
</pre>
* OTS tries to publish the result url with input plugin and fails.

=== /ots.server/ots/server/testrun_host/testrun_host.py:192: ===
<pre>
try:
if self.testrun.email_enabled() and self.emailbackend:
self.emailbackend.send_mail()
self.log.error("Sending result email failed. Traceback follows:",
</pre>
* OTS fails to send result email

=== /ots.server/ots/server/testrun_host/testrun_host.py:269: ===
<pre>
self.log.info("Publishing testrun log link")
try:
if not ots_config.debug_mode:
self.input_plugin.store_url(url, text)
except:
self.log.error("Publishing testrun log link failed. "\
"Traceback follows:", exc_info = True)

</pre>
* Another input plugin warning

=== /ots.server/ots/server/testrun_host/testrun_host.py:461: ===
<pre>
try:
target_packages = self.input_plugin.get_changed_packages(build_id)
except:
self.log.error("Getting target packages failed. "\
"Traceback follows:", exc_info = True)
target_packages = []
</pre>
* OTS tries to fetch information about the target packages changed in the build request and fails.

=== /ots.server/ots/server/email_backend/mailmessage.py:145: ===
<pre>
try:
attach_as_zip_file(msg, result_files, unique_id)
except:
LOG.error("Something went wrong when creating email attachment. "\
"Traceback follows:", exc_info = True)
</pre>
* Creating the attachment zip package failed.

=== /ots.server/ots/server/email_backend/email_backend.py:92: ===
<pre>
if not (type(to_address_list) == type(list()) \
or type(to_address_list) == type("")):
self.log.error("Error in sending mail: expected list or string in "\
"to_address_list: %s " % str(to_address_list))
return
</pre>
* Bad email address input

=== /ots.server/ots/server/email_backend/email_backend.py:121: ===
<pre>
self.log.error("Error in sending mail to following addresses:")
self.log.error(str(failed_addresses))
</pre>

=== /ots.qareports_plugin/ots/qareports_plugin/qareports_client.py:60: ===
<pre>
LOG.error("Upload failed. Server returned: %s" % response)
</pre>
* Failed to send test results to Meego QA Reports reporting tool.

Quality/QA-tools/OTS/DeveloperDocs/ErrorSituations

2010-12-29T14:08:52Z

Tvainio: /* Critical Errors */

= OTS 0.1 Error Situations =

OTS 0.1 has 2 ways of handling error information. The testrun has a specific "set_error_info() and set_error_code()" mechanism for critical errors. This is the main mechanism. In addition to that error messages are printed to http log. The error info mechanism is a legacy thing from the old days we didn't have global http logging or amqp.

In OTS 0.8 we have the same http logging. The set_error_info() stuff is gone but it is replaced by exceptions that can be sent over the wire with amqp.

== Critical Errors ==

These cause the testrun to end with result ERROR

=== /ots/common/results/resultjudge.py:169: ===
<pre>
self.testrun.set_error_info("No test packages defined nor found.")
</pre>
* Raised if user didn't specify test packages and conductor couldn't found any.
* This should practically never happen. Conductor should raise other errors before this if image does not contain packages.

=== /ots/common/results/resultjudge.py:179: ===
<pre>
self.testrun.set_error_info("Results missing from: %s" \
</pre>
* Raised if server didn't receive results.xml for all testpackages this.
* This can happen for example if global timeout happens. (Conductor detects timeouts and raises Timeout error so this shouldn't happen anymore?)

=== /ots/server/testrun_host/testrun_host.py:252: ===
<pre>
try:
return ots_config.default_options[sw_product]
except (KeyError, ValueError):
raise Exception("Unknown sw_product %s" % sw_product)
</pre>
* Raised if user tries to use sw product that is not defined in the config file
* '''System test: log_tests.TestErrorConditions.test_non_existing_devicegroup'''

=== /ots/server/testrun_host/testrun_host.py:392: ===
<pre>
for package in test_packages:
if not (package.endswith("-tests")\
or package.endswith("-test")\
or package.endswith("-benchmark")):
invalid_packages.append(package)
</pre>
* error_msg = "Invalid testpackage(s): %s" % invalid_packages
* This is raised if user input contains test packages that do not end -test, -tests, -benchmark
* '''System test: log_tests.TestErrorConditions.test_bad_testpackage_names

=== /ots/server/testrun_host/testrun_host.py:410: ===
<pre>
if not (self.testrun.get_rootstrap_url() \
or self.testrun.get_image_url()):
</pre>
* error_msg = "No image url or rootstrap url defined."
* Raised if image url is not defined in input. (rootstrap is deprecated!)
* '''System test: log_tests.TestErrorConditions.test_no_image_url

=== /ots/server/testrun_host/testrun_host.py:420: ===
<pre>
if not self.testrun.get_image_url() and \
self.testrun.get_host_testpackages():

error_msg = "Image url missing. Executing host based tests "+\
"requires image url."
</pre>
* Deprecated! Image url is mandatory always so this will never happen. The previous error will be raised first.

=== /ots/server/testrun_host/testrun_host.py:443: ===
<pre>
valid_values = ['default', 'perpackage']
for model in custom_models:
valid_values.append(model[0])
model = self.testrun.get_option('distribution_model')

if model not in valid_values:
</pre>
* error_msg = "Invalid distribution model: %s" % model
* Raised if user gives unknown distribution model parameter. (Extension points can introduce custom distribution models in addition to the default values shown.)

=== /ots/server/testrun_host/testrun_host.py:451: ===
<pre>
if model == 'perpackage' and len(self.testrun.get_testpackages()) == 0:
error_msg = "Test packages must be defined for specified "\
+"distribution model '%s'" % model

</pre>
* Raised if user tries to use perpackage distribution model but does not explicitly specify testpackages.

=== /ots/server/conductorengine/conductorengine.py:168: ===
<pre>
except OtsQueueDoesNotExistError:
error_info = "Queue '%s' does not exist" \
% (self._routing_key)
</pre>

* Raised if requested worker queue is not available in amqp server. (No suitable workers have ever been added to the system)
* '''System test: log_tests.TestErrorConditions.test_non_existing_devicegroup'''

=== /ots/server/conductorengine/conductorengine.py:176: ===
<pre>

error_info = ("Server side timeout. (Worker went offline during "+\
"testrun or some tasks were not started in time)")
</pre>

* Raised if server side global timeout is hit. See timeouts section in wiki for more details.
* This should be very rare. (but might still happen)

=== /ots/server/conductorengine/conductorengine.py:182: ===
<pre>
except OtsConnectionError:
error_info = "A connectivity issue was encountered"
</pre>
* Raised if amqp connection fails
* Should be very rare

=== /ots/server/conductorengine/conductorengine.py:190: ===
<pre>
except OtsQueueTimeoutError, exception:
error_info = ("The job was not started within %s minutes. "+\
"A worker in that devicegroup may be down or there may be "+\
"exceptional demand") % str(exception.timeout_length / 60)
</pre>
* Queue timeout. Raised if a worker does not take the task from the queue in time.

=== /ots/server/conductorengine/conductorengine.py:196: ===
<pre>
except Exception:
error_info = "A miscellaneous error was encountered"

</pre>
* Generic ots distributor error. This is bad!

=== /ots/worker/conductor/executor.py:277: ===
<pre>
def _test_execution_error_handler(self, error_info, error_code):
"""
Handler for testrun timed out errors and ConductorError exceptions
that are raised during test execution.
"""
self.log.error("Test execution error: %s " % error_info)
if not self.stand_alone:
self.responseclient.set_error(error_info, error_code)
</pre>
* Generic conductor test execution & timeout handler.
* More detailed analysis is part of ots 0.9. For 0.8 just make sure these are sent to server and communicated to user properly.
* '''System test: log_tests.TestErrorConditions.test_timeout'''

=== /ots/worker/conductor/conductor.py:371: ===
<pre>
except ConductorError, exc:
log.error("%s (ots error code: %s)" % (exc.error_info, exc.error_code))
if not stand_alone:
responseclient.set_error(exc.error_info, exc.error_code)
</pre>
* Generic conductor exception handler.
* More detailed analysis is part of ots 0.9. For 0.8 just make sure these are sent to server and communicated to user properly.

=== /ots/worker/conductor/conductor.py:378: ===
<pre>
except:
log.error("Unknown error when trying to execute tests.")
log.error("Traceback follows:", exc_info = True)
if not stand_alone:
responseclient.set_error(str(sys.exc_info()[1]), "999")
</pre>
* Conductor last resport try-except.

=== /ots.worker/ots/worker/task_broker.py:206: ===
<pre>
try:
self._dispatch(command)

except (CommandFailed):
LOGGER.error("Process failed")
error_info = "Task execution failed"
</pre>
* Raised if the task command process (=conductor) returns something else than 0.

== Noncritical Errors ==

These produce an error message to log but do not affect the testrun result. These should be warnings.

=== /ots.server/ots/server/testrun_host/testrun_host.py:185: ===
<pre>
try:
if not ots_config.debug_mode:
for text, url in self.testrun.get_result_links():
self.input_plugin.store_url(url, text)
except:
self.log.error("Publishing result url failed. "\
"Traceback follows:", exc_info = True)
</pre>
* OTS tries to publish the result url with input plugin and fails.

=== /ots.server/ots/server/testrun_host/testrun_host.py:192: ===
<pre>
try:
if self.testrun.email_enabled() and self.emailbackend:
self.emailbackend.send_mail()
self.log.error("Sending result email failed. Traceback follows:",
</pre>
* OTS fails to send result email

=== /ots.server/ots/server/testrun_host/testrun_host.py:269: ===
<pre>
self.log.info("Publishing testrun log link")
try:
if not ots_config.debug_mode:
self.input_plugin.store_url(url, text)
except:
self.log.error("Publishing testrun log link failed. "\
"Traceback follows:", exc_info = True)

</pre>
* Another input plugin warning

=== /ots.server/ots/server/testrun_host/testrun_host.py:461: ===
<pre>
try:
target_packages = self.input_plugin.get_changed_packages(build_id)
except:
self.log.error("Getting target packages failed. "\
"Traceback follows:", exc_info = True)
target_packages = []
</pre>
* OTS tries to fetch information about the target packages changed in the build request and fails.

=== /ots.server/ots/server/email_backend/mailmessage.py:145: ===
<pre>
try:
attach_as_zip_file(msg, result_files, unique_id)
except:
LOG.error("Something went wrong when creating email attachment. "\
"Traceback follows:", exc_info = True)
</pre>
* Creating the attachment zip package failed.

=== /ots.server/ots/server/email_backend/email_backend.py:92: ===
<pre>
if not (type(to_address_list) == type(list()) \
or type(to_address_list) == type("")):
self.log.error("Error in sending mail: expected list or string in "\
"to_address_list: %s " % str(to_address_list))
return
</pre>
* Bad email address input

=== /ots.server/ots/server/email_backend/email_backend.py:121: ===
<pre>
self.log.error("Error in sending mail to following addresses:")
self.log.error(str(failed_addresses))
</pre>

=== /ots.qareports_plugin/ots/qareports_plugin/qareports_client.py:60: ===
<pre>
LOG.error("Upload failed. Server returned: %s" % response)
</pre>
* Failed to send test results to Meego QA Reports reporting tool.

Quality/QA-tools/OTS/DeveloperDocs/ErrorSituations

2010-12-29T14:01:03Z

Tvainio: /* Critical Errors */

= OTS 0.1 Error Situations =

OTS 0.1 has 2 ways of handling error information. The testrun has a specific "set_error_info() and set_error_code()" mechanism for critical errors. This is the main mechanism. In addition to that error messages are printed to http log. The error info mechanism is a legacy thing from the old days we didn't have global http logging or amqp.

In OTS 0.8 we have the same http logging. The set_error_info() stuff is gone but it is replaced by exceptions that can be sent over the wire with amqp.

== Critical Errors ==

These cause the testrun to end with result ERROR

=== /ots/common/results/resultjudge.py:169: ===
<pre>
self.testrun.set_error_info("No test packages defined nor found.")
</pre>
* Raised if user didn't specify test packages and conductor couldn't found any.
* This should practically never happen. Conductor should raise other errors before this if image does not contain packages.

=== /ots/common/results/resultjudge.py:179: ===
<pre>
self.testrun.set_error_info("Results missing from: %s" \
</pre>
* Raised if server didn't receive results.xml for all testpackages this.
* This can happen for example if global timeout happens. (Conductor detects timeouts and raises Timeout error so this shouldn't happen anymore?)

=== /ots/server/testrun_host/testrun_host.py:252: ===
<pre>
try:
return ots_config.default_options[sw_product]
except (KeyError, ValueError):
raise Exception("Unknown sw_product %s" % sw_product)
</pre>
* Raised if user tries to use sw product that is not defined in the config file
* '''System test: log_tests.TestErrorConditions.test_non_existing_devicegroup'''

=== /ots/server/testrun_host/testrun_host.py:392: ===
<pre>
for package in test_packages:
if not (package.endswith("-tests")\
or package.endswith("-test")\
or package.endswith("-benchmark")):
invalid_packages.append(package)
</pre>
* error_msg = "Invalid testpackage(s): %s" % invalid_packages
* This is raised if user input contains test packages that do not end -test, -tests, -benchmark
* '''System test: log_tests.TestErrorConditions.test_bad_testpackage_names

=== /ots/server/testrun_host/testrun_host.py:410: ===
<pre>
if not (self.testrun.get_rootstrap_url() \
or self.testrun.get_image_url()):
</pre>
* error_msg = "No image url or rootstrap url defined."
* Raised if image url is not defined in input. (rootstrap is deprecated!)

=== /ots/server/testrun_host/testrun_host.py:420: ===
<pre>
if not self.testrun.get_image_url() and \
self.testrun.get_host_testpackages():

error_msg = "Image url missing. Executing host based tests "+\
"requires image url."
</pre>
* Deprecated! Image url is mandatory always so this will never happen. The previous error will be raised first.

=== /ots/server/testrun_host/testrun_host.py:443: ===
<pre>
valid_values = ['default', 'perpackage']
for model in custom_models:
valid_values.append(model[0])
model = self.testrun.get_option('distribution_model')

if model not in valid_values:
</pre>
* error_msg = "Invalid distribution model: %s" % model
* Raised if user gives unknown distribution model parameter. (Extension points can introduce custom distribution models in addition to the default values shown.)

=== /ots/server/testrun_host/testrun_host.py:451: ===
<pre>
if model == 'perpackage' and len(self.testrun.get_testpackages()) == 0:
error_msg = "Test packages must be defined for specified "\
+"distribution model '%s'" % model

</pre>
* Raised if user tries to use perpackage distribution model but does not explicitly specify testpackages.

=== /ots/server/conductorengine/conductorengine.py:168: ===
<pre>
except OtsQueueDoesNotExistError:
error_info = "Queue '%s' does not exist" \
% (self._routing_key)
</pre>

* Raised if requested worker queue is not available in amqp server. (No suitable workers have ever been added to the system)
* '''System test: log_tests.TestErrorConditions.test_non_existing_devicegroup'''

=== /ots/server/conductorengine/conductorengine.py:176: ===
<pre>

error_info = ("Server side timeout. (Worker went offline during "+\
"testrun or some tasks were not started in time)")
</pre>

* Raised if server side global timeout is hit. See timeouts section in wiki for more details.
* This should be very rare. (but might still happen)

=== /ots/server/conductorengine/conductorengine.py:182: ===
<pre>
except OtsConnectionError:
error_info = "A connectivity issue was encountered"
</pre>
* Raised if amqp connection fails
* Should be very rare

=== /ots/server/conductorengine/conductorengine.py:190: ===
<pre>
except OtsQueueTimeoutError, exception:
error_info = ("The job was not started within %s minutes. "+\
"A worker in that devicegroup may be down or there may be "+\
"exceptional demand") % str(exception.timeout_length / 60)
</pre>
* Queue timeout. Raised if a worker does not take the task from the queue in time.

=== /ots/server/conductorengine/conductorengine.py:196: ===
<pre>
except Exception:
error_info = "A miscellaneous error was encountered"

</pre>
* Generic ots distributor error. This is bad!

=== /ots/worker/conductor/executor.py:277: ===
<pre>
def _test_execution_error_handler(self, error_info, error_code):
"""
Handler for testrun timed out errors and ConductorError exceptions
that are raised during test execution.
"""
self.log.error("Test execution error: %s " % error_info)
if not self.stand_alone:
self.responseclient.set_error(error_info, error_code)
</pre>
* Generic conductor test execution & timeout handler.
* More detailed analysis is part of ots 0.9. For 0.8 just make sure these are sent to server and communicated to user properly.
* '''System test: log_tests.TestErrorConditions.test_timeout'''

=== /ots/worker/conductor/conductor.py:371: ===
<pre>
except ConductorError, exc:
log.error("%s (ots error code: %s)" % (exc.error_info, exc.error_code))
if not stand_alone:
responseclient.set_error(exc.error_info, exc.error_code)
</pre>
* Generic conductor exception handler.
* More detailed analysis is part of ots 0.9. For 0.8 just make sure these are sent to server and communicated to user properly.

=== /ots/worker/conductor/conductor.py:378: ===
<pre>
except:
log.error("Unknown error when trying to execute tests.")
log.error("Traceback follows:", exc_info = True)
if not stand_alone:
responseclient.set_error(str(sys.exc_info()[1]), "999")
</pre>
* Conductor last resport try-except.

=== /ots.worker/ots/worker/task_broker.py:206: ===
<pre>
try:
self._dispatch(command)

except (CommandFailed):
LOGGER.error("Process failed")
error_info = "Task execution failed"
</pre>
* Raised if the task command process (=conductor) returns something else than 0.

== Noncritical Errors ==

These produce an error message to log but do not affect the testrun result. These should be warnings.

=== /ots.server/ots/server/testrun_host/testrun_host.py:185: ===
<pre>
try:
if not ots_config.debug_mode:
for text, url in self.testrun.get_result_links():
self.input_plugin.store_url(url, text)
except:
self.log.error("Publishing result url failed. "\
"Traceback follows:", exc_info = True)
</pre>
* OTS tries to publish the result url with input plugin and fails.

=== /ots.server/ots/server/testrun_host/testrun_host.py:192: ===
<pre>
try:
if self.testrun.email_enabled() and self.emailbackend:
self.emailbackend.send_mail()
self.log.error("Sending result email failed. Traceback follows:",
</pre>
* OTS fails to send result email

=== /ots.server/ots/server/testrun_host/testrun_host.py:269: ===
<pre>
self.log.info("Publishing testrun log link")
try:
if not ots_config.debug_mode:
self.input_plugin.store_url(url, text)
except:
self.log.error("Publishing testrun log link failed. "\
"Traceback follows:", exc_info = True)

</pre>
* Another input plugin warning

=== /ots.server/ots/server/testrun_host/testrun_host.py:461: ===
<pre>
try:
target_packages = self.input_plugin.get_changed_packages(build_id)
except:
self.log.error("Getting target packages failed. "\
"Traceback follows:", exc_info = True)
target_packages = []
</pre>
* OTS tries to fetch information about the target packages changed in the build request and fails.

=== /ots.server/ots/server/email_backend/mailmessage.py:145: ===
<pre>
try:
attach_as_zip_file(msg, result_files, unique_id)
except:
LOG.error("Something went wrong when creating email attachment. "\
"Traceback follows:", exc_info = True)
</pre>
* Creating the attachment zip package failed.

=== /ots.server/ots/server/email_backend/email_backend.py:92: ===
<pre>
if not (type(to_address_list) == type(list()) \
or type(to_address_list) == type("")):
self.log.error("Error in sending mail: expected list or string in "\
"to_address_list: %s " % str(to_address_list))
return
</pre>
* Bad email address input

=== /ots.server/ots/server/email_backend/email_backend.py:121: ===
<pre>
self.log.error("Error in sending mail to following addresses:")
self.log.error(str(failed_addresses))
</pre>

=== /ots.qareports_plugin/ots/qareports_plugin/qareports_client.py:60: ===
<pre>
LOG.error("Upload failed. Server returned: %s" % response)
</pre>
* Failed to send test results to Meego QA Reports reporting tool.

Quality/QA-tools/OTS/UserDocumentation/Installation

2010-12-29T13:53:57Z

Tvainio:

== Installation Instructions for OTS 0.1 ==

These instructions have been tested with Ubuntu 10.04.

=== Network Setup ===

Using static IP addresses on all OTS machines is strongly recommended. It is possible to setup the system also with dynamic IP addresses but it might cause problems every now and then because IP's and hostnames change.

All OTS machines need to be able to access other OTS machines based on their hostname.

=== Server ===

==== prerequisites ====

* An smtp server is needed for OTS to be able to send result emails.
** You can use the smtp server provided by your ISP or organization
** Or you can setup your own. for example [http://my.opera.com/Contrid/blog/show.dml/478684 postfix]

==== Basic Setup ====

* Download and install RabbitMQ server from http://www.rabbitmq.com/server.html

* Install dependencies from ubuntu repository:
sudo apt-get update
sudo apt-get install python-setuptools python-amqplib python-django

* Install rest of the dependencies with easy_install:
sudo easy_install minixsv

* Get source code from http://meego.gitorious.org/meego-quality-assurance/ots/ (latest tag)

* Install ots.common:
cd ots/ots.common/
sudo python setup.py install

* Install ots.server:
cd ots/ots.server/
sudo python setup.py install

* Install ots.tools: (Required only for triggering testruns easily from command line over xmlrpc)
cd ots/ots.tools/
sudo python setup.py install

* Download and install test definition schema files from http://meego.gitorious.org/meego-quality-assurance/test-definition

* Create a directory for testrun logs and make sure the user running ots.server has write permissions to it:
sudo mkdir /var/log/ots

* At this point it should be possible to trigger testruns from python shell if you have a worker already set up and configured to use this server. Here's an example ipython session:
<pre>
In [1]: from ots.server.xmlrpc.public import request_sync

In [4]: program = "example_sw_product"

In [5]: request = "5"

In [6]: notify_list = ['tvainio@localhost']

In [7]: options = dict()

In [8]: options['image'] = "http://somewhere/some_image.bin"

In [9]: options['email'] = "off"

In [13]: options["device"] = "devicegroup:default" # Make this match the worker

In [14]: request_sync(program, request, notify_list, options)
Out[14]: 'FAIL'
</pre>

The result will be PASS/FAIL/ERROR. ERROR means that something went wrong in server side and testrun was not even started. FAIL means failed testrun and PASS means that everything went fine and all tests passed. Testrun logs with more information about the possible errors are available in /var/log/ots/

==== Configuring the Server ====

The previous example was not very useful. To get OTS server running as a real server you have two options. A simple xmlrpc server or a more advanced django based server. First thing you need to do is setup a custom config file.

===== Custom Config File =====

Ots server tries to import a configuration module ots_extensions.ots_config. If it does not succeed it uses the default from ots.server.testrun_host.default_ots_config. You can set up your own ots_config by following these instructions:

* Create a directory for your extensions:
mkdir my_ots_extensions

* Add it to your python path
export PYTHONPATH=/full/path/to/my_ots_extensions/:$PYTHONPATH

* Create custom ots_extensions python package:
mkdir my_ots_extensions/ots_extensions
touch my_ots_extensions/ots_extensions/__init__.py

* Copy the default config file from ots/ots.server/ots/server/testrun_host/default_ots_config.py to my_ots_extensions/ots_extensions/ots_config.py

* Modify the file to suit your needs. Check at least the following settings:
** Email settings (The smtp server OTS uses to send testrun results)
** xmlrpc server config (if you are setting the simple xmlrpc server setup. More info below.)
** The default options for your sw product:
You can define default values for all options here. The example_sw_product contains and example how to define options. You cannot use an sw_product if it does not contain the default options in ots_config. Here's a small example. I'm developing a Meego based coffeemaker and creating custom software images for it. To automate the testing I create an sw_product called "my_meego_variant":
'''TODO: Document all options'''
<pre>
#
# Defaults for my_meego_variant
#

# Lets start with the global_defaults
default_options["my_meego_variant"] = global_defaults.copy()

# The images work only on meego coffeemakers so the default devicegroup will be "meego_coffeemaker"
default_options["my_meego_variant"]['device'] = {'devicegroup':'meego_coffeemaker'}

# Coffeemaker needs to be fast so we define a 3 minutes global timeout for the test execution.
default_options["my_meego_variant"]['timeout'] = 3
</pre>

===== Extension Points =====

OTS uses plugins to do various things during the testrun. You can write your own plugins to add more functionality by implementing a module ots_extensions.extension_points. An example file can be found from ots/ots.server/ots/server/testrun_host/example_extension_points.py

==== Simple XMLRPC Server ====

* After you have implemented your custom config file and defined suitable xmlrpc_host and xmlrpc_port values you can start the simple xmlrpc server:
ots_server

* After that you can start testruns on your ots_server by calling the request_sync() method over xmlrpc just like we did locally in the example above. There's also a command line tool called ots_trigger in ots.tools package.

* A testrun can be triggered from command line with ots_trigger:

ots_trigger -s your_xmlrpc_host:your_xmlrpc_port -b 333 -i url_to_dummyimage -p my_meego_variant -e your_email_address

The result should be FAIL and if your email server settings are correct you should also receive an email.

* If we look into the new testrun log file in /var/log/ots/ we should find something like this:

<pre>
2010-09-24 15:48:21,188 conductorengine ERROR Device group 'meego_coffeemaker' does not exist
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/ots.server-0.1dev-py2.6.egg/ots/server/conductorengine/conductorengine.py", line 150, in execute
self._taskrunner.run()
File "/usr/local/lib/python2.6/dist-packages/ots.server-0.1dev-py2.6.egg/ots/server/distributor/taskrunner.py", line 417, in run
(self._routing_key))
OtsQueueDoesNotExistError: No queue for meego_coffeemaker
</pre>

* Next step is to set up an OTS worker into the devicegroup meego_coffeemaker. Instructions can be found below.

==== Django Based Server ====

With Django and Apache it's possible to set up a more advanced ots server.

There's for example a global logger application available in ots.server.logger. By setting up the logger it is possible to get log messages for a specific testrun from all parts of the system (ots server, ots conductor and testrunner-lite). The logs are stored into database and can be accessed with a web UI. It's also easy to implement a result db and web views for the test results.

If you are not familiar with Django you can study the basics from the tutorial: http://docs.djangoproject.com/en/1.2/intro/tutorial01/

These instructions assume that you have already done the setup described in: [[http://wiki.meego.com/Quality/QA-tools/OTS/UserDocumentation/Installation#Server]]

There is an example django project in ots.server.django_ots. It is possible to use that directly but in most cases you should create your own django project to be able to modify the settings.

===== Installation Using the Default Django Project in ots.server.django_ots =====

* Install required packages:
<pre>
sudo apt-get update
sudo apt-get install python-django python-setuptools rabbitmq-server python-amqplib
</pre>

* Install django-xmlrpc from https://launchpad.net/django-xmlrpc to somewhere in your pythonpath

* After that you should be able to execute tests for the ots django applications:
<pre>
python ots/server/django_ots/manage.py test logger testrun_db
</pre>

* Create the database. The default ots django project uses sqlite. The DB file is located in /opt/ots.sqlite.
<pre>
python ots/server/django_ots/manage.py syncdb
</pre>

The script asks about creating a superuser for the Django's auth system. OTS does not use the auth system at the moment but it might be useful to create the account if you start writing custom django applications or want to use django admin interface at some point.

If you get errors like "unable to open database file" make sure your user has write permissions to /opt/

* Install apache and mod-python: (OTS requires the prefork version of apache)

sudo apt-get install apache2-mpm-prefork libapache2-mod-python

* Setup your custom ots_config and extension points
** Instructions are available in [[http://wiki.meego.com/Quality/QA-tools/OTS/UserDocumentation/Installation#Configuring_the_Server]]
** Use ots/server/django_ots/extension_points.py as the basis of your extension points. It takes the testrun_db into use.
** Changes http_loggin_enabled to True in your ots_config.py if you want to have global testrun logs:
http_logging_enabled = True

* Configure your django_ots to /etc/apache2/apache2.conf by adding the following:

<pre>
<Location "/">
SetHandler python-program
PythonHandler django.core.handlers.modpython
SetEnv DJANGO_SETTINGS_MODULE ots.server.django_ots.settings
PythonDebug On
PythonPath "['ADD HERE THE PATH WHERE YOUR ots_extensions directory with CUSTOM OTS ENTRYPOINTS AND CONFIGFILE CAN BE FOUND'] + sys.path"
</Location>
</pre>

* Open http://localhost/logger/view/ in your web browser. You should see an empty logging message page.
** If you get "Unable to open database file" make sure that the user apache is running on has read and write permissions to the db file (/opt/ots.sqlite).
* Try triggering a testrun with ots_trigger (note the /xmlrpc in server url):
ots_trigger -s localhost/xmlrpc -b 333 -i url_to_dummyimage -p my_meego_variant -e your_email_address

If everything went ok you should see a succesful testrun and logs from OTS server, OTS Worker.conductor and testrunner-lite appearing in the http://localhost/logger/view/ and http://localhost/logger/view/ots/<your_testrun_id>/

If you only see log messages from your server most probably yuor hostnames are not set up correctly. The worker machine needs to be able to access the server by it's host name. Or you can try to play around with the "results_storage_config["host"]" setting in ots_config.py. It is the hostname ots_server sends to worker to use for global logging.

===== Installation Using a Custom Django Project =====

If you want to use different django setup you can create a custom django project. You can use ots/server/django_ots/ as an example. Django documentation contains lots of information about the features and settings: http://www.djangoproject.com/

The most important files are:
* settings.py - DB settings, installed django applications, email etc.
* urls.py - Url mappings. For example the "xmlrpc/" url is defined here.

You also need to modify your apache configuration to point to your own django project instead of the default one. Also note that if you change your code, mod_python does not automatically reload the changed code so you need to restart apache to make sure you are using the right version of your code. If you do a full apache restart all ongoing testruns will fail. If you do a graceful restart (<code>"apache2ctl graceful"</code>)the testruns won't fail but updated code is reloaded only after the testruns have fininshed.

==== Uploading Results to Meego QA Reports Tool ====

Starting from version 0.1.6 OTS can automatically upload results to Meego QA Reports tool.

* Make sure ots.qareports_plugin is installed.
* Modify values in /etc/ots_qareports_plugin.conf to match your setup:

<pre>
# More information about the options is available in
# http://wiki.meego.com/Quality/QA-tools/QAReports/API

[ots.qareports_plugin]
host = "localhost"
url = "/api/import"
auth_token = ""
release_version = "1.0"
target = "Core"
testtype = "Acceptance"
hwproduct = "N900"
</pre>

* Modify "get_resultbackends()" in your custom extension_points.py to take QaReportsBackend into use:

<pre>
from ots.qareports_plugin.qareports_backend import QAReportsBackend

def get_resultbackends(testrun):
"""Returns custom ResultBackends as a list"""
return [QAReportsBackend()]
</pre>
===== Dynamic Report Control =====

Starting from OTS 0.1.7 it's possible to control reporting options dynamically when triggering a testrun with "--options" command line parameter in ots_trigger:
<pre>
ots_trigger --options="qa_hwproduct:<HWPRODUCT> qa_testtype:<TESTTYPE> qa_target:<TARGET> qa_release_version:<RELEASE_VERSION>"
</pre>

=== Worker ===

* Install dependencies:
sudo apt-get update
sudo apt-get install python-setuptools python-amqplib

* Get source code from http://meego.gitorious.org/meego-quality-assurance/ots/ (latest tag)

* Install ots.common:
cd ots/ots.common/
sudo python setup.py install

* Install ots.worker:
cd ots/ots.worker/
sudo python setup.py install

* Configure the worker by editing devicegroup and server address in /etc/ots.ini:
<pre>
###########################################
# REPLACE `fix_me` WITH YOUR DEVICEGROUP
###########################################
[Device]
devicegroup = fix_me # Change this to the devicegroup this worker belongs to

#############################
# Optional device properties
#############################

#devicename = name of the device type # Uncomment these if you want more detailed testrun routing
#deviceid = Specific HW id of the device # See http://wiki.meego.com/Quality/QA-tools/OTS/Routing for details

#############################
# ADVANCED CONFIGURATION
#############################
[Worker]
host = your_server_address # Change this to your ots server address
vhost = /
port = 5672
username = guest
password = guest
log_file = /var/log/ots.log
</pre>

More info about testrun routing is available here: [[Quality/QA-tools/OTS/Routing]]

* Start the worker:

sudo ots_worker

* If everything went ok, you should see something like this:
<pre>
2010-11-08 12:55:10,790 - ots.worker.worker - DEBUG - Initialising the server
2010-11-08 12:55:10,790 - ots.worker.connection - DEBUG - Connecting to RabbitMQ
2010-11-08 12:55:10,894 - amqplib - DEBUG - Start from server, version: 8.0, properties: {u'platform': 'Erlang/OTP', u'product': 'RabbitMQ', u'version': '2.1.0', u'copyright': 'Copyright (C) 2007-2010 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.', u'information': 'Licensed under the MPL. See http://www.rabbitmq.com/'}, mechanisms: ['PLAIN', 'AMQPLAIN'], locales: ['en_US']
2010-11-08 12:55:10,935 - amqplib - DEBUG - Open OK! known_hosts []
2010-11-08 12:55:10,935 - amqplib - DEBUG - using channel_id: 1
2010-11-08 12:55:10,937 - amqplib - DEBUG - Channel open
2010-11-08 12:55:10,938 - ots.worker.worker - DEBUG - Starting the worker. server: 192.168.1.102:5672, device_properties: {'devicegroup': 'meego_coffeemaker'}
2010-11-08 12:55:10,957 - ots.worker.task_broker - INFO - consume on queue: meego_coffeemaker
2010-11-08 12:55:10,958 - ots.worker.task_broker - DEBUG - Starting main loop...
</pre>

* There's also an init script available in ots.worker/debian/python-ots-worker.ots-worker.init to make running the worker easier

* After that the worker should be ready for receiving tasks. To make sure the actual test execution works please see the instructions for ots.worker.conductor: [[Quality/QA-tools/OTS/UserDocumentation/Conductor| Conductor]]

Quality/QA-tools/OTS/DeveloperDocs/ErrorSituations

2010-12-29T13:44:20Z

Tvainio: /* Critical Errors */

= OTS 0.1 Error Situations =

OTS 0.1 has 2 ways of handling error information. The testrun has a specific "set_error_info() and set_error_code()" mechanism for critical errors. This is the main mechanism. In addition to that error messages are printed to http log. The error info mechanism is a legacy thing from the old days we didn't have global http logging or amqp.

In OTS 0.8 we have the same http logging. The set_error_info() stuff is gone but it is replaced by exceptions that can be sent over the wire with amqp.

== Critical Errors ==

These cause the testrun to end with result ERROR

=== /ots/common/results/resultjudge.py:169: ===
<pre>
self.testrun.set_error_info("No test packages defined nor found.")
</pre>
* Raised if user didn't specify test packages and conductor couldn't found any.
* This should practically never happen. Conductor should raise other errors before this if image does not contain packages.

=== /ots/common/results/resultjudge.py:179: ===
<pre>
self.testrun.set_error_info("Results missing from: %s" \
</pre>
* Raised if server didn't receive results.xml for all testpackages this.
* This can happen for example if global timeout happens. (Conductor detects timeouts and raises Timeout error so this shouldn't happen anymore?)

=== /ots/server/testrun_host/testrun_host.py:252: ===
<pre>
try:
return ots_config.default_options[sw_product]
except (KeyError, ValueError):
raise Exception("Unknown sw_product %s" % sw_product)
</pre>
* Raised if user tries to use sw product that is not defined in the config file
* '''System test: log_tests.TestErrorConditions.test_non_existing_devicegroup'''
=== /ots/server/testrun_host/testrun_host.py:392: ===
<pre>
for package in test_packages:
if not (package.endswith("-tests")\
or package.endswith("-test")\
or package.endswith("-benchmark")):
invalid_packages.append(package)
</pre>
* error_msg = "Invalid testpackage(s): %s" % invalid_packages
* This is raised if user input contains test packages that do not end -test, -tests, -benchmark

=== /ots/server/testrun_host/testrun_host.py:410: ===
<pre>
if not (self.testrun.get_rootstrap_url() \
or self.testrun.get_image_url()):
</pre>
* error_msg = "No image url or rootstrap url defined."
* Raised if image url is not defined in input. (rootstrap is deprecated!)

=== /ots/server/testrun_host/testrun_host.py:420: ===
<pre>
if not self.testrun.get_image_url() and \
self.testrun.get_host_testpackages():

error_msg = "Image url missing. Executing host based tests "+\
"requires image url."
</pre>
* Deprecated! Image url is mandatory always so this will never happen. The previous error will be raised first.

=== /ots/server/testrun_host/testrun_host.py:443: ===
<pre>
valid_values = ['default', 'perpackage']
for model in custom_models:
valid_values.append(model[0])
model = self.testrun.get_option('distribution_model')

if model not in valid_values:
</pre>
* error_msg = "Invalid distribution model: %s" % model
* Raised if user gives unknown distribution model parameter. (Extension points can introduce custom distribution models in addition to the default values shown.)

=== /ots/server/testrun_host/testrun_host.py:451: ===
<pre>
if model == 'perpackage' and len(self.testrun.get_testpackages()) == 0:
error_msg = "Test packages must be defined for specified "\
+"distribution model '%s'" % model

</pre>
* Raised if user tries to use perpackage distribution model but does not explicitly specify testpackages.

=== /ots/server/conductorengine/conductorengine.py:168: ===
<pre>
except OtsQueueDoesNotExistError:
error_info = "Queue '%s' does not exist" \
% (self._routing_key)
</pre>

* Raised if requested worker queue is not available in amqp server. (No suitable workers have ever been added to the system)
* '''System test: log_tests.TestErrorConditions.test_non_existing_devicegroup'''

=== /ots/server/conductorengine/conductorengine.py:176: ===
<pre>

error_info = ("Server side timeout. (Worker went offline during "+\
"testrun or some tasks were not started in time)")
</pre>

* Raised if server side global timeout is hit. See timeouts section in wiki for more details.
* This should be very rare. (but might still happen)

=== /ots/server/conductorengine/conductorengine.py:182: ===
<pre>
except OtsConnectionError:
error_info = "A connectivity issue was encountered"
</pre>
* Raised if amqp connection fails
* Should be very rare

=== /ots/server/conductorengine/conductorengine.py:190: ===
<pre>
except OtsQueueTimeoutError, exception:
error_info = ("The job was not started within %s minutes. "+\
"A worker in that devicegroup may be down or there may be "+\
"exceptional demand") % str(exception.timeout_length / 60)
</pre>
* Queue timeout. Raised if a worker does not take the task from the queue in time.

=== /ots/server/conductorengine/conductorengine.py:196: ===
<pre>
except Exception:
error_info = "A miscellaneous error was encountered"

</pre>
* Generic ots distributor error. This is bad!

=== /ots/worker/conductor/executor.py:277: ===
<pre>
def _test_execution_error_handler(self, error_info, error_code):
"""
Handler for testrun timed out errors and ConductorError exceptions
that are raised during test execution.
"""
self.log.error("Test execution error: %s " % error_info)
if not self.stand_alone:
self.responseclient.set_error(error_info, error_code)
</pre>
* Generic conductor test execution & timeout handler.
* More detailed analysis is part of ots 0.9. For 0.8 just make sure these are sent to server and communicated to user properly.
* '''System test: log_tests.TestErrorConditions.test_timeout'''

=== /ots/worker/conductor/conductor.py:371: ===
<pre>
except ConductorError, exc:
log.error("%s (ots error code: %s)" % (exc.error_info, exc.error_code))
if not stand_alone:
responseclient.set_error(exc.error_info, exc.error_code)
</pre>
* Generic conductor exception handler.
* More detailed analysis is part of ots 0.9. For 0.8 just make sure these are sent to server and communicated to user properly.

=== /ots/worker/conductor/conductor.py:378: ===
<pre>
except:
log.error("Unknown error when trying to execute tests.")
log.error("Traceback follows:", exc_info = True)
if not stand_alone:
responseclient.set_error(str(sys.exc_info()[1]), "999")
</pre>
* Conductor last resport try-except.

=== /ots.worker/ots/worker/task_broker.py:206: ===
<pre>
try:
self._dispatch(command)

except (CommandFailed):
LOGGER.error("Process failed")
error_info = "Task execution failed"
</pre>
* Raised if the task command process (=conductor) returns something else than 0.

== Noncritical Errors ==

These produce an error message to log but do not affect the testrun result. These should be warnings.

=== /ots.server/ots/server/testrun_host/testrun_host.py:185: ===
<pre>
try:
if not ots_config.debug_mode:
for text, url in self.testrun.get_result_links():
self.input_plugin.store_url(url, text)
except:
self.log.error("Publishing result url failed. "\
"Traceback follows:", exc_info = True)
</pre>
* OTS tries to publish the result url with input plugin and fails.

=== /ots.server/ots/server/testrun_host/testrun_host.py:192: ===
<pre>
try:
if self.testrun.email_enabled() and self.emailbackend:
self.emailbackend.send_mail()
self.log.error("Sending result email failed. Traceback follows:",
</pre>
* OTS fails to send result email

=== /ots.server/ots/server/testrun_host/testrun_host.py:269: ===
<pre>
self.log.info("Publishing testrun log link")
try:
if not ots_config.debug_mode:
self.input_plugin.store_url(url, text)
except:
self.log.error("Publishing testrun log link failed. "\
"Traceback follows:", exc_info = True)

</pre>
* Another input plugin warning

=== /ots.server/ots/server/testrun_host/testrun_host.py:461: ===
<pre>
try:
target_packages = self.input_plugin.get_changed_packages(build_id)
except:
self.log.error("Getting target packages failed. "\
"Traceback follows:", exc_info = True)
target_packages = []
</pre>
* OTS tries to fetch information about the target packages changed in the build request and fails.

=== /ots.server/ots/server/email_backend/mailmessage.py:145: ===
<pre>
try:
attach_as_zip_file(msg, result_files, unique_id)
except:
LOG.error("Something went wrong when creating email attachment. "\
"Traceback follows:", exc_info = True)
</pre>
* Creating the attachment zip package failed.

=== /ots.server/ots/server/email_backend/email_backend.py:92: ===
<pre>
if not (type(to_address_list) == type(list()) \
or type(to_address_list) == type("")):
self.log.error("Error in sending mail: expected list or string in "\
"to_address_list: %s " % str(to_address_list))
return
</pre>
* Bad email address input

=== /ots.server/ots/server/email_backend/email_backend.py:121: ===
<pre>
self.log.error("Error in sending mail to following addresses:")
self.log.error(str(failed_addresses))
</pre>

=== /ots.qareports_plugin/ots/qareports_plugin/qareports_client.py:60: ===
<pre>
LOG.error("Upload failed. Server returned: %s" % response)
</pre>
* Failed to send test results to Meego QA Reports reporting tool.

Quality/QA-tools/OTS/DeveloperDocs/TypicalUsage

2010-12-29T09:59:06Z

Tvainio: /* Users */

== Typical OTS Usage ==

This page tries to describe how end users typically use OTS. The motivation for this page is to gain common understanding about the problem domain.

=== Users ===

Most of the users can be grouped to one of these 3 groups.

==== Basic User ====

Typically a test engineer or mobile SW developer.

* "I want to execute my tests on real target device."

===== Expected Knowledge =====

* Knows test definition and test packaging
* Knows his/her particular application/test area in detailed level
* Knows what SW product he/she is developing
* Understands OTS testrun in a rough level. (Download and flash the image, execute tests over ssh.)
* Knows some of the OTS input parameters. For example email address, timeout, SW Product

* Does not know python
* Does not know details about the particular OTS system setup (devicegroups or workers available)

===== OTS Usage =====

In most cases does not directly interact with OTS. Uses OTS service provided by others.

* Testrun from build machine
** User makes changes to Testpackage or target SW package.
** User creates image with build machine.
** Build machine triggers automatic testrun for the new image containing new version of user's software or tests.
** OTS uses default parameters for the particular SW Product. (User does not define devicegroup or other parameters. User knows only the SW Product.)
** User follows testrun process from html log page.
** User receives test results as an email. They are also available in a reporting tool.
** If there was errors in the testrun, the testrun log gives clear explanation what went wrong. (Flashing failed, Testpackage missing from the image, Unknown SW Product etc.)

* Nightly testruns
** User makes changes to Testpackage or target SW package.
** Build machine automatically builds a nightly image with new changes included.
** Build machine triggers automatic testrun for the new version of user's packages.
** OTS parameters for nightly build are defined by an advanced user responsible for the nightly testruns. (Basic user does not define devicegroup or other parameters)
** User receives test results as an email. They are also available in a reporting tool.
** If there was errors in the testrun, the testrun log gives clear explanation what went wrong. (Flashing failed, Testpackage missing from the image, Unknown SW Product etc.)

==== Advanced User / Worker Maintainer ====

Typically a test engineer or mobile SW developer. Has own device and worker pc connected to the OTS system. Motivation for own worker might be that the test cases require a specific environment or the user just wants to see what's happening while tests are executed.

* "I want to execute my tests in my own environment."

===== Expected Knowledge =====

In addition to the basic user knowledge:

* Knows how to set up and configure OTS worker.
* Knows the device properties and how to use the own device for a testrun

===== OTS Usage =====

Uses OTS service provided by others.

In addition to the basic user use cases, defines the device group (or device parameters) in the test run parameters so that testrun is routed to the specific worker.

* Local conductor run
** Executes conductor from command line with custom built image.
** Follows testrun from conductor output & log.
** Result files are available in file system. (~/conductor/)
** If errors or failures happen, the logs/output gives enough data for debugging

==== OTS System Maintainer ====

A system administrator is responsible for the OTS server and the whole testing system instance.

* "I want to provide a common test environment for my peers."

===== Expected Knowledge =====

In addition to the Advanced User / Worker Maintainer:

* Knows how to set up and configure OTS server
* Knows all OTS input parameters and how to trigger runs from command line
* Knows the common device pool setup
* Knows the default settings for each SW Product
* Knows testrun process in detailed level. Is able to debug problems in the system.
* Some python knowledge. Can read tracebacks and understands some of the common exceptions (import errors etc.)
* Is able to make detailed bug reports

===== OTS Usage =====

Provides OTS service for others.

* Setup
** Installs ots server and common workers
** Defines supported SW products for the particular OTS instance.
** Defines device parameters for the common devices.
** Defines default values for supported SW products so that the basic user does not need to know any details about the system.
*** Basic users can just trigger a run and everything goes right thanks to the default values.
*** For example the test run will be routed to a suitable device because default devicegroup is defined in the config by the system maintainer.
*** The testrun result is reported correctly in the reporting tool because the system maintainer has defined correct reporting category in the config file and that will be automatically used when communicating with reporting tool.

* Debug a testrun
** A testrun triggered by a user ends with errors. User is not able to solve the problem so system maintainer is asked for help.
** System maintainer studies the logs and debugs the problem
*** System maintainer can re-trigger the testrun for debugging purposes by checking the exact input parameters from the log.
** System maintainer files bugs against OTS SW component if needed
** System maintainer fixes problems in the OTS setup if needed
** System maintainer instructs the user in OTS usage if needed

Quality/QA-tools/OTS/Logging

2010-12-29T09:22:59Z

Tvainio: /* Log Levels */

= Logging =

== Log Levels ==

Different log levels are targeted for different user groups. See [[Quality/QA-tools/OTS/DeveloperDocs/TypicalUsage| Typical OTS users and usage scenarios]] for more information.

=== For Basic Users ===

''These are available in the centralized testrun http log.''

* INFO shows what's happening in high level
** What parameters where used for the request
** What's happening right know. (Downloading image x from http://..., Flashing, Executing package asdf-tests.)

* ERROR shows what went wrong.
** It needs to be explained so that the basic user can understand it
** Always think from the basic user point of view. What makes sense to you as a developer probably does not make sense for the user at all.
** Don't say "_get_file("tests.xml") failed!", say "Testpackage asdf-tests is not in the image".
** Tracebacks are always good if they provide additional information for debugging. Just make sure that the actual error message is still there. Most of the users don't know python and do not want to read tracebacks. Instead of "Error! Traceback follows:", say "Custom flasher was not able to flash the image:". Then the basic user gets a rough idea what's going on and system maintainer/ots developer can solve the issue by looking at the traceback.

* WARNING warns if something is wrong but it's not critical
** For example user gives unknown input parameter. It might be a typo so the user should be warned.

=== For Advanced User / Worker Maintainer and Server Maintainer ===

''These logs are available in log files. Currently server debug log for each testrun is in /var/log/ots/. Conductor logs to ~/conductor.log in the worker machine. OTS worker process logs to /var/log/ots.log.''

* DEBUG on worker components
** In most cases INFO and ERROR level messages should be enough but we expect worker maintainers to be able to follow also DEBUG level messages
** DEBUG messages should give enough information to sort out problems in the worker setup.
** Don't say "calling flasher...". Say "executing command '/usr/bin/flasher -f imagefile'"
** Don't log too much. It makes reading the log difficult. Tracebacks should tell where we were when error happened.
** Input parameters and sub commands (parameters, return values, stdout, stderr) are likely to be valuable information when debugging.

== Logs ==

There are a number of ways to view the OTS testrun:

=== Main Log ===

The Main OTS log is written to

<code>
/var/log/ots
</code>

=== Server Log ===

This is [http://docs.python.org/release/2.5.2/lib/logging-config-fileformat.html configurable] by the 'logging.conf' file in the root directory

=== Worker Log ===

The Worker can be configured to log to a file specified by the "log_file" parameter in the Worker config file.

=== Conductor Log ===

Writes to the <ahem> home directory

<code>
/path/to/home/conductor.log
</code>

=== Apache Log ===

If you are running OTS with Apache:

<code>
/var/log/apache2/error.log
</code>

== Http Logger ==

In addition there is an [http://wiki.meego.com/Quality/QA-tools/OTS/Plugins/HTTP_logger http_logger].

Quality/QA-tools/OTS/Logging

2010-12-29T09:20:28Z

Tvainio: /* Log Levels */

= Logging =

== Log Levels ==

Different log levels are targeted for different user groups. See [[Quality/QA-tools/OTS/DeveloperDocs/TypicalUsage| Typical OTS users and usage scenarios]] for more information.

=== For Basic Users ===

''These are available in the centralized http log.''

* INFO shows what's happening in high level
** What parameters where used for the request
** What's happening right know. (Downloading image x from http://..., Flashing, Executing package asdf-tests.)

* ERROR shows what went wrong.
** It needs to be explained so that the basic user can understand it
** Always think from the basic user point of view. What makes sense to you as a developer probably does not make sense for the user at all.
** Don't say "_get_file("tests.xml") failed!", say "Testpackage asdf-tests is not in the image".
** Tracebacks are always good if they provide additional information for debugging. Just make sure that the actual error message is still there. Most of the users don't know python and do not want to read tracebacks. Instead of "Error! Traceback follows:", say "Custom flasher was not able to flash the image:". Then the basic user gets a rough idea what's going on and system maintainer/ots developer can solve the issue by looking at the traceback.

* WARNING warns if something is wrong but it's not critical
** For example user gives unknown input parameter. It might be a typo so the user should be warned.

=== For Advanced User / Worker Maintainer and Server Maintainer ===

''These logs are available in log files. Currently server debug log for each testrun is in /var/log/ots/. Conductor logs to ~/conductor.log in the worker machine. OTS worker process logs to /var/log/ots.log.''

* DEBUG on worker components
** In most cases INFO and ERROR level messages should be enough but we expect worker maintainers to be able to follow also DEBUG level messages
** DEBUG messages should give enough information to sort out problems in the worker setup.
** Don't say "calling flasher...". Say "executing command '/usr/bin/flasher -f imagefile'"
** Don't log too much. It makes reading the log difficult. Tracebacks should tell where we were when error happened.
** Input parameters and sub commands (parameters, return values, stdout, stderr) are likely to be valuable information when debugging.

== Logs ==

There are a number of ways to view the OTS testrun:

=== Main Log ===

The Main OTS log is written to

<code>
/var/log/ots
</code>

=== Server Log ===

This is [http://docs.python.org/release/2.5.2/lib/logging-config-fileformat.html configurable] by the 'logging.conf' file in the root directory

=== Worker Log ===

The Worker can be configured to log to a file specified by the "log_file" parameter in the Worker config file.

=== Conductor Log ===

Writes to the <ahem> home directory

<code>
/path/to/home/conductor.log
</code>

=== Apache Log ===

If you are running OTS with Apache:

<code>
/var/log/apache2/error.log
</code>

== Http Logger ==

In addition there is an [http://wiki.meego.com/Quality/QA-tools/OTS/Plugins/HTTP_logger http_logger].

Quality/QA-tools/OTS/Logging

2010-12-29T09:19:25Z

Tvainio: /* Logging */

= Logging =

== Log Levels ==

Different log levels are targeted for different user groups.

=== For Basic Users ===

''These are available in the centralized http log.''

* INFO shows what's happening in high level
** What parameters where used for the request
** What's happening right know. (Downloading image x from http://..., Flashing, Executing package asdf-tests.)

* ERROR shows what went wrong.
** It needs to be explained so that the basic user can understand it
** Always think from the basic user point of view. What makes sense to you as a developer probably does not make sense for the user at all.
** Don't say "_get_file("tests.xml") failed!", say "Testpackage asdf-tests is not in the image".
** Tracebacks are always good if they provide additional information for debugging. Just make sure that the actual error message is still there. Most of the users don't know python and do not want to read tracebacks. Instead of "Error! Traceback follows:", say "Custom flasher was not able to flash the image:". Then the basic user gets a rough idea what's going on and system maintainer/ots developer can solve the issue by looking at the traceback.

* WARNING warns if something is wrong but it's not critical
** For example user gives unknown input parameter. It might be a typo so the user should be warned.

=== For Advanced User / Worker Maintainer and Server Maintainer ===

''These logs are available in log files. Currently server debug log for each testrun is in /var/log/ots/. Conductor logs to ~/conductor.log in the worker machine. OTS worker process logs to /var/log/ots.log.''

* DEBUG on worker components
** In most cases INFO and ERROR level messages should be enough but we expect worker maintainers to be able to follow also DEBUG level messages
** DEBUG messages should give enough information to sort out problems in the worker setup.
** Don't say "calling flasher...". Say "executing command '/usr/bin/flasher -f imagefile'"
** Don't log too much. It makes reading the log difficult. Tracebacks should tell where we were when error happened.
** Input parameters and sub commands (parameters, return values, stdout, stderr) are likely to be valuable information when debugging.

== Logs ==

There are a number of ways to view the OTS testrun:

=== Main Log ===

The Main OTS log is written to

<code>
/var/log/ots
</code>

=== Server Log ===

This is [http://docs.python.org/release/2.5.2/lib/logging-config-fileformat.html configurable] by the 'logging.conf' file in the root directory

=== Worker Log ===

The Worker can be configured to log to a file specified by the "log_file" parameter in the Worker config file.

=== Conductor Log ===

Writes to the <ahem> home directory

<code>
/path/to/home/conductor.log
</code>

=== Apache Log ===

If you are running OTS with Apache:

<code>
/var/log/apache2/error.log
</code>

== Http Logger ==

In addition there is an [http://wiki.meego.com/Quality/QA-tools/OTS/Plugins/HTTP_logger http_logger].

Quality/QA-tools/OTS/DeveloperDocs/TypicalUsage

2010-12-29T09:17:33Z

Tvainio: Created page with "== Typical OTS Usage == This page tries to describe how end users typically use OTS. The motivation for this page is to gain common understanding about the problem domain. === …"

== Typical OTS Usage ==

This page tries to describe how end users typically use OTS. The motivation for this page is to gain common understanding about the problem domain.

=== Users ===

==== Basic User ====

Typically a test engineer or mobile SW developer.

* "I want to execute my tests on real target device."

===== Expected Knowledge =====

* Knows test definition and test packaging
* Knows his/her particular application/test area in detailed level
* Knows what SW product he/she is developing
* Understands OTS testrun in a rough level. (Download and flash the image, execute tests over ssh.)
* Knows some of the OTS input parameters. For example email address, timeout, SW Product

* Does not know python
* Does not know details about the particular OTS system setup (devicegroups or workers available)

===== OTS Usage =====

In most cases does not directly interact with OTS.

* Testrun from build machine
** User makes changes to Testpackage or target SW package.
** User creates image with build machine.
** Build machine triggers automatic testrun for the new image containing new version of user's software or tests.
** OTS uses default parameters for the particular SW Product. (User does not define devicegroup or other parameters. User knows only the SW Product.)
** User follows testrun process from html log page.
** User receives test results as an email. They are also available in a reporting tool.
** If there was errors in the testrun, the testrun log gives clear explanation what went wrong. (Flashing failed, Testpackage missing from the image, Unknown SW Product etc.)

* Nightly testruns
** User makes changes to Testpackage or target SW package.
** Build machine automatically builds a nightly image with new changes included.
** Build machine triggers automatic testrun for the new version of user's packages.
** OTS parameters for nightly build are defined by an advanced user responsible for the nightly testruns. (Basic user does not define devicegroup or other parameters)
** User receives test results as an email. They are also available in a reporting tool.
** If there was errors in the testrun, the testrun log gives clear explanation what went wrong. (Flashing failed, Testpackage missing from the image, Unknown SW Product etc.)

==== Advanced User / Worker Maintainer ====

Typically a test engineer or mobile SW developer. Has own device and worker pc connected to the OTS system. Motivation for own worker might be that the test cases require a specific environment or the user just wants to see what's happening while tests are executed.

* "I want to execute my tests in my own environment."

===== Expected Knowledge =====

In addition to the basic user knowledge:

* Knows how to set up and configure OTS worker.
* Knows the device properties and how to use the own device for a testrun

===== OTS Usage =====

*In addition to the basic user use cases, defines the device group (or device parameters) in the test run parameters so that testrun is routed to the specific worker.

* Local conductor run
** Executes conductor from command line with custom built image.
** Follows testrun from conductor output & log.
** Result files are available in file system. (~/conductor/)
** If errors or failures happen, the logs/output gives enough data for debugging

==== OTS System Maintainer ====

A system administrator is responsible for the OTS server and the whole testing system instance.

* "I want to provide a common test environment for my peers."

===== Expected Knowledge =====

In addition to the Advanced User / Worker Maintainer:

* Knows how to set up and configure OTS server
* Knows all OTS input parameters and how to trigger runs from command line
* Knows the common device pool setup
* Knows the default settings for each SW Product
* Knows testrun process in detailed level. Is able to debug problems in the system.
* Some python knowledge. Can read tracebacks and understands some of the common exceptions (import errors etc.)
* Is able to make detailed bug reports

===== OTS Usage =====

* Setup
** Installs ots server and common workers
** Defines supported SW products for the particular OTS instance.
** Defines device parameters for the common devices.
** Defines default values for supported SW products so that the basic user does not need to know any details about the system.
*** Basic users can just trigger a run and everything goes right thanks to the default values.
*** For example the test run will be routed to a suitable device because default devicegroup is defined in the config by the system maintainer.
*** The testrun result is reported correctly in the reporting tool because the system maintainer has defined correct reporting category in the config file and that will be automatically used when communicating with reporting tool.

* Debug a testrun
** A testrun triggered by a user ends with errors. User is not able to solve the problem so system maintainer is asked for help.
** System maintainer studies the logs and debugs the problem
*** System maintainer can re-trigger the testrun for debugging purposes by checking the exact input parameters from the log.
** System maintainer files bugs against OTS SW component if needed
** System maintainer fixes problems in the OTS setup if needed
** System maintainer instructs the user in OTS usage if needed

Quality/QA-tools/OTS/DeveloperDocs

2010-12-29T09:00:45Z

Tvainio: /* Analysis */

= OTS - Open Testing Service =

== Related Documentation ==

* [[Quality/QA-tools/OTS/Architecture]]
* [[Quality/QA-tools/OTS/Glossary]]

== Synopsis ==

OTS is a Distributed Test System that takes a Data Driven Approach using XML Files to allow parallel testing of Applications on evolving Hardware platforms.

Design has been driven by the need for a language neutral test platform that allows testing under different device architectures in parallel.

== High Level Overview ==

OTS allows a Test Sequence to be fanned out from a hub known as a '''Distributor''' to a remote machine known as a '''Worker'''.
Each '''Worker''' has a '''Device Group''' associated with it, this is defined in the configuration. The Tests are routed to the '''Worker'''
on the basis of the '''Device Group'''.

An XML '''Test Definition File''' describes the Tests that are run. The results are returned to the '''Distributor''' in an XML '''Results File'''.

The '''Distributor''' takes a '''Device Group''' and '''Timeout''' as inputs, as well as the '''Command''' for running '''Tasks'''. Tests are run within a '''Task''' which
is currently run as a process. The '''Command''' will typically take a path to the '''Test Definition File''' as an input.
If the process exceeds the '''Timeout''' the '''Task''' is stopped (the process is killed).

A number of '''Tasks''' can be added to a '''Testrun'''. The '''Testrun''' will normally wait for all '''Tasks''' to finish before completion.

Data (status, error and results) are communicated back from the '''Task''' to the '''Worker'''. The '''Results Object''' is a container for the files outputted: '''Test Definition File''', '''Results File'''
and any other files produced by the test as well as metadata relating to the run.

== Technology ==

OTS is written in Python 2.5.

OTS has the following FIXME (provided as separate PDF)

=== XmlFileFormats ===

The XML file formats provide the datum for the system.

Download [http://meego.gitorious.org/meego-quality-assurance/test-definition/trees/master them] and add the path to the '''results_xsd''' parameter in your [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/dev_branch_0.8/ots.server/ots/server/ots_server.ini ots_server.ini] file

=== Third Party Dependencies ===

==== RabbitMQ ====

You need to install [http://www.rabbitmq.com/server.html RabbitMQ].

===== Managing the Queues =====

If you need to delete or empty the queues there are [http://meego.gitorious.org/meego-quality-assurance/ots/trees/master/ots.server/ots/server/distributor/dev_utils utilities] to help with this.

==== Python Packages ====

The following python packages are needed

* [http://pypi.python.org/pypi/setuptools setuptools]

* [http://www.voidspace.org.uk/python/configobj.html configobj]

* [http://pypi.python.org/pypi/multiprocessing multiprocessing]

* [http://barryp.org/software/py-amqplib/ pyamqplib]

* [http://www.familieleuthe.de/DownloadMiniXsv.html minixsv]

* [http://www.djangoproject.com/download/1.1.2/tarball/ django]

* [http://code.google.com/p/python-nose/ nose]

== Diving In ==

The following steps show how to get a Development Environment up and running

=== 1. Install Dependencies ===

* Install the Third Party Dependencies

* Get the XML File Formats and set the OTS_TESTDEFINITION environment variable

=== 2. Build the Eggs ===

From the root directory

<code>
./setup.sh
source ./paths.sh
</code>

=== 3. Unittests ===

Run the unittests from the root directory

<code>
./nose.sh
</code>

=== 4. Component Tests ===

Component Tests run high level tests on an OTS Component using Mocks of the dependencies.

You can run these with the component_tests shell script in the root directory.

<code>
./component_tests.sh
</code>

=== 5. Hello OTS World ===

The source contains a simple demonstration of the way the key elements work.

==== 1. Open two terminal windows. ====

==== 2. In the first window run the server ====

<code>
cd $SERVER
cd hub/demos
python demo_hub.py
</code>

==== 3. You should see this: ====

<code>
<snip>
20XX-XX-XX XX:XX:XX,XXX - ots.server.distributor.taskrunner - DEBUG - Sending command '['echo', 'hello world']' with key 'foo'
</code>

==== 4. In the second window run the worker ====

<code>
cd $WORKER
python worker.py -c config.ini
</code>

==== 5. Now in the Server terminal the command should run to completion. You should see the following line in the logs: ====

<code>
20XX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world
</code>

It will raise a PackageException. This is normal as no Tasks that are run return Packages to the Server

==== 6. And on the logs on the Worker side should contain: ====

<code>
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - DEBUG - Task in state: 'finish'
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - INFO - Recommence consume on queue: foo
</code>

== Installation ==

It is intended that the '''Distributor''' and each '''Worker''' should be installed to separate machines.

== Configuration ==

=== Distributor ===

The format of the configuration file is as follows.

The `host` is the name of the RabbitMQ server

<code>

[Client]
host = localhost
vhost = /
port = 5672
username = guest
password = guest
queue = FIXME
consumer_tag = worker
services_exchange = services

</code>

=== Worker ===

Edit your config.ini so that the

* `queue`
* `routing_key`
* `services_exchange`

Are set for your '''Device Group'''.

And the `host` is the name of your RabbitMQ server.

<code>
[Worker]
host = localhost
vhost = /
port = 5672
username = guest
password = guest
queue = foo
routing_key = foo
services_exchange = foo
</code>

==== Starting the Worker ====

To start your Worker:

<code>
python worker.py -c config.ini
</code>

== Automated System Tests ==

''''' Not yet merged to dev_branch_0.8 '''''

Automated system tests are available in ots.system_tests. They trigger various testruns with xmlrpc and check return value and log messages.

=== Requirements ===

* Fully functional OTS system with one worker.
* A SW Product "ots-system-tests" configured so that it defaults to the worker.
* A meego image with test-definition-tests and testrunner-lite-regression-tests.
* Django based advanced OTS setup.
* BeautifulSoup python library.

=== Configuration ===

* make the system test image available for the workers. (for example copy it to your apache static content directory /var/www/static/)
* Modify system_tests.conf in ots.system_tests directory to match your system.
* Make sure all the urls in system_tests.conf work.

=== Notes ===

* System tests execute multiple full testruns. Running all tests can easily take more than 1.5 hours with N900 worker.
* Make sure there's no other activities ongoing in the system while running the tests.
* Some error handling tests expect certain devicegroups and sw products _not_ to be available. If these test cases fail you can remove those queues with ots tools.

== License ==

OTS is distributed under an LGPL license.

== Contributions ==

From [http://meego.gitorious.org/meego-quality-assurance/ qa-tools]

== Wiki ==

[[Quality/QA-tools/OTS | OTS Wiki Home]]

== Source Code Documentation ==

FIXME

== Mailing List ==

FIXME

== IRC Channel ==

The #meego-qa-tools irc channel in freenode

== Bugs ==

The [http://bugs.meego.com/ Meego bugs page]

== Platform Dependencies ==

OTS is tested to run on Linux only. Productization environment is Ubuntu 8.04 LTS (Long Term Support).

== Experimental ==

Documentation for experimental branches, spikes, WIPs etc. can be found in [[Quality/QA-tools/OTS/DeveloperDocs/Experimental]].

== Analysis ==

[[Quality/QA-tools/OTS/DeveloperDocs/ErrorSituations| OTS 0.1 Error Situations]]

[[Quality/QA-tools/OTS/DeveloperDocs/TypicalUsage| Typical OTS users and usage scenarios]]

Quality/QA-tools/OTS/DeveloperDocs

2010-12-29T09:00:31Z

Tvainio: /* Analysis */

= OTS - Open Testing Service =

== Related Documentation ==

* [[Quality/QA-tools/OTS/Architecture]]
* [[Quality/QA-tools/OTS/Glossary]]

== Synopsis ==

OTS is a Distributed Test System that takes a Data Driven Approach using XML Files to allow parallel testing of Applications on evolving Hardware platforms.

Design has been driven by the need for a language neutral test platform that allows testing under different device architectures in parallel.

== High Level Overview ==

OTS allows a Test Sequence to be fanned out from a hub known as a '''Distributor''' to a remote machine known as a '''Worker'''.
Each '''Worker''' has a '''Device Group''' associated with it, this is defined in the configuration. The Tests are routed to the '''Worker'''
on the basis of the '''Device Group'''.

An XML '''Test Definition File''' describes the Tests that are run. The results are returned to the '''Distributor''' in an XML '''Results File'''.

The '''Distributor''' takes a '''Device Group''' and '''Timeout''' as inputs, as well as the '''Command''' for running '''Tasks'''. Tests are run within a '''Task''' which
is currently run as a process. The '''Command''' will typically take a path to the '''Test Definition File''' as an input.
If the process exceeds the '''Timeout''' the '''Task''' is stopped (the process is killed).

A number of '''Tasks''' can be added to a '''Testrun'''. The '''Testrun''' will normally wait for all '''Tasks''' to finish before completion.

Data (status, error and results) are communicated back from the '''Task''' to the '''Worker'''. The '''Results Object''' is a container for the files outputted: '''Test Definition File''', '''Results File'''
and any other files produced by the test as well as metadata relating to the run.

== Technology ==

OTS is written in Python 2.5.

OTS has the following FIXME (provided as separate PDF)

=== XmlFileFormats ===

The XML file formats provide the datum for the system.

Download [http://meego.gitorious.org/meego-quality-assurance/test-definition/trees/master them] and add the path to the '''results_xsd''' parameter in your [http://meego.gitorious.org/meego-quality-assurance/ots/blobs/dev_branch_0.8/ots.server/ots/server/ots_server.ini ots_server.ini] file

=== Third Party Dependencies ===

==== RabbitMQ ====

You need to install [http://www.rabbitmq.com/server.html RabbitMQ].

===== Managing the Queues =====

If you need to delete or empty the queues there are [http://meego.gitorious.org/meego-quality-assurance/ots/trees/master/ots.server/ots/server/distributor/dev_utils utilities] to help with this.

==== Python Packages ====

The following python packages are needed

* [http://pypi.python.org/pypi/setuptools setuptools]

* [http://www.voidspace.org.uk/python/configobj.html configobj]

* [http://pypi.python.org/pypi/multiprocessing multiprocessing]

* [http://barryp.org/software/py-amqplib/ pyamqplib]

* [http://www.familieleuthe.de/DownloadMiniXsv.html minixsv]

* [http://www.djangoproject.com/download/1.1.2/tarball/ django]

* [http://code.google.com/p/python-nose/ nose]

== Diving In ==

The following steps show how to get a Development Environment up and running

=== 1. Install Dependencies ===

* Install the Third Party Dependencies

* Get the XML File Formats and set the OTS_TESTDEFINITION environment variable

=== 2. Build the Eggs ===

From the root directory

<code>
./setup.sh
source ./paths.sh
</code>

=== 3. Unittests ===

Run the unittests from the root directory

<code>
./nose.sh
</code>

=== 4. Component Tests ===

Component Tests run high level tests on an OTS Component using Mocks of the dependencies.

You can run these with the component_tests shell script in the root directory.

<code>
./component_tests.sh
</code>

=== 5. Hello OTS World ===

The source contains a simple demonstration of the way the key elements work.

==== 1. Open two terminal windows. ====

==== 2. In the first window run the server ====

<code>
cd $SERVER
cd hub/demos
python demo_hub.py
</code>

==== 3. You should see this: ====

<code>
<snip>
20XX-XX-XX XX:XX:XX,XXX - ots.server.distributor.taskrunner - DEBUG - Sending command '['echo', 'hello world']' with key 'foo'
</code>

==== 4. In the second window run the worker ====

<code>
cd $WORKER
python worker.py -c config.ini
</code>

==== 5. Now in the Server terminal the command should run to completion. You should see the following line in the logs: ====

<code>
20XX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world
</code>

It will raise a PackageException. This is normal as no Tasks that are run return Packages to the Server

==== 6. And on the logs on the Worker side should contain: ====

<code>
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.command - DEBUG - process.communicate() returned echo hello world
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - DEBUG - Task in state: 'finish'
XXXX-XX-XX XX:XX:XX,XXX - ots.worker.task_broker - INFO - Recommence consume on queue: foo
</code>

== Installation ==

It is intended that the '''Distributor''' and each '''Worker''' should be installed to separate machines.

== Configuration ==

=== Distributor ===

The format of the configuration file is as follows.

The `host` is the name of the RabbitMQ server

<code>

[Client]
host = localhost
vhost = /
port = 5672
username = guest
password = guest
queue = FIXME
consumer_tag = worker
services_exchange = services

</code>

=== Worker ===

Edit your config.ini so that the

* `queue`
* `routing_key`
* `services_exchange`

Are set for your '''Device Group'''.

And the `host` is the name of your RabbitMQ server.

<code>
[Worker]
host = localhost
vhost = /
port = 5672
username = guest
password = guest
queue = foo
routing_key = foo
services_exchange = foo
</code>

==== Starting the Worker ====

To start your Worker:

<code>
python worker.py -c config.ini
</code>

== Automated System Tests ==

''''' Not yet merged to dev_branch_0.8 '''''

Automated system tests are available in ots.system_tests. They trigger various testruns with xmlrpc and check return value and log messages.

=== Requirements ===

* Fully functional OTS system with one worker.
* A SW Product "ots-system-tests" configured so that it defaults to the worker.
* A meego image with test-definition-tests and testrunner-lite-regression-tests.
* Django based advanced OTS setup.
* BeautifulSoup python library.

=== Configuration ===

* make the system test image available for the workers. (for example copy it to your apache static content directory /var/www/static/)
* Modify system_tests.conf in ots.system_tests directory to match your system.
* Make sure all the urls in system_tests.conf work.

=== Notes ===

* System tests execute multiple full testruns. Running all tests can easily take more than 1.5 hours with N900 worker.
* Make sure there's no other activities ongoing in the system while running the tests.
* Some error handling tests expect certain devicegroups and sw products _not_ to be available. If these test cases fail you can remove those queues with ots tools.

== License ==

OTS is distributed under an LGPL license.

== Contributions ==

From [http://meego.gitorious.org/meego-quality-assurance/ qa-tools]

== Wiki ==

[[Quality/QA-tools/OTS | OTS Wiki Home]]

== Source Code Documentation ==

FIXME

== Mailing List ==

FIXME

== IRC Channel ==

The #meego-qa-tools irc channel in freenode

== Bugs ==

The [http://bugs.meego.com/ Meego bugs page]

== Platform Dependencies ==

OTS is tested to run on Linux only. Productization environment is Ubuntu 8.04 LTS (Long Term Support).

== Experimental ==

Documentation for experimental branches, spikes, WIPs etc. can be found in [[Quality/QA-tools/OTS/DeveloperDocs/Experimental]].

== Analysis ==

[[Quality/QA-tools/OTS/DeveloperDocs/ErrorSituations| OTS 0.1 Error Situations]]
[[Quality/QA-tools/OTS/DeveloperDocs/TypicalUsage| Typical OTS users and usage scenarios]]