MeeGo wiki - User contributions [en]

Quality/QA-tools/Test plan

2011-06-15T06:09:40Z

Slahtinen: /* Measurement data */

== Test plan overview ==

The basic principle of test plan XML and tool support is that you can use 'any' executable for testing.
Test results are checked from exit codes (automated testing) or prompt (manual testing). Executed test plan XMLs produce XML results - wrapping variety of test methods in consistent format required by test automation and data processing.

The test plan information stored in Test Definition XML files consists of:

1. suite, set, case: Hierarchical structure of the tests.
2. feature, subfeature, requirement: Information about why the tested software has been implemented in the first place (which is why it’s being tested as well).
3. type: Information about the viewpoint of the tests (which quality aspect of the software they are testing, see DevelopmentTestArea in Agile Testing Wiki).
4. level: Information about which test level the tests belong - for test level.
5. domain: Information about which architectural domains the tests are focused on.
6. description: descriptions of the tests (what each test is, and what it is supposed to do).
7. step: Execution instructions (for automated tests), which determine the actual commands to execute to run each test.

''Note:'' that all of the above are not mandatory. Mandatory fields for executing tests are defined with test plan XML validation schema.

== Creating Test plan ==

There are certain mandatory things you’ll need to provide in a test definition XML.

The structure of the XML and the possible attributes/values are defined in the [[Quality/QA-tools/Test-definition|Test Definition XML schema]]. It’s important to validate your XML against the up-to-date schema.

Test plans can be created either by using [[Quality/QA-tools/Testplanner|Testplanner]] tool or manually.

=== Test Cases ===

The main thing in test plan are the test cases, which define what to execute (test steps), what to expect from the execution (expected results) and also some additional information for reporting purposes. An example test case could be:

<case name="my-first-case">
<description>Creating my very first test case</description>
<step>ls</step>
<step>uname -r</step>
</case>

What did we just do? We created a simple test case [[#About_name_attribute|named]] <code>my-first-test-case</code>, which executes two steps and expects that they return the common “success return value” 0. Want to check for some other return value? Use <code>expected_result</code>:

<step expected_result="-1">ls</step>

In case you feel that some test cases are insignificant, i.e. that they shouldn’t be taken into consideration when choosing whether the test run is passed or failed, you can use insignificant-attribute (defaults to <code>false</code>, i.e. every case is considered to be significant):

<case name="not-so-important-test" insignificant="true">

In addition to test execution, the “story” of the test case should also be told, and for telling this story, we have fields for test type (what quality characteristics are tested) and test level (what level of system is being exercised. The possible values for both of these are listed in the appendices. Let’s add level and type to our test case:

<case name="my-first-case" level="Product" type="Functional">
<description>Creating my very first test case</description>
<step expected_result="-1">ls</step>
<step>uname -r</step>
</case>

And that’s our first test case! Not that difficult...

=== Grouping Cases - Sets and Suites ===

Grouping cases to sets and suites makes your life easier and more organized. A good idea is to group test cases that test the same features into same sets, and sets that test the same architectural domain under same suites. E.g.:

<suite name="my-multimedia-tests" domain="Multimedia">
<set name="video-playback-tests" feature="Video Playback">
<description>Video</description>
<case ...
<case ...
</set>
</suite>

So, sets contain test cases, and may also contain description (the same way as the cases do). More about sets in Test Definition Execution part

'''Note:''' There are several attributes that need to be defined on the test case, test set, or test suite level in the test definition XML.<br /> You can find more information about these attributes in the Appendices section of this page.

=== Time outs ===
Test steps will time out by default after 90 seconds. You can change the default time out by adding timeout attribute to your test case. for example

<case name="my-test-case" timeout="120">

Note that this the timeout is for single test step not for the whole case.

For [pre|post]-steps the the default timeout is 180 seconds, it can also be changed:

<pre_steps timeout="600">

=== Putting It All Together ===

Now that you have some knowledge on grouping and test cases in general, it is time to put it all together. Before showing an example, it is important to note that all the mentioned case attributes (e.g. level, type) are inheritable. So, if you have e.g. a set which contains only certain type of cases, you can define the type at the set-level, instead of writing it separately into each case.

Now, the example. Let’s first add the mandatory tags, which each test definition should have:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
</testdefinition>

That’s the mandatory part; we defined that we have a XML-document, and that this particular one is test definition version 1.0. Let’s add more beef to the bones by adding one suite with couple of sets:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
<suite name="my-multimedia-tests" domain="Multimedia">
<description>Testing AF stuff</description>
<set name="video-playback-tests" feature="Video Playback">
<description>Video playback tests</description>
</set>
<set name="video-recording-tests" feature="Video Recording">
<description>Video recording tests</description>
</set>
</suite>
</testdefinition>

Okay, we have one suite named <code>my-multimedia-tests with two sets</code>, testing video playback and recording features. Cases are still missing:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
<suite name="my-multimedia-tests" domain="Multimedia">
<description>Video playback tests</description>
<set name="video-playback-tests" feature="Video Playback">
<description>Video playback tests</description>
<case name="playback1" type="Functional" level="Component">
<step>execute_playback_test</step>
</case>
</set>
<set name="video-recording-tests" feature="Video Recording">
<description>Video playback tests</description>
<case ...
</set>
</suite>
</testdefinition>

And that’s the basic story. Next chapters cover more details about the definition, but you should now have the general understanding of the subject!

== Controlling Environment for Execution ==

=== Setup and Teardown ===

In case you want to do some setup and cleaning before and after the cases are executed, sets may have pre- and post-steps in them:

<set ...>
<pre_steps>
<step>do_some_setup</step>
</pre_steps>
<case ...
<post_steps>
<step>clean_up</step>
</post_steps>
</set>

If you want to ensure that pre-step is executed properly before starting the real testing, you can use expected result also in those:

<pre_steps>
<step expected_result="1">do_some_setup_that_may_fail</step>
...

:'''Warning''' When using expected result in this context, the process return value will be waited for, thus you cannot use this with daemon or background processes since they basically never return and cause execution to jam. Another thing to note is that the steps are executed in separate shells, so it is not possible to e.g. set environment variables or change directories in pre-steps as those will be lost.

=== Filtering Based on Hardware Identifier ===

If different tests sets for different hardware are required then hwiddetect feature can be utilised. User can define a command used to get a hardware identifier within hwiddetect tag. The hardware identifier returned by the command is matched with optional hwid attribute of a test set. If not equal, test cases in the set are skipped and are not written to the result file. A test set will never be skipped if hwid attribute has not been defined for it. You can also define multiple hwid values separated by comma for a set.

Command defined by hwiddetect can be a shell command or a separate executable. The executable should be included in the test package. Testrunner-lite removes extra whitespace and linefeeds from the the output of the hwiddetect command so that test developer does not need to care about it.

Example usage of hwiddetect:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
<hwiddetect>/usr/bin/getmyhwid</hwiddetect>
<suite name="suite1">
<set name="test_feature_X_on_hw_bar" hwid="bar">
<case name="test_X_1">
<step>echo "hwid is bar"</step>
</case>
</set>
<set name="test_feature_X_on_hw_foo" hwid="foo">
<case name="test_X_1">
<step>echo "hwid is foo"</step>
</case>
</set>
<set name="test_feature_X_on_hw_foo_or_bar" hwid="foo,bar">
<case name="test_X_1">
<step>echo "hwid is foo or bar"</step>
</case>
</set>
</suite>
</testdefinition>

=== Fetching Additional Files ===

In addition to normal result file, you can also fetch what ever files you need with get-tag:

<set ...>
...
<get>
<file>/tmp/myadditionalresult.1</file>
<file delete_after="true">/tmp/myadditionalresult.2</file>
</get>
</set>

In the example above my ''additionalresult.2'' file is deleted after fetching it (think of command ''mv'').

=== Measurement data ===

In case file is tagged as measurement data as in the below example, the data will be evaluated and transferred to results.

<case ...>
...
<get>
<file measurement="true">/path/to/measurement/measurement.txt</file>
</get>
</case>

The measurement data has the following CSV format:
name;value;unit;
name;value;unit;target;failure;
name;value;unit;target;failure;

Where ''name'' and ''unit'' are strings; ''value'' ''target'' and ''failure'' floating point numbers.
Example:
bt.upload;1.4123432;MB/s;
cpu.load;23.41;%;5;90;
mem.load;80.16;%;80;99;

If ''target'' and ''failure'' are specified the measurement can affect (fail) test case result. When ''target'' is greater than ''failure'', the ''value'' should be smaller than ''failure'' and vise versa.

When measurements consists of a series of data, series attribute can be used in file element:

<file measurement="true" series="true">/path/to/measurement/series.txt</file>

Measurement series CSV file format is different from the one consisting of single measurement values (optional parts are enclosed by brackets):

name;unit[;target;failure]
[yyyy-mm-ddThh:mm:ss[.ssssss];]value
[yyyy-mm-ddThh:mm:ss[.ssssss];]value
[yyyy-mm-ddThh:mm:ss[.ssssss];]value
...

The first line specifies series name and unit. In addition, target and failure limits can be specified. Next lines list all measurement values with optional timestamp (conforming to [http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]). A CSV file can contain only a single measurement series.

== Manual, Automatic and Semi-automatic Test Cases ==

If not specified by the "manual" attribute all cases are automatic. The value for the attribute is inherited from higher entity (set->case->step). By semi-automatic test case we mean a manual case that has some automatic parts; the idea being that only those steps that have to be manual, are. Note that this does not work the other way around, we cannot have automatic test case with manual steps (would be semantically weird thus our tools do not support it).

The example below tries to clarify the above. The "example_set" shows a manual, automatic and semi-automatic case. The example works with testrunner-lite/testrunner-ui, give it a try: [[File:example_definition.xml]].

<?xml version="1.0" encoding="UTF-8"?>

<testdefinition version="1.0">
<suite name="example_suite">
<set name="example_set">
<description>Example test set with manual, automatic and semi-automatic case.</description>
<case manual="true" name="manual_case">
<description>Manual test case with three steps inside one step tag.</description>
<step>Step 1: execute command ttt on shell.
Step 2: write something into edit box.
Step 3: press ok button.
Expected: Text should be updated into label.
</step>
</case>
<case timeout="96" name="automatic_case">
<description>Automatic test case that executes some shell commands.</description>
<step>ls /tmp</step>
<step expected_result="2">ls /nosuchfile</step>
<step>pwd</step>
</case>
<case manual="true" name="semi_automatic_case">
<description>A case with two automatic and two manual steps.</description>
<step manual="false">xcalc &amp;</step>
<step>Step: Type in 2 + 2 =. Expected: 4 is displayed </step>
<step>Press x² button. Expected: 16 is displayed.</step>
<step manual="false">killall xcalc</step>
</case>
</set>
</suite>
</testdefinition>
(A real test developer / tester might come up with nicer examples, any input welcome).

== Test Plan Execution ==

Test plans are run with test execution tools such as [[Quality/QA-tools/Testrunner|Testrunner]] and [[Quality/QA-tools/Testrunner-lite|testrunner-lite]], which read the plan and produce a result XML file.

== About name attribute ==

Note that the name attribute is of type [http://www.ietf.org/rfc/rfc2396.txt anyURI]. Do not use the following characters in suite,set or case names.
*'''''; / ? : @ & = + $ ,''''' (reserved)
or
* '''''{ } | \ ^ [ ] `''''' (unwise)

Also the use of '''''<space>''''' is not recommended, since some validators accept if for anyURI, some don't. Use '''''_''''' or '''''-''''' instead.

Quality/QA-tools/Test plan

2011-06-14T11:40:48Z

Slahtinen: /* Measurement data */

== Test plan overview ==

The basic principle of test plan XML and tool support is that you can use 'any' executable for testing.
Test results are checked from exit codes (automated testing) or prompt (manual testing). Executed test plan XMLs produce XML results - wrapping variety of test methods in consistent format required by test automation and data processing.

The test plan information stored in Test Definition XML files consists of:

1. suite, set, case: Hierarchical structure of the tests.
2. feature, subfeature, requirement: Information about why the tested software has been implemented in the first place (which is why it’s being tested as well).
3. type: Information about the viewpoint of the tests (which quality aspect of the software they are testing, see DevelopmentTestArea in Agile Testing Wiki).
4. level: Information about which test level the tests belong - for test level.
5. domain: Information about which architectural domains the tests are focused on.
6. description: descriptions of the tests (what each test is, and what it is supposed to do).
7. step: Execution instructions (for automated tests), which determine the actual commands to execute to run each test.

''Note:'' that all of the above are not mandatory. Mandatory fields for executing tests are defined with test plan XML validation schema.

== Creating Test plan ==

There are certain mandatory things you’ll need to provide in a test definition XML.

The structure of the XML and the possible attributes/values are defined in the [[Quality/QA-tools/Test-definition|Test Definition XML schema]]. It’s important to validate your XML against the up-to-date schema.

Test plans can be created either by using [[Quality/QA-tools/Testplanner|Testplanner]] tool or manually.

=== Test Cases ===

The main thing in test plan are the test cases, which define what to execute (test steps), what to expect from the execution (expected results) and also some additional information for reporting purposes. An example test case could be:

<case name="my-first-case">
<description>Creating my very first test case</description>
<step>ls</step>
<step>uname -r</step>
</case>

What did we just do? We created a simple test case [[#About_name_attribute|named]] <code>my-first-test-case</code>, which executes two steps and expects that they return the common “success return value” 0. Want to check for some other return value? Use <code>expected_result</code>:

<step expected_result="-1">ls</step>

In case you feel that some test cases are insignificant, i.e. that they shouldn’t be taken into consideration when choosing whether the test run is passed or failed, you can use insignificant-attribute (defaults to <code>false</code>, i.e. every case is considered to be significant):

<case name="not-so-important-test" insignificant="true">

In addition to test execution, the “story” of the test case should also be told, and for telling this story, we have fields for test type (what quality characteristics are tested) and test level (what level of system is being exercised. The possible values for both of these are listed in the appendices. Let’s add level and type to our test case:

<case name="my-first-case" level="Product" type="Functional">
<description>Creating my very first test case</description>
<step expected_result="-1">ls</step>
<step>uname -r</step>
</case>

And that’s our first test case! Not that difficult...

=== Grouping Cases - Sets and Suites ===

Grouping cases to sets and suites makes your life easier and more organized. A good idea is to group test cases that test the same features into same sets, and sets that test the same architectural domain under same suites. E.g.:

<suite name="my-multimedia-tests" domain="Multimedia">
<set name="video-playback-tests" feature="Video Playback">
<description>Video</description>
<case ...
<case ...
</set>
</suite>

So, sets contain test cases, and may also contain description (the same way as the cases do). More about sets in Test Definition Execution part

'''Note:''' There are several attributes that need to be defined on the test case, test set, or test suite level in the test definition XML.<br /> You can find more information about these attributes in the Appendices section of this page.

=== Time outs ===
Test steps will time out by default after 90 seconds. You can change the default time out by adding timeout attribute to your test case. for example

<case name="my-test-case" timeout="120">

Note that this the timeout is for single test step not for the whole case.

For [pre|post]-steps the the default timeout is 180 seconds, it can also be changed:

<pre_steps timeout="600">

=== Putting It All Together ===

Now that you have some knowledge on grouping and test cases in general, it is time to put it all together. Before showing an example, it is important to note that all the mentioned case attributes (e.g. level, type) are inheritable. So, if you have e.g. a set which contains only certain type of cases, you can define the type at the set-level, instead of writing it separately into each case.

Now, the example. Let’s first add the mandatory tags, which each test definition should have:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
</testdefinition>

That’s the mandatory part; we defined that we have a XML-document, and that this particular one is test definition version 1.0. Let’s add more beef to the bones by adding one suite with couple of sets:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
<suite name="my-multimedia-tests" domain="Multimedia">
<description>Testing AF stuff</description>
<set name="video-playback-tests" feature="Video Playback">
<description>Video playback tests</description>
</set>
<set name="video-recording-tests" feature="Video Recording">
<description>Video recording tests</description>
</set>
</suite>
</testdefinition>

Okay, we have one suite named <code>my-multimedia-tests with two sets</code>, testing video playback and recording features. Cases are still missing:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
<suite name="my-multimedia-tests" domain="Multimedia">
<description>Video playback tests</description>
<set name="video-playback-tests" feature="Video Playback">
<description>Video playback tests</description>
<case name="playback1" type="Functional" level="Component">
<step>execute_playback_test</step>
</case>
</set>
<set name="video-recording-tests" feature="Video Recording">
<description>Video playback tests</description>
<case ...
</set>
</suite>
</testdefinition>

And that’s the basic story. Next chapters cover more details about the definition, but you should now have the general understanding of the subject!

== Controlling Environment for Execution ==

=== Setup and Teardown ===

In case you want to do some setup and cleaning before and after the cases are executed, sets may have pre- and post-steps in them:

<set ...>
<pre_steps>
<step>do_some_setup</step>
</pre_steps>
<case ...
<post_steps>
<step>clean_up</step>
</post_steps>
</set>

If you want to ensure that pre-step is executed properly before starting the real testing, you can use expected result also in those:

<pre_steps>
<step expected_result="1">do_some_setup_that_may_fail</step>
...

:'''Warning''' When using expected result in this context, the process return value will be waited for, thus you cannot use this with daemon or background processes since they basically never return and cause execution to jam. Another thing to note is that the steps are executed in separate shells, so it is not possible to e.g. set environment variables or change directories in pre-steps as those will be lost.

=== Filtering Based on Hardware Identifier ===

If different tests sets for different hardware are required then hwiddetect feature can be utilised. User can define a command used to get a hardware identifier within hwiddetect tag. The hardware identifier returned by the command is matched with optional hwid attribute of a test set. If not equal, test cases in the set are skipped and are not written to the result file. A test set will never be skipped if hwid attribute has not been defined for it. You can also define multiple hwid values separated by comma for a set.

Command defined by hwiddetect can be a shell command or a separate executable. The executable should be included in the test package. Testrunner-lite removes extra whitespace and linefeeds from the the output of the hwiddetect command so that test developer does not need to care about it.

Example usage of hwiddetect:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
<hwiddetect>/usr/bin/getmyhwid</hwiddetect>
<suite name="suite1">
<set name="test_feature_X_on_hw_bar" hwid="bar">
<case name="test_X_1">
<step>echo "hwid is bar"</step>
</case>
</set>
<set name="test_feature_X_on_hw_foo" hwid="foo">
<case name="test_X_1">
<step>echo "hwid is foo"</step>
</case>
</set>
<set name="test_feature_X_on_hw_foo_or_bar" hwid="foo,bar">
<case name="test_X_1">
<step>echo "hwid is foo or bar"</step>
</case>
</set>
</suite>
</testdefinition>

=== Fetching Additional Files ===

In addition to normal result file, you can also fetch what ever files you need with get-tag:

<set ...>
...
<get>
<file>/tmp/myadditionalresult.1</file>
<file delete_after="true">/tmp/myadditionalresult.2</file>
</get>
</set>

In the example above my ''additionalresult.2'' file is deleted after fetching it (think of command ''mv'').

=== Measurement data ===

In case file is tagged as measurement data as in the below example, the data will be evaluated and transferred to results.

<case ...>
...
<get>
<file measurement="true">/path/to/measurement/measurement.txt</file>
</get>
</case>

The measurement data has the following CSV format:
name;value;unit;
name;value;unit;target;failure;
name;value;unit;target;failure;

Where ''name'' and ''unit'' are strings; ''value'' ''target'' and ''failure'' floating point numbers.
Example:
bt.upload;1.4123432;MB/s;
cpu.load;23.41;%;5;90;
mem.load;80.16;%;80;99;

If ''target'' and ''failure'' are specified the measurement can affect (fail) test case result. When ''target'' is greater than ''failure'', the ''value'' should be smaller than ''failure'' and vise versa.

When measurements consists of a series of data, series attribute can be used in file element:

<file measurement="true" series="true">/path/to/measurement/measurement.txt</file>

Measurement series CSV file format is different from the one consisting of single measurement values:

name;unit[;target;failure]
[yyyy-mm-ddThh:mm:ss[.ssssss];]value
[yyyy-mm-ddThh:mm:ss[.ssssss];]value
[yyyy-mm-ddThh:mm:ss[.ssssss];]value
...

The first line specifies series name and unit. In addition, target and failure limits can be specified. Next lines lists measurement values with optional timestamps (conforming to [http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]).

== Manual, Automatic and Semi-automatic Test Cases ==

If not specified by the "manual" attribute all cases are automatic. The value for the attribute is inherited from higher entity (set->case->step). By semi-automatic test case we mean a manual case that has some automatic parts; the idea being that only those steps that have to be manual, are. Note that this does not work the other way around, we cannot have automatic test case with manual steps (would be semantically weird thus our tools do not support it).

The example below tries to clarify the above. The "example_set" shows a manual, automatic and semi-automatic case. The example works with testrunner-lite/testrunner-ui, give it a try: [[File:example_definition.xml]].

<?xml version="1.0" encoding="UTF-8"?>

<testdefinition version="1.0">
<suite name="example_suite">
<set name="example_set">
<description>Example test set with manual, automatic and semi-automatic case.</description>
<case manual="true" name="manual_case">
<description>Manual test case with three steps inside one step tag.</description>
<step>Step 1: execute command ttt on shell.
Step 2: write something into edit box.
Step 3: press ok button.
Expected: Text should be updated into label.
</step>
</case>
<case timeout="96" name="automatic_case">
<description>Automatic test case that executes some shell commands.</description>
<step>ls /tmp</step>
<step expected_result="2">ls /nosuchfile</step>
<step>pwd</step>
</case>
<case manual="true" name="semi_automatic_case">
<description>A case with two automatic and two manual steps.</description>
<step manual="false">xcalc &amp;</step>
<step>Step: Type in 2 + 2 =. Expected: 4 is displayed </step>
<step>Press x² button. Expected: 16 is displayed.</step>
<step manual="false">killall xcalc</step>
</case>
</set>
</suite>
</testdefinition>
(A real test developer / tester might come up with nicer examples, any input welcome).

== Test Plan Execution ==

Test plans are run with test execution tools such as [[Quality/QA-tools/Testrunner|Testrunner]] and [[Quality/QA-tools/Testrunner-lite|testrunner-lite]], which read the plan and produce a result XML file.

== About name attribute ==

Note that the name attribute is of type [http://www.ietf.org/rfc/rfc2396.txt anyURI]. Do not use the following characters in suite,set or case names.
*'''''; / ? : @ & = + $ ,''''' (reserved)
or
* '''''{ } | \ ^ [ ] `''''' (unwise)

Also the use of '''''<space>''''' is not recommended, since some validators accept if for anyURI, some don't. Use '''''_''''' or '''''-''''' instead.

Quality/QA-tools

2011-01-28T13:09:35Z

Slahtinen: Test environment validation and equipment control in roadmap

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://meego.gitorious.org/meego-quality-assurance/ git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner|Testrunner]] || [http://meego.gitorious.org/meego-quality-assurance/testrunner Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://meego.gitorious.org/meego-quality-assurance/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://meego.gitorious.org/meego-quality-assurance/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| [[Quality/TestSuite/MCTS|MeeGo Core Test Suite]] || [http://meego.gitorious.org/meego-quality-assurance/mcts/ Gitorious] ||Matti Salmi || Jeff Zheng
|-
| [[Quality/Netbook Test Suite and Utilities|MeeGo Netbook Test Suite]] || [http://gitorious.org/qa-tools/mnts/ Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://meego.gitorious.org/meego-quality-assurance/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://meego.gitorious.org/meego-quality-assurance/rich-core Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter settings || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter-settings-public Gitorious] || Carol Rus || Raimo Gratseff
|-
| [[Quality/QA-tools/hat-control|Hardware Accessory for Testing (HAT)]] || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| [[Quality/QA-tools/QAReports|QA Reports]] || [http://meego.gitorious.org/meego-quality-assurance/qa-reports Gitorious] || Sami Hangaslammi || Jarno Keskikangas
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] || [http://meego.gitorious.com/meego-quality-assurance/scripts-and-utils Gitorious] || N/A || N/A
|-
| [[Quality/TestSuite/handset-test-suite/handset-ux-test_Releases | handset_ux_tests]] || [http://meego.gitorious.com/meego-quality-assurance/handset-ux-tests Gitorious] || JessicaJi || N/A
|-
| [[Quality/QA-tools/MeeGo_Fast_Feedback_Testing|MeeGo Fast Feedback Testing (MeeGo-FFT)]] || [http://meego.gitorious.org/meego-quality-assurance/meego-testing-hudson-plugin Gitorious] || Alexey Kuznetsov || Timo Härkönen
|-
| Service OS based Flasher || [http://meego.gitorious.org/meego-quality-assurance/meego-ai-serviceos Gitorious] || Jing Wang || N/A
|-
| Qpid C wrapper library - libcqpid || [http://meego.gitorious.org/meego-quality-assurance/libcqpid Gitorious] || Sami Lahtinen || N/A
|}

You can install Testrunner, testrunner-lite, Testplanner, OTS, Meego-ai and eat from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here is the workflow for QA tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file: [[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Roadmap ==
(Only partial, will be updated asap as we file rest of the bugs -timakima)

The release dates defined in MeeGo Release plans [[Release_Engineering/Plans/1.1|1.1]] and [[Release_Engineering/Plans/1.2|1.2]]. These dates are the latest estimation. They will be updated as work progresses.

{| border="1" cellspacing="0"
|+
|'''Feature''' || '''Release'''
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12980&hide_resolved=0 Measurement support] || 1.2
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12937&hide_resolved=0 MCTS coverage support] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12833&hide_resolved=0 Test environment validation] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=13092&hide_resolved=0 Test equipment control] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12454&hide_resolved=0 Parallel testing] || TBD
|-
| [https://bugs.meego.com/show_bug.cgi?id=12810 Easy install] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12971&hide_resolved=0 Qt Creator integration] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12981&hide_resolved=0 Events feature in automatic testing] || TBD
|}

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=MeeGo%20Quality%20Assurance Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

We'd like to have the best guess of the moment about the delivery of features. Remember to set the target build of the bug you're working with according to: [[Release Engineering/Plans/1.1]] and [[Release Engineering/Plans/1.2]]

When you add a new bug, add correct dependencies to the corresponding [[Quality/QA-tools#Roadmap|roadmap]] meta bug.

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS test automation design|MCTS Test Automation]] (QA Tools support for MCTS)

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

* Team meetings
** QA tools team meetings will be held on need basis for specific topics. This was agreed in [http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-07-11.59.html the last weekly meeting held on December 7th 2010]
** [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

* Architecture meetings
** Architecture meetings will be held on-demand basis.
** [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

* Workshops
** Face-2-face meetings within the team.
** [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarmo Savinen || Team member || Digia || jasavi
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
| Sami Hangaslammi || Team member || Leonidas || sahangas
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
** [http://timoph.fi/qa-tools-stats/ #meego-qa-tools statistics]
** [http://timoph.fi/qa-tools-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools

2011-01-28T07:39:25Z

Slahtinen: Updated roadmap item for parallel testing

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://meego.gitorious.org/meego-quality-assurance/ git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner|Testrunner]] || [http://meego.gitorious.org/meego-quality-assurance/testrunner Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://meego.gitorious.org/meego-quality-assurance/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://meego.gitorious.org/meego-quality-assurance/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| [[Quality/TestSuite/MCTS|MeeGo Core Test Suite]] || [http://meego.gitorious.org/meego-quality-assurance/mcts/ Gitorious] ||Matti Salmi || Jeff Zheng
|-
| [[Quality/Netbook Test Suite and Utilities|MeeGo Netbook Test Suite]] || [http://gitorious.org/qa-tools/mnts/ Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://meego.gitorious.org/meego-quality-assurance/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://meego.gitorious.org/meego-quality-assurance/rich-core Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter settings || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter-settings-public Gitorious] || Carol Rus || Raimo Gratseff
|-
| [[Quality/QA-tools/hat-control|Hardware Accessory for Testing (HAT)]] || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| [[Quality/QA-tools/QAReports|QA Reports]] || [http://meego.gitorious.org/meego-quality-assurance/qa-reports Gitorious] || Sami Hangaslammi || Jarno Keskikangas
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] || [http://meego.gitorious.com/meego-quality-assurance/scripts-and-utils Gitorious] || N/A || N/A
|-
| [[Quality/TestSuite/handset-test-suite/handset-ux-test_Releases | handset_ux_tests]] || [http://meego.gitorious.com/meego-quality-assurance/handset-ux-tests Gitorious] || JessicaJi || N/A
|-
| [[Quality/QA-tools/MeeGo_Fast_Feedback_Testing|MeeGo Fast Feedback Testing (MeeGo-FFT)]] || [http://meego.gitorious.org/meego-quality-assurance/meego-testing-hudson-plugin Gitorious] || Alexey Kuznetsov || Timo Härkönen
|-
| Service OS based Flasher || [http://meego.gitorious.org/meego-quality-assurance/meego-ai-serviceos Gitorious] || Jing Wang || N/A
|-
| Qpid C wrapper library - libcqpid || [http://meego.gitorious.org/meego-quality-assurance/libcqpid Gitorious] || Sami Lahtinen || N/A
|}

You can install Testrunner, testrunner-lite, Testplanner, OTS, Meego-ai and eat from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here is the workflow for QA tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file: [[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Roadmap ==
(Only partial, will be updated asap as we file rest of the bugs -timakima)

The release dates defined in MeeGo Release plans [[Release_Engineering/Plans/1.1|1.1]] and [[Release_Engineering/Plans/1.2|1.2]]. These dates are the latest estimation. They will be updated as work progresses.

{| border="1" cellspacing="0"
|+
|'''Feature''' || '''Release'''
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12980&hide_resolved=0 Measurement support] || 1.2
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12937&hide_resolved=0 MCTS coverage support] || TBD
|-
| Test environment validation || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12454&hide_resolved=0 Parallel testing] || TBD
|-
| [https://bugs.meego.com/show_bug.cgi?id=12810 Easy install] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12971&hide_resolved=0 Qt Creator integration] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12981&hide_resolved=0 Events feature in automatic testing] || TBD
|}

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=MeeGo%20Quality%20Assurance Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

We'd like to have the best guess of the moment about the delivery of features. Remember to set the target build of the bug you're working with according to: [[Release Engineering/Plans/1.1]] and [[Release Engineering/Plans/1.2]]

When you add a new bug, add correct dependencies to the corresponding [[Quality/QA-tools#Roadmap|roadmap]] meta bug.

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS test automation design|MCTS Test Automation]] (QA Tools support for MCTS)

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

* Team meetings
** QA tools team meetings will be held on need basis for specific topics. This was agreed in [http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-07-11.59.html the last weekly meeting held on December 7th 2010]
** [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

* Architecture meetings
** Architecture meetings will be held on-demand basis.
** [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

* Workshops
** Face-2-face meetings within the team.
** [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarmo Savinen || Team member || Digia || jasavi
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
| Sami Hangaslammi || Team member || Leonidas || sahangas
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
** [http://timoph.fi/qa-tools-stats/ #meego-qa-tools statistics]
** [http://timoph.fi/qa-tools-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools

2011-01-27T14:41:21Z

Slahtinen: Added libcqpid to Tools

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://meego.gitorious.org/meego-quality-assurance/ git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner|Testrunner]] || [http://meego.gitorious.org/meego-quality-assurance/testrunner Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://meego.gitorious.org/meego-quality-assurance/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://meego.gitorious.org/meego-quality-assurance/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| [[Quality/TestSuite/MCTS|MeeGo Core Test Suite]] || [http://meego.gitorious.org/meego-quality-assurance/mcts/ Gitorious] ||Matti Salmi || Jeff Zheng
|-
| [[Quality/Netbook Test Suite and Utilities|MeeGo Netbook Test Suite]] || [http://gitorious.org/qa-tools/mnts/ Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://meego.gitorious.org/meego-quality-assurance/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://meego.gitorious.org/meego-quality-assurance/rich-core Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter settings || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter-settings-public Gitorious] || Carol Rus || Raimo Gratseff
|-
| [[Quality/QA-tools/hat-control|Hardware Accessory for Testing (HAT)]] || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| [[Quality/QA-tools/QAReports|QA Reports]] || [http://meego.gitorious.org/meego-quality-assurance/qa-reports Gitorious] || Sami Hangaslammi || Jarno Keskikangas
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] || [http://meego.gitorious.com/meego-quality-assurance/scripts-and-utils Gitorious] || N/A || N/A
|-
| [[Quality/TestSuite/handset-test-suite/handset-ux-test_Releases | handset_ux_tests]] || [http://meego.gitorious.com/meego-quality-assurance/handset-ux-tests Gitorious] || JessicaJi || N/A
|-
| [[Quality/QA-tools/MeeGo_Fast_Feedback_Testing|MeeGo Fast Feedback Testing (MeeGo-FFT)]] || [http://meego.gitorious.org/meego-quality-assurance/meego-testing-hudson-plugin Gitorious] || Alexey Kuznetsov || Timo Härkönen
|-
| Service OS based Flasher || [http://meego.gitorious.org/meego-quality-assurance/meego-ai-serviceos Gitorious] || Jing Wang || N/A
|-
| Qpid C wrapper library - libcqpid || [http://meego.gitorious.org/meego-quality-assurance/libcqpid Gitorious] || Sami Lahtinen || N/A
|}

You can install Testrunner, testrunner-lite, Testplanner, OTS, Meego-ai and eat from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here is the workflow for QA tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file: [[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Roadmap ==
(Only partial, will be updated asap as we file rest of the bugs -timakima)

The release dates defined in MeeGo Release plans [http://wiki.meego.com/Release_Engineering/Plans/1.1 1.1] and [http://wiki.meego.com/Release_Engineering/Plans/1.2 1.2]. These dates are the latest estimation. They will be updated as work progresses.

{| border="1" cellspacing="0"
|+
|'''Feature''' || '''Release'''
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12980&hide_resolved=0 Measurement support] || 1.2
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12937&hide_resolved=0 MCTS coverage support] || TBD
|-
| Test environment validation || TBD
|-
| Parallel testing || TBD
|-
| [https://bugs.meego.com/show_bug.cgi?id=12810 Easy install] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12971&hide_resolved=0 Qt Creator integration] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12981&hide_resolved=0 Events feature in automatic testing] || TBD
|}

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=MeeGo%20Quality%20Assurance Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=testrunner%20UI&component=testrunner-lite&classification=MeeGo%20Projects&product=Meego%20Quality%20Assurance Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=testrunner%20UI&component=testrunner-lite&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

We'd like to have the best guess of the moment about the delivery of features. Remember to set the target build of the bug you're working with according to: [[Release Engineering/Plans/1.1]] and [[Release Engineering/Plans/1.2]]

When you add a new bug, add correct dependencies to the corresponding [[Quality/QA-tools#Roadmap|roadmap]] meta bug.

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS test automation design|MCTS Test Automation]] (QA Tools support for MCTS)

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

* Team meetings
** QA tools team meetings will be held on need basis for specific topics. This was agreed in [http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-07-11.59.html the last weekly meeting held on December 7th 2010]
** [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

* Architecture meetings
** Architecture meetings will be held on-demand basis.
** [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

* Workshops
** Face-2-face meetings within the team.
** [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarmo Savinen || Team member || Digia || jasavi
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
| Sami Hangaslammi || Team member || Leonidas || sahangas
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
** [http://timoph.fi/qa-tools-stats/ #meego-qa-tools statistics]
** [http://timoph.fi/qa-tools-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools

2011-01-27T14:36:14Z

Slahtinen: /* Roadmap */

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://meego.gitorious.org/meego-quality-assurance/ git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner|Testrunner]] || [http://meego.gitorious.org/meego-quality-assurance/testrunner Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://meego.gitorious.org/meego-quality-assurance/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://meego.gitorious.org/meego-quality-assurance/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| [[Quality/TestSuite/MCTS|MeeGo Core Test Suite]] || [http://meego.gitorious.org/meego-quality-assurance/mcts/ Gitorious] ||Matti Salmi || Jeff Zheng
|-
| [[Quality/Netbook Test Suite and Utilities|MeeGo Netbook Test Suite]] || [http://gitorious.org/qa-tools/mnts/ Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://meego.gitorious.org/meego-quality-assurance/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://meego.gitorious.org/meego-quality-assurance/rich-core Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter settings || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter-settings-public Gitorious] || Carol Rus || Raimo Gratseff
|-
| [[Quality/QA-tools/hat-control|Hardware Accessory for Testing (HAT)]] || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| [[Quality/QA-tools/QAReports|QA Reports]] || [http://meego.gitorious.org/meego-quality-assurance/qa-reports Gitorious] || Sami Hangaslammi || Jarno Keskikangas
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] || [http://meego.gitorious.com/meego-quality-assurance/scripts-and-utils Gitorious] || N/A || N/A
|-
| [[Quality/TestSuite/handset-test-suite/handset-ux-test_Releases | handset_ux_tests]] || [http://meego.gitorious.com/meego-quality-assurance/handset-ux-tests Gitorious] || JessicaJi || N/A
|-
| [[Quality/QA-tools/MeeGo_Fast_Feedback_Testing|MeeGo Fast Feedback Testing (MeeGo-FFT)]] || [http://meego.gitorious.org/meego-quality-assurance/meego-testing-hudson-plugin Gitorious] || Alexey Kuznetsov || Timo Härkönen
|-
| Service OS based Flasher || [http://meego.gitorious.org/meego-quality-assurance/meego-ai-serviceos Gitorious] || Jing Wang || N/A
|}

You can install Testrunner, testrunner-lite, Testplanner, OTS, Meego-ai and eat from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here is the workflow for QA tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file: [[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Roadmap ==
(Only partial, will be updated asap as we file rest of the bugs -timakima)

The release dates defined in MeeGo Release plans [http://wiki.meego.com/Release_Engineering/Plans/1.1 1.1] and [http://wiki.meego.com/Release_Engineering/Plans/1.2 1.2]. These dates are the latest estimation. They will be updated as work progresses.

{| border="1" cellspacing="0"
|+
|'''Feature''' || '''Release'''
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12980&hide_resolved=0 Measurement support] || 1.2
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12937&hide_resolved=0 MCTS coverage support] || TBD
|-
| Test environment validation || TBD
|-
| Parallel testing || TBD
|-
| [https://bugs.meego.com/show_bug.cgi?id=12810 Easy install] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12971&hide_resolved=0 Qt Creator integration] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12981&hide_resolved=0 Events feature in automatic testing] || TBD
|}

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=MeeGo%20Quality%20Assurance Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=testrunner%20UI&component=testrunner-lite&classification=MeeGo%20Projects&product=Meego%20Quality%20Assurance Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=testrunner%20UI&component=testrunner-lite&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

We'd like to have the best guess of the moment about the delivery of features. Remember to set the target build of the bug you're working with according to: [[Release Engineering/Plans/1.1]] and [[Release Engineering/Plans/1.2]]

When you add a new bug, add correct dependencies to the corresponding [[Quality/QA-tools#Roadmap|roadmap]] meta bug.

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS test automation design|MCTS Test Automation]] (QA Tools support for MCTS)

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

* Team meetings
** QA tools team meetings will be held on need basis for specific topics. This was agreed in [http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-07-11.59.html the last weekly meeting held on December 7th 2010]
** [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

* Architecture meetings
** Architecture meetings will be held on-demand basis.
** [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

* Workshops
** Face-2-face meetings within the team.
** [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarmo Savinen || Team member || Digia || jasavi
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
| Sami Hangaslammi || Team member || Leonidas || sahangas
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
** [http://timoph.fi/qa-tools-stats/ #meego-qa-tools statistics]
** [http://timoph.fi/qa-tools-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools/MCTS test automation design/poc-implementation

2010-12-29T08:40:28Z

Slahtinen:

= Required components =
* [http://qpid.apache.org/ Qpid AMQP message broker and client libary]
* [http://gitorious.org/qpid-c-wrapper/libcqpid Experimental C wrapper for Qpid client library]
* [http://oss.metaparadigm.com/json-c/ JSON C implementation]
* testrunner-lite

=== Alternative AMQP implementation ===
* [http://www.openamq.org/ OpenAMQ message broker and client API]

== Building and installing required components ==
== Qpid ==
To ensure successful building of Qpid you should install package libboost-all-dev. Check also file INSTALL of Qpid release for other prerequisites.
* Download [http://mirror.eunet.fi/apache//qpid/0.8/qpid-cpp-0.8.tar.gz sources]
tar xzvf qpid-cpp-0.8.tar.gz
cd qpidc-0.8
./configure --without-sasl
make
sudo make install
== libcqpid C-wrapper for Qpid ==
Requires Qpid client libraries.
git clone git://gitorious.org/qpid-c-wrapper/libcqpid.git
cd libcqpid
./configure
make
sudo make install
== JSON ==
* Download [http://oss.metaparadigm.com/json-c/json-c-0.9.tar.gz sources]
tar xzvf json-c-0.9.tar.gz
cd json-c-0.9
./configure
make
sudo make install
sudo ldconfig
=== OpenAMQ (not used anymore) ===
This implementation was tested before Qpid.
* Download [http://download.imatix.com/openamq/stable/OpenAMQ-1.3d1.tar.gz sources]
tar xzvf OpenAMQ-1.3d1.tar.gz
cd OpenAMQ-1.3d1
IBASE=$PWD/ibase sh build.sh
We need to install archives and headers to system directories
sudo mkdir /usr/local/openamq
sudo mkdir /usr/local/openamq/lib
sudo mkdir /usr/local/openamq/include
sudo cp ibase/lib/*.a /usr/local/openamq/lib
sudo cp ibase/include/*.h /usr/local/openamq/include

= Testrunner-lite =
Testrunner-lite using libcqpid and JSON for event messaging can be found from [http://meego.gitorious.org/~slahtinen/meego-quality-assurance/slahtinens-testrunner-lite a testrunner-lite clone] in branch multidut.

In this proof of concepts, a separate testrunner-lite processes are used to run tests on each DUT. This produces multiple result XML files and requires a tool which is able to combine the results.

= Events =
JSON formatted events structures are carried in body of AMQP message. Each event should have at least the following fields:
* name (string)
Example event with additional data:
{
"name" : "event1",
"data" :
{
"description" : "Measurement value",
"value" : 123
}
}

== Status 20.12.2010 ==
Testrunner-lite is able to send and receive events through OpenAMQ message broker. Event data is serialized in JSON format and data is carried in the body of AMQP message.

Currently special test steps (ones with event attribute) can be used to send an event or wait for an event. This way execution of two distinct test runs can be synchronized. Also test steps run on DUT may generate/receive events, thus enabling communication between testrunner-lite or other DUTs.

== Status 28.12.2010 ==
Minimal implementation of Qpid cleint library C wrapper has been done. Event tags and libcqpid usage implementation work in testrunner-lite is in progress.

Quality/QA-tools/MCTS test automation design/poc-implementation

2010-12-29T08:39:17Z

Slahtinen:

= Required components =
* [http://qpid.apache.org/ Qpid AMQP message broker and client libary]
* [http://gitorious.org/qpid-c-wrapper/libcqpid Experimental C wrapper for Qpid client library]
* [http://oss.metaparadigm.com/json-c/ JSON C implementation]
* testrunner-lite

=== Alternative AMQP implementation ===
* [http://www.openamq.org/ OpenAMQ message broker and client API]

== Building and installing required components ==
== Qpid ==
To ensure successful building of Qpid you should install package libboost-all-dev. Check also file INSTALL of Qpid release for other prerequisites.
* Download [http://mirror.eunet.fi/apache//qpid/0.8/qpid-cpp-0.8.tar.gz sources]
tar xzvf qpid-cpp-0.8.tar.gz
cd qpidc-0.8
./configure --without-sasl
make
sudo make install
== libcqpid C-wrapper for Qpid ==
Requires Qpid client libraries.
git clone git://gitorious.org/qpid-c-wrapper/libcqpid.git
cd libcqpid
./configure
make
sudo make install
== JSON ==
* Download [http://oss.metaparadigm.com/json-c/json-c-0.9.tar.gz sources]
tar xzvf json-c-0.9.tar.gz
cd json-c-0.9
./configure
make
sudo make install
sudo ldconfig
=== OpenAMQ (not used anymore) ===
This implementation was tested before Qpid.
* Download [http://download.imatix.com/openamq/stable/OpenAMQ-1.3d1.tar.gz sources]
tar xzvf OpenAMQ-1.3d1.tar.gz
cd OpenAMQ-1.3d1
IBASE=$PWD/ibase sh build.sh
We need to install archives and headers to system directories
sudo mkdir /usr/local/openamq
sudo mkdir /usr/local/openamq/lib
sudo mkdir /usr/local/openamq/include
sudo cp ibase/lib/*.a /usr/local/openamq/lib
sudo cp ibase/include/*.h /usr/local/openamq/include

= Testrunner-lite =
Testrunner-lite using libcqpid and JSON for event messaging can be found from [http://meego.gitorious.org/~slahtinen/meego-quality-assurance/slahtinens-testrunner-lite a testrunner-lite clone] in branch multidut.

In this proof of concepts, a separate testrunner-lite processes are used to run tests on each DUT. This produces multiple result XML files and requires a tool which is able to combine the results.

== Status 20.12.2010 ==
Testrunner-lite is able to send and receive events through OpenAMQ message broker. Event data is serialized in JSON format and data is carried in the body of AMQP message.

Currently special test steps (ones with event attribute) can be used to send an event or wait for an event. This way execution of two distinct test runs can be synchronized. Also test steps run on DUT may generate/receive events, thus enabling communication between testrunner-lite or other DUTs.

== Status 28.12.2010 ==
Minimal implementation of Qpid cleint library C wrapper has been done. Event tags and libcqpid usage implementation work in testrunner-lite is in progress.

=== Events ===
JSON formatted events structures are carried in body of AMQP message. Each event should have at least the following fields:
* name (string)
Example event with additional data:
{
"name" : "event1",
"data" :
{
"description" : "Measurement value",
"value" : 123
}
}

Quality/QA-tools/MCTS test automation design/poc-implementation

2010-12-28T13:23:13Z

Slahtinen: Qpid and libcqpid changes

= Required components =
* [http://qpid.apache.org/ Qpid AMQP message broker and client libary]
* [http://gitorious.org/qpid-c-wrapper/libcqpid Experimental C wrapper for Qpid client library]
* [http://oss.metaparadigm.com/json-c/ JSON C implementation]
* testrunner-lite

=== Alternative AMQP implementation ===
* [http://www.openamq.org/ OpenAMQ message broker and client API]

== Building and installing required components ==
== Qpid ==
To ensure successful building of Qpid you should install package libboost-all-dev. Check also file INSTALL of Qpid release for other prerequisites.
* Download [http://mirror.eunet.fi/apache//qpid/0.8/qpid-cpp-0.8.tar.gz sources]
tar xzvf qpid-cpp-0.8.tar.gz
cd qpidc-0.8
./configure --without-sasl
make
sudo make install
== libcqpid C-wrapper for Qpid ==
Requires Qpid client libraries.
git clone git://gitorious.org/qpid-c-wrapper/libcqpid.git
cd libcqpid
./configure
make
sudo make install
== JSON ==
* Download [http://oss.metaparadigm.com/json-c/json-c-0.9.tar.gz sources]
tar xzvf json-c-0.9.tar.gz
cd json-c-0.9
./configure
make
sudo make install
sudo ldconfig
=== OpenAMQ (not used anymore) ===
This implementation was tested before Qpid.
* Download [http://download.imatix.com/openamq/stable/OpenAMQ-1.3d1.tar.gz sources]
tar xzvf OpenAMQ-1.3d1.tar.gz
cd OpenAMQ-1.3d1
IBASE=$PWD/ibase sh build.sh
We need to install archives and headers to system directories
sudo mkdir /usr/local/openamq
sudo mkdir /usr/local/openamq/lib
sudo mkdir /usr/local/openamq/include
sudo cp ibase/lib/*.a /usr/local/openamq/lib
sudo cp ibase/include/*.h /usr/local/openamq/include

= Testrunner-lite =
Testrunner-lite using libcqpid and JSON for event messaging can be found from [http://meego.gitorious.org/~slahtinen/meego-quality-assurance/slahtinens-testrunner-lite a testrunner-lite clone] in branch multidut.

In this proof of concepts, a separate testrunner-lite processes are used to run tests on each DUT. This produces multiple result XML files and requires a tool which is able to combine the results.

== Status 20.12.2010 ==
Testrunner-lite is able to send and receive events through OpenAMQ message broker. Event data is serialized in JSON format and data is carried in the body of AMQP message.

Currently special test steps (ones with event attribute) can be used to send an event or wait for an event. This way execution of two distinct test runs can be synchronized. Also test steps run on DUT may generate/receive events, thus enabling communication between testrunner-lite or other DUTs.

== Status 28.12.2010 ==
Minimal implementation of Qpid cleint library C wrapper has been done. Event tags and libcqpid usage implementation work in testrunner-lite is in progress.

Quality/QA-tools/MCTS test automation design/poc-implementation

2010-12-20T13:02:27Z

Slahtinen: /* Current status 20.12.2010 */

= Required components =
* [http://www.openamq.org/ OpenAMQ message broker and client API]
* [http://oss.metaparadigm.com/json-c/ JSON C implementation]
* testrunner-lite

== Building and installing required components ==
== OpenAMQ ==
* Download [http://download.imatix.com/openamq/stable/OpenAMQ-1.3d1.tar.gz sources]
tar xzvf OpenAMQ-1.3d1.tar.gz
cd OpenAMQ-1.3d1
IBASE=$PWD/ibase sh build.sh
We need to install archives and headers to system directories
sudo mkdir /usr/local/openamq
sudo mkdir /usr/local/openamq/lib
sudo mkdir /usr/local/openamq/include
sudo cp ibase/lib/*.a /usr/local/openamq/lib
sudo cp ibase/include/*.h /usr/local/openamq/include
== JSON ==
* Download [http://oss.metaparadigm.com/json-c/json-c-0.9.tar.gz sources]
tar xzvf json-c-0.9.tar.gz
cd json-c-0.9
./configure
make
sudo make install
sudo ldconfig
= Testrunner-lite =
Testrunner-lite using OpenAMQ and JSON for event messaging can be found from [http://meego.gitorious.org/~slahtinen/meego-quality-assurance/slahtinens-testrunner-lite a testrunner-lite clone] in branch multidut.

In this proof of concepts, a separate testrunner-lite processes are used to run tests on each DUT. This produces multiple result XML files and requires a tool which is able to combine the results.

== Current status 20.12.2010 ==
Testrunner-lite is able to send and receive events through OpenAMQ message broker. Event data is serialized in JSON format and data is carried in the body of AMQP message.

Currently special test steps (ones with event attribute) can be used to send an event or wait for an event. This way execution of two distinct test runs can be synchronized. Also test steps run on DUT may generate/receive events, thus enabling communication between testrunner-lite or other DUTs.

Quality/QA-tools/MCTS test automation design/poc-implementation

2010-12-20T12:59:33Z

Slahtinen: /* Testrunner-lite */

= Required components =
* [http://www.openamq.org/ OpenAMQ message broker and client API]
* [http://oss.metaparadigm.com/json-c/ JSON C implementation]
* testrunner-lite

== Building and installing required components ==
== OpenAMQ ==
* Download [http://download.imatix.com/openamq/stable/OpenAMQ-1.3d1.tar.gz sources]
tar xzvf OpenAMQ-1.3d1.tar.gz
cd OpenAMQ-1.3d1
IBASE=$PWD/ibase sh build.sh
We need to install archives and headers to system directories
sudo mkdir /usr/local/openamq
sudo mkdir /usr/local/openamq/lib
sudo mkdir /usr/local/openamq/include
sudo cp ibase/lib/*.a /usr/local/openamq/lib
sudo cp ibase/include/*.h /usr/local/openamq/include
== JSON ==
* Download [http://oss.metaparadigm.com/json-c/json-c-0.9.tar.gz sources]
tar xzvf json-c-0.9.tar.gz
cd json-c-0.9
./configure
make
sudo make install
sudo ldconfig
= Testrunner-lite =
Testrunner-lite using OpenAMQ and JSON for event messaging can be found from [http://meego.gitorious.org/~slahtinen/meego-quality-assurance/slahtinens-testrunner-lite a testrunner-lite clone] in branch multidut.

In this proof of concepts, a separate testrunner-lite processes are used to run tests on each DUT. This produces multiple result XML files and requires a tool which is able to combine the results.

== Current status 20.12.2010 ==
Testrunner-lite is able to send and receive events through OpenAMQ message broker. Event data is serialized in JSON format and data is carried in body of AMQP message.

Currently special test steps (ones with event attribute) can be used to send an event or wait for event. This way execution of two distinct test runs can be synchronized. Also test steps may generate/receive events.

Quality/QA-tools/MCTS test automation design/poc-implementation

2010-12-20T12:45:42Z

Slahtinen:

Quality/QA-tools/MCTS test automation design/poc-implementation

2010-12-20T12:38:20Z

Slahtinen:

= Used components =
* [http://www.openamq.org/ OpenAMQ message broker and client API]
* [http://oss.metaparadigm.com/json-c/ JSON C implementation]

= Building and installing required components =
== OpenAMQ ==
* Download [http://download.imatix.com/openamq/stable/OpenAMQ-1.3d1.tar.gz sources]
tar xzvf OpenAMQ-1.3d1.tar.gz
cd OpenAMQ-1.3d1
IBASE=$PWD/ibase sh build.sh
We need to install archives and headers to system directories
sudo mkdir /usr/local/openamq
sudo mkdir /usr/local/openamq/lib
sudo mkdir /usr/local/openamq/include
sudo cp ibase/lib/*.a /usr/local/openamq/lib
sudo cp ibase/include/*.h /usr/local/openamq/include
== JSON ==
* Download [http://oss.metaparadigm.com/json-c/json-c-0.9.tar.gz sources]
tar xzvf json-c-0.9.tar.gz
cd json-c-0.9
./configure
make
sudo make install
sudo ldconfig
= Testrunner-lite =
Testrunner-lite using OpenAMQ and JSON for event messaging can be found from [http://meego.gitorious.org/~slahtinen/meego-quality-assurance/slahtinens-testrunner-lite a testrunner-lite clone] in branch multidut.

Quality/QA-tools/MCTS test automation design/poc-implementation

2010-12-20T12:36:37Z

Slahtinen: /* Testrunner-lite */

= Used components =
* [http://www.openamq.org/ OpenAMQ message broker and client API]
* [http://oss.metaparadigm.com/json-c/ JSON C implementation]

= Building and installing required components =
== OpenAMQ ==
* Download [http://download.imatix.com/openamq/stable/OpenAMQ-1.3d1.tar.gz sources]
* tar xzvf OpenAMQ-1.3d1.tar.gz
* cd OpenAMQ-1.3d1
* IBASE=$PWD/ibase sh build.sh
We need to install archives and headers to system directories
* sudo mkdir /usr/local/openamq
* sudo mkdir /usr/local/openamq/lib
* sudo mkdir /usr/local/openamq/include
* sudo cp ibase/lib/*.a /usr/local/openamq/lib
* sudo cp ibase/include/*.h /usr/local/openamq/include
== JSON ==
Download [http://oss.metaparadigm.com/json-c/json-c-0.9.tar.gz sources]
* tar xzvf json-c-0.9.tar.gz
* cd json-c-0.9
* ./configure
* make
* sudo make install
* sudo ldconfig
= Testrunner-lite =
Testrunner-lite using OpenAMQ and JSON for event messaging can be found from [http://meego.gitorious.org/~slahtinen/meego-quality-assurance/slahtinens-testrunner-lite a testrunner-lite clone] in branch multidut.

Quality/QA-tools/MCTS test automation design/poc-implementation

2010-12-20T12:35:30Z

Slahtinen:

= Used components =
* [http://www.openamq.org/ OpenAMQ message broker and client API]
* [http://oss.metaparadigm.com/json-c/ JSON C implementation]

= Building and installing required components =
== OpenAMQ ==
* Download [http://download.imatix.com/openamq/stable/OpenAMQ-1.3d1.tar.gz sources]
* tar xzvf OpenAMQ-1.3d1.tar.gz
* cd OpenAMQ-1.3d1
* IBASE=$PWD/ibase sh build.sh
We need to install archives and headers to system directories
* sudo mkdir /usr/local/openamq
* sudo mkdir /usr/local/openamq/lib
* sudo mkdir /usr/local/openamq/include
* sudo cp ibase/lib/*.a /usr/local/openamq/lib
* sudo cp ibase/include/*.h /usr/local/openamq/include
== JSON ==
Download [http://oss.metaparadigm.com/json-c/json-c-0.9.tar.gz sources]
* tar xzvf json-c-0.9.tar.gz
* cd json-c-0.9
* ./configure
* make
* sudo make install
* sudo ldconfig
= Testrunner-lite =
Testrunner-lite using OpenAMQ and JSON for event messaging can be found from [http://meego.gitorious.org/~slahtinen/meego-quality-assurance/slahtinens a testrunner-lite clone] in branch multidut.

Quality/QA-tools/MCTS test automation design/poc-implementation

2010-12-20T12:30:36Z

Slahtinen: Created page with "= Used components = * [http://www.openamq.org/ OpenAMQ message broker and client API] * [http://oss.metaparadigm.com/json-c/ JSON C implementation] = Building and installing req…"

= Used components =
* [http://www.openamq.org/ OpenAMQ message broker and client API]
* [http://oss.metaparadigm.com/json-c/ JSON C implementation]

= Building and installing required components =
== OpenAMQ ==
* Download [http://download.imatix.com/openamq/stable/OpenAMQ-1.3d1.tar.gz
sources]
* tar xzvf OpenAMQ-1.3d1.tar.gz
* cd OpenAMQ-1.3d1
* IBASE=$PWD/ibase sh build.sh
We need to install archives and headers to system directories
* sudo mkdir /usr/local/openamq
* sudo mkdir /usr/local/openamq/lib
* sudo mkdir /usr/local/openamq/include
* sudo cp ibase/lib/*.a /usr/local/openamq/lib
* sudo cp ibase/include/*.h /usr/local/openamq/include
== JSON ==
Download [http://oss.metaparadigm.com/json-c/json-c-0.9.tar.gz
sources]
* tar xzvf json-c-0.9.tar.gz
* cd json-c-0.9
* ./configure
* make
* sudo make install
* sudo ldconfig
= Testrunner-lite
Testrunner-lite using OpenAMQ and JSON for event messaging can be found from [http://meego.gitorious.org/~slahtinen/meego-quality-assurance/slahtinens-testrunner-lite] in branch multidut.

Quality/QA-tools/MCTS test automation design

2010-12-20T12:14:34Z

Slahtinen: /* Meetings/ Planning */

'''!!!!!!!!!!! WORK IN PROGRESS !!!!!!!!!!!'''

= Introduction =

== Purpose ==

This document provides a technical architecture and system design for MCTS Test Automation environment. The purpose is to introduce a new subsystem to the existing MeeGo test automation environment (OTS), where external devices can be connected and controlled on test case or test step level. In this context an external device can for example be, a network simulator, a WLAN analyzer or even another DUT. The new subsystem utilizes test tools already in place in MeeGo by extending their functionality, and also introduces additional components to provide needed new functionality. Suitable open-source will also be utilized as much as it is rationale to do. Intended readers for this document are the developers contributing MeeGo QA Tools area and MCTS test asset developers. '''Contributions to this document are welcome'''.

== Scope ==

This document first gives an insight to the overall system with some background information before moving to the system architecture and detailed design chapters. Document focuses on solving following design issues:

* Control and interaction between simulators, analyzers and DUT(s)
* Interfaces of different architectural layers and components
* Adaptation of devices to the test automation environment

== Glossary ==

{| border="1" cellspacing="0"
|+
!| Term || Definition
|-
| DUT || Device Under Test
|-
| OTS || [[Quality/QA-tools/OTS|Link to wiki]]
|-
| test-definition || [[Quality/QA-tools/Test-definition|Link to wiki]]
|-
| testrunner-lite || [[Quality/QA-tools/Testrunner-lite|Link to wiki]]
|-
| testrunner || [[Quality/QA-tools/Testrunner-ui|Link to wiki]]
|}

= System Overview =

Current OTS environment allows to execute test cases in a single DUT. When executing automated functional test cases in a simulated live environment, it is usually required to verify the result from another device. An example from such test case would be a voice call between two or more DUTs. This is a typical master – slave test case, where master DUT is used to initiate MO call, which is then verified in the receiving slave device. Another situation, when test case verdict needs to be verified from external device is for example tests related to audio playback. In this case the result is fetched from audio analyzer for example. This design will introduce new messaging and communication method to the system, which is used to handle interaction between devices. The below picture illustrates the system overview.

[[File:Mcts_automation_system_overview_diagram.png|500px|thumb|center|System Overview]]

= Design Considerations =

== Assumptions and Dependencies ==

Any assumptions, dependencies and requirements regarding the system and operating environment are listed here.

* External device connecting to the test automation environment can be any type from any vendor
* Communication
** Devices need to be controlled from host and target DUT
** Test case/ testrunner-lite/ single component can communicate with other devices via messaging
* Message types
** Request-Reply
** Events/ signals
** Variable exchange in messages
** synchronous/ asynchronous messages
* Both parallel and sequential controlling of the devices shall be possible
* Interaction (events) can happen any time between devices
* External devices can be connected to a remote host
* In case of automated execution is not possible, manual execution mode must be available as a backup
* System needs to be able to run in both Linux and Windows (Priority is in Linux)

== Design Constraints ==

Any global limitations or constraints that have a significant impact on the design of the system's software are listed here.

* Usage of proprietary software components is not allowed
* System needs to extensible and distributable
* API for communicating with the devices has to be generic enough covering at least the basic and selected complex uses cases
* Backwards compatibility with the current test definitions MUST be maintained

= System Architecture =

For controlling multiple devices and executing commands either sequential or parallel, some short of event/mesaging mechanism is clearly needed. Implementing such subsystem from scratch would require too much effort. Thus existing open solutions are studied and utilized for this. Also extending some of the existing test tools (e.g. OTS) with such functionality would make the architecture too complex. To keep architecture modular, the component controlling handling the communication should be its own subsystem, which then will be connected to OTS.

== Architectural Design ==

This section describes the chosen architecture for the system. Also, the alternative architectures, which have been considered are discussed here.

=== Layered Architecture View ===

The below picture illustrates the layered architecture of the system. '''NOTE''': only the essential components are shown in the picture.

[[File:Mcts_layered_architecture.png|500px|thumb|center|Layered View]]

* '''Test Tools Layer''' is the top most layer in the architecture. It contains the tools for execution test binaries both in host and device side and recipe for executing commands.
* '''Test Case Layer''' contains the test binaries executed on the host side. Test cases itself can contain lots of state information and logic, which would be impossible to handle on tools side. Therefore, a support for host side test cases is provied.
* '''Device API Layer''' contains the different APIs, which are used to communicate with external devices. This layer contains an API, which common for all devices and device type specific APIs.
* '''Messaging Layer''' is a vertical layer providing the communication and message routing between devices
* '''Device Adaptation Layer''' contains the adaptations to the device drivers.

=== Component Architecture View ===

The picture below, illustrates the component view of the architecture. Each component in the diagram represents an individual subsystem (executable, library or data). For simplicity, only the relevant component and interfaces are described. '''Note''': component names are preliminary.

[[File:Mcts_component_diagram.png|600px|thumb|center|Component View]]

----

;test-definition (existing)
* '''Purpose''': Collection of XML schema files, which are used to validate test plan xml files.
* '''Provides''': Means to validate test plan XML syntax
* '''Uses''': -
* '''Changes Needed''':
** Support for:
*** test step level attributes, to mark, in which device command is executed
*** measurement values
*** fetching additional log files from different devices
** Test plan syntax needs be to extended by evolving current testdefinition-complex.xsd or by providing new even more complex schema.
* '''Type''': Data

;testrunner-lite (existing)
* '''Purpose''': Runs on host test system. Parses test plan XML and executes commands (test binaries) on host, remote device (over ssh) and external devices over messaging service.
* '''Provides''': Universal way to execute test cases, automated or manual.
* '''Uses''':
** test-defininition to validate test plan XML
** libxml2 for XML parsing
** ssh for communicating with the device
** libcurl for HTTP logging
** Messaging API for external device communication
* '''Changes Needed''':
** support for parsing new attributes from test plan XML
** execute test commands in the device instructed in test plan XML
** fetch additional log files from devices
** for test case development, support for simulating external devices is needed
** send/ receive AMQP messages
* '''Type''': Executable

;testrunner (existing)
* '''Purpose''': Executing test binaries manually
* '''Provides''': UI for the manual execution
* '''Uses''':
** testrunner-lite to execute test plan
** Messaging API for external device communication
* '''Changes Needed''':
** send/ receive AMQP messages
* '''Type''': Executable

;Messaging API (new)
* '''Purpose''': Hides the complexity of AMQP protocol from the client applications
* '''Provides''': Simple API for handling AMQP messaging
* '''Uses''': [[http://www.openamq.org/doc:prog-wireapi#toc27 OpenAMQ C API]]
* '''Type''': Library

== Sub-system Architecture ==

== Design Rationale ==

Rationale, why this approach was chosen.

== Design Alternatives ==

This section describes architectures which have been considered.

= Detailed System Design =

= Meetings/ Planning =

This section is meant for meeting and planning activities related to this work.

* [[Quality/QA-tools/MCTS-Test-Automation-Design/meeting-2010-11-30|Workshop on November 30th 2010]]
* [[Quality/QA-tools/MCTS-Test-Automation-Design/proof-of-concept-planning|Planning the PoC]]
* [[Quality/QA-tools/MCTS-Test-Automation-Design/poc-implementation|PoC Implementation]]

Quality/QA-tools

2010-08-26T06:21:26Z

Slahtinen:

== Quality Assurance (QA) Tools ==

Quality Assurance tools are developed to ensure MeeGo SW quality.

Work in progress includes open sourcing Nokia / Intel quality assurance tools and making open source test tools available for anyone interested in MeeGo software quality.

== QA Tools ==

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

QA tools team develops and maintains tools for Quality Assurance.

Anyone is welcome to contribute and non-member contributions will be treated
with same process and review as member contributions.

Summer 2010: We are in the process of making Nokia and Intel QA tools available for MeeGo. Please bear with us.

Team communication is in English.

Team meetings will be held in <code>#meego-meeting on irc.freenode.net</code> starting September - '''Meeting time has not been fixed - proposals welcome'''

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| Name
| Role
| Affiliation
| IRC nickname
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || -
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || -
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || -
|-
| Sampo Saaristo || Team member || Sofica || -
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
|}

Our collaboration spaces are:

* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
* Gitorious team, http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MeeGo OBS
* This wiki area
* [http://wiki.meego.com/Quality/QA-tools/ServiceOS ServiceOS]

[[Category:QA]]