MeeGo wiki - User contributions [en]

Quality/QA-tools/Corelysis

2011-06-15T07:44:57Z

Samposa: /* How does it work? */

= Corelysis =

== Introduction ==

Corelysis is a daemon service written in [http://www.python.org/ Python], which extracts, processes [[Quality/QA-tools/rich-core|rich core dumps]] from saved from MeeGo devices and finally publishes crash data in [[Quality/QA-tools/CrashReports|Crash Reports]]. Corelysis runs on back-end server and works closely with [[Quality/QA-tools/OTS|OTS]] service.

Corelysis has originated from [http://www.nokia.com Nokia], where it has been utilized for [http://maemo.org Maemo] related QA work. Later on Corelysis has been developed for [[ARM/N900|MeeGo N900 DeveloperEdition]] to report process crashes occured in automated test execution runs.

[[File:Text9867-0-4-8.png|600px|thumb|center|Automated testing and crash reporting]]

== How does it work? ==

The idea of Corelysis is quite simple. It takes a rich core dump as an input extracts the data from it and runs it through a set of "tasks", which each manipulate the data in different way. Maybe the most relevant task is the one which generates the backtrace from plain core dump. To be able to do this, Corelysis needs a fs-type meego image with corresponding debuginfo-packages installed. To generate the backtrace, a [[Quality/QA-tools/Corelysis#Setting_up_cross_gdb|cross-compiled GDB]] is executed in chrooted target.

[[File:backend.png|600px|thumb|center|Corelysis work flow]]

== Installation ==

=== Preparation - Adding MeeGo Tools:Testing repository ===

'''Ubuntu'''

* Depending on your distribution, add either of the below lines to /etc/apt/sources.list.

deb http://download.meego.com/live/Tools:/Testing/xUbuntu_10.04/ /

deb http://download.meego.com/live/Tools:/Testing/xUbuntu_10.10/ /

* Add the repository keys

gpg --keyserver subkeys.pgp.net --recv-keys 0BC7BEC479FC1F8A
gpg --export --armor 0BC7BEC479FC1F8A | sudo apt-key add -

* Update the package list

sudo apt-get update

'''Fedora'''

* Add the repository

cd /etc/yum.repos.d/
sudo wget http://download.meego.com/live/Tools:/Testing/Fedora13/Tools:Testing.repo

* Add the repository keys

gpg --keyserver subkeys.pgp.net --recv-keys 0BC7BEC479FC1F8A
gpg --export --armor 0BC7BEC479FC1F8A | sudo apt-key add -

=== Install from MeeGo package repositories ===

'''Ubuntu'''

sudo apt-get install python-corelysis

'''Fedora'''

sudo yum install python-corelysis

=== Install from sources ===

* Get sources
git clone git://gitorious.org/meego-quality-assurance/corelysis.git

* Install dependencies

'''Ubuntu'''

sudo apt-get install python-setuptools python-pyinotify sp-rich-core-postproc

'''Fedora'''

sudo yum install python-setuptools python-inotify sp-rich-core-postproc

* Install everything

cd corelysis
sudo python setup.py install

=== Setting up cross gdb ===

* obtain cross gdb by e.g. installing [http://meego.com/downloads/releases/1.2/meego-v1.2-sdk meego SDK].
* the gdb needs to be statically linked, this can be achieved by [http://statifier.sourceforge.net statifier tool]
cd /opt/meego/meego-1.2-sdk-armv7hl-toolchain/arm-meego-linux-gnueabi/bin
statifier arm-meego-linux-gnueabi-gdb arm-meego-linux-gnueabi-gdb-static
* for the static binary to work, process address space randomization must be disabled
# echo 0 > /proc/sys/kernel/randomize_va_space

=== Executing Corelysis ===

sudo /usr/bin/corelysis

== Contact ==

In Corelysis related issues/ questions contact us in the following channels:

* [http://webchat.freenode.net/?channels=meego-qa #meego-qa IRC channel on irc.freenode.net]
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]

Quality/QA-tools/Corelysis

2011-06-15T07:38:25Z

Samposa: /* Prepare environment */

= Corelysis =

== Introduction ==

Corelysis is a daemon service written in [http://www.python.org/ Python], which extracts, processes [[Quality/QA-tools/rich-core|rich core dumps]] from saved from MeeGo devices and finally publishes crash data in [[Quality/QA-tools/CrashReports|Crash Reports]]. Corelysis runs on back-end server and works closely with [[Quality/QA-tools/OTS|OTS]] service.

Corelysis has originated from [http://www.nokia.com Nokia], where it has been utilized for [http://maemo.org Maemo] related QA work. Later on Corelysis has been developed for [[ARM/N900|MeeGo N900 DeveloperEdition]] to report process crashes occured in automated test execution runs.

[[File:Text9867-0-4-8.png|600px|thumb|center|Automated testing and crash reporting]]

== How does it work? ==

The idea of Corelysis is quite simple. It takes a rich core dump as an input extracts the data from it and runs it through a set of "tasks", which each manipulate the data in different way. Maybe the most relevant task is the one which generates the backtrace from plain core dump. To be able to do this, Corelysis needs a fs-type meego image with corresponding debuginfo-packages installed. To generate the backtrace, a cross-compiled GDB is executed in chrooted target.

[[File:backend.png|600px|thumb|center|Corelysis work flow]]

== Installation ==

=== Preparation - Adding MeeGo Tools:Testing repository ===

'''Ubuntu'''

* Depending on your distribution, add either of the below lines to /etc/apt/sources.list.

deb http://download.meego.com/live/Tools:/Testing/xUbuntu_10.04/ /

deb http://download.meego.com/live/Tools:/Testing/xUbuntu_10.10/ /

* Add the repository keys

gpg --keyserver subkeys.pgp.net --recv-keys 0BC7BEC479FC1F8A
gpg --export --armor 0BC7BEC479FC1F8A | sudo apt-key add -

* Update the package list

sudo apt-get update

'''Fedora'''

* Add the repository

cd /etc/yum.repos.d/
sudo wget http://download.meego.com/live/Tools:/Testing/Fedora13/Tools:Testing.repo

* Add the repository keys

gpg --keyserver subkeys.pgp.net --recv-keys 0BC7BEC479FC1F8A
gpg --export --armor 0BC7BEC479FC1F8A | sudo apt-key add -

=== Install from MeeGo package repositories ===

'''Ubuntu'''

sudo apt-get install python-corelysis

'''Fedora'''

sudo yum install python-corelysis

=== Install from sources ===

* Get sources
git clone git://gitorious.org/meego-quality-assurance/corelysis.git

* Install dependencies

'''Ubuntu'''

sudo apt-get install python-setuptools python-pyinotify sp-rich-core-postproc

'''Fedora'''

sudo yum install python-setuptools python-inotify sp-rich-core-postproc

* Install everything

cd corelysis
sudo python setup.py install

=== Setting up cross gdb ===

* obtain cross gdb by e.g. installing [http://meego.com/downloads/releases/1.2/meego-v1.2-sdk meego SDK].
* the gdb needs to be statically linked, this can be achieved by [http://statifier.sourceforge.net statifier tool]
cd /opt/meego/meego-1.2-sdk-armv7hl-toolchain/arm-meego-linux-gnueabi/bin
statifier arm-meego-linux-gnueabi-gdb arm-meego-linux-gnueabi-gdb-static
* for the static binary to work, process address space randomization must be disabled
# echo 0 > /proc/sys/kernel/randomize_va_space

=== Executing Corelysis ===

sudo /usr/bin/corelysis

== Contact ==

In Corelysis related issues/ questions contact us in the following channels:

* [http://webchat.freenode.net/?channels=meego-qa #meego-qa IRC channel on irc.freenode.net]
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]

Quality/QA-tools/Corelysis

2011-06-15T07:03:10Z

Samposa: /* Install from sources */

= Corelysis =

== Introduction ==

Corelysis is a daemon service written in [http://www.python.org/ Python], which extracts, processes [[Quality/QA-tools/rich-core|rich core dumps]] from saved from MeeGo devices and finally publishes crash data in [[Quality/QA-tools/CrashReports|Crash Reports]]. Corelysis runs on back-end server and works closely with [[Quality/QA-tools/OTS|OTS]] service.

Corelysis has originated from [http://www.nokia.com Nokia], where it has been utilized for [http://maemo.org Maemo] related QA work. Later on Corelysis has been developed for [[ARM/N900|MeeGo N900 DeveloperEdition]] to report process crashes occured in automated test execution runs.

[[File:Text9867-0-4-8.png|600px|thumb|center|Automated testing and crash reporting]]

== How does it work? ==

The idea of Corelysis is quite simple. It takes a rich core dump as an input extracts the data from it and runs it through a set of "tasks", which each manipulate the data in different way. Maybe the most relevant task is the one which generates the backtrace from plain core dump. To be able to do this, Corelysis needs a fs-type meego image with corresponding debuginfo-packages installed. To generate the backtrace, a cross-compiled GDB is executed in chrooted target.

[[File:backend.png|600px|thumb|center|Corelysis work flow]]

== Installation ==

=== Preparation - Adding MeeGo Tools:Testing repository ===

'''Ubuntu'''

* Depending on your distribution, add either of the below lines to /etc/apt/sources.list.

deb http://download.meego.com/live/Tools:/Testing/xUbuntu_10.04/ /

deb http://download.meego.com/live/Tools:/Testing/xUbuntu_10.10/ /

* Add the repository keys

gpg --keyserver subkeys.pgp.net --recv-keys 0BC7BEC479FC1F8A
gpg --export --armor 0BC7BEC479FC1F8A | sudo apt-key add -

* Update the package list

sudo apt-get update

'''Fedora'''

* Add the repository

cd /etc/yum.repos.d/
sudo wget http://download.meego.com/live/Tools:/Testing/Fedora13/Tools:Testing.repo

* Add the repository keys

gpg --keyserver subkeys.pgp.net --recv-keys 0BC7BEC479FC1F8A
gpg --export --armor 0BC7BEC479FC1F8A | sudo apt-key add -

=== Install from MeeGo package repositories ===

'''Ubuntu'''

sudo apt-get install python-corelysis

'''Fedora'''

sudo yum install python-corelysis

=== Install from sources ===

* Get sources
git clone git://gitorious.org/meego-quality-assurance/corelysis.git

* Install dependencies

'''Ubuntu'''

sudo apt-get install python-setuptools python-pyinotify sp-rich-core-postproc

'''Fedora'''

sudo yum install python-setuptools python-inotify sp-rich-core-postproc

* Install everything

cd corelysis
sudo python setup.py install

=== Prepare environment ===

=== Executing Corelysis ===

sudo /usr/bin/corelysis

== Contact ==

In Corelysis related issues/ questions contact us in the following channels:

* [http://webchat.freenode.net/?channels=meego-qa #meego-qa IRC channel on irc.freenode.net]
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]

Quality/Meetings/QA leads update 1.2

2011-06-07T06:46:50Z

Samposa: /* QA-tools Update */

=== Core Update ===
31.05.2011
* Link to latest daily testing results: http://qa-reports.meego.com/1.2/Core/Acceptance/
* QA kicked off 1.3 daily validation on Netbook, the latest run rate is 78%, pass rate of total is 65%, pass rate of executed is 84%.
* MeeGo 1.2.80 image can be installed and live boot, bug(17737) has been fixed
** ''Key Issues:''
**17838 - [Trunk:Testing][REG] No help information print out when execute powertop -h o powertop --help
**13537 - A black screen displayed when running OpenGL-Blit case
**17835 - [Trunk:Testing][REG]Fail to run ut_qtcontacts_trackerplugin

10.05.2011
* Link to latest daily testing results: http://qa-reports.meego.com/1.2/Core/Acceptance/
Latest Acceptance iCDK:RR 95% PR:75; Netbook:RR:78% PR:68;IVI testing blocked by #17001
*Some regression issues were found in this week.
**17001 - [IVI][REG] missing meego-ux-ivi packages in ivi images（fixed in T:T）
**17065 - [REG]Error initializing garage client services library
**17248 - [IVI][REG][1.2:Testing] tear screen showed when unlocked the screen

06.06.2011
*Quality summary:
* No 1.3 images available during wk22.
* 1.2 Status:
* Acceptance and sanity tests done for 1.2
* Some test case tuning in acceptance test asset
*Test results for Core: http://qa-reports.meego.com/1.2/Core
*Latest N900 basic feature test result: http://qa-reports.meego.com/1.2/Core/Basic%20Feature%20Testing/N900/2635
*Latest acceptance results last Friday:Run Rate / Pass Rate of total: 60% / 42%

*DE testing:
** Not much testing done since no images available last week.
** Improved basic feature set taken into use.

=== Handset Update ===
06.06.2011
** Test results for Handset: http://qa-reports.meego.com/1.2/Handset/
* No images available last week for public Meego.
* In wk21 we tested upgrade release 1.2.1. Results in the QA reports.
* Open itm is what is the plan with these upgrade images. Was this one shot only or is there more to come?
* Blocking issues
**14135- [testing:daily]meegophotos crash (FIXED)
**15947- [META][REG][MTF]People, SMS, Dialer, Chat application fail to launch. (ASSIGNED)
**16664- No response when tapping "send a new message" button in sms application. (FIXED)
**17501- [Trunk:Testing] there is no audio in voice call. (FIXED)
**16943- [Trunk:Testing] Bluetooth cannot pair with headset. (NEW)
* Feature Verified status Meego 1.2:
** 11 Nokia committed features total
*** 3 Verified
*** 3 blocked by bugs
*** 3 blocked by missing implementation
*** 2 Failed

* Latest DE N900 test results:
** No DE images available last week.
** Next thing is to start concentrate and plan summer release testing.
** SF release tested wk21. Results are in QA reports. Also ET testing done for that.
*** Maturity status table updated according to results: http://wiki.meego.com/ARM/N900/Status
*** Blocking issues:
****16857- [DE] Email application is missing (FIXED)
****17844- [DE] there is no call duration while outgoing/incoming call. (NEW)
****12149- [REG]when adding more than one events,calendar application crashes (WAITING FOR UPSTREAM)
****17751- [DE] Unable to make conference call (NEW)
****18060- [DE]No browser shortcut on quicklaunch bar (WONTFIX)

=== Tablet Update ===
2011.05.24
* Weekly
** QA complete key feature testing for Pinetrail platform
*** Pass rate is 77%;
*** Improvement
**** Panels: fix four bug Bug 15154, Bug 16739, Bug 15502, Bug 16737;
**** Meego-ux-Components: for 0512 image, unable reproduce Bug 15835 - Unable to switch time format between 12hr and 24hr mode;
**** Setting: fix two Bug 16028 and Bug 15728;
*** Key Issue
**** Bug 15208 - Pinch zoom does not work
**** Bug 16735 - Unable to delete bookmark item
**** Bug 17670 - [Reg]Sometime crashed after input incorrect username and password via IM app
* Pinetrail Weekly Test Report: http://qa-reports.meego.com/1.2/Tablet/Key%20Feature/Pinetrail

=== Netbook Update ===
2011.05.24
* MeeGo 1.0 SW Update: No update yet.
* MeeGo 1.1 SW Update: QA Complete one round of sanity testing for 1.1 SW Update, and verified 15 bugs and reported 2 regression bugs.
* MeeGo 1.2 Netbook:
** MeeGo 1.2 is released in 19th May, 2011;
** To align with 1.2 release, QA started weekly system functional testing for Netbook until 1.2 released and completed 3 rounds of system functional testing, for 0503 image, pass rate is 86.78%, compared to 89% for 1.1 release and 92% for 1.1 Update 4 release; QA also completed one round of Chinese L10N testing on 0426 image and report 9 translation bugs.
** All approved release blocker bugs (19) are fixed and verified
* 1.2 System Functioanl test report for Netbook: http://qa-reports.meego.com/1.2/Netbook/System%20Function/Pinetrail
* Trunk test report for Netbook UX: http://qa-reports.meego.com/1.2/Netbook/Sanity/Pinetrail
* Trunk-tetsing report for Netbook UX: http://qa-reports.meego.com/1.2/Netbook/Acceptance/Pinetrail

=== IVI Update ===
31.05.2011
* For IVI, update for 1.2.0 will branch from 1.2.1, with limited bug fixing, but not all codes in 1.2.1.
* QA will focus on 1.2.0 update till end of Jun, and defer QA effort on 1.3 images before Jun 1.2.0.1 update
** ''Top Issues:''
*** [major] 17467 - IVI/Crossville-OKI: touchscreen NOT function
*** [major] 17502 - [IVI] boot time is too long ~35 second
*** [major] 16929 - IVI/Russellville: hang when switch from command line to X
*** [major] 17177 - [IVI]tear/black screen showed sometimes on LVDS'13
*** [major] 16997 - IVI: meego-ux-daemon crashes QT when screen is locked with EMGD
*** [major] 16760 - Russellville: /dev/radio0 not found
*** [major] 17135 - CrossvilleOKI: ioh_video_in initialize failed
*** [major] 16876 - [IVI] RV: GUI Installer takes a long time to launch
** See all open bugs about MeeGo 1.2 IVI: http://bugs.meego.com/buglist.cgi?emailcc1=1&classification=MeeGo%20Platform&emailtype1=substring&query_format=advanced&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&version=1.2&email1=ivi-bugs%40meego.com
** See all wontfix bugs of MeeGo 1.2 IVI: https://bugs.meego.com/buglist.cgi?resolution=WONTFIX&classification=MeeGo%20Platform&query_format=advanced&rep_platform=Automotive&version=1.2

=== SDK Update ===
10.05.2011

ARM -target:
* Test results for SDK : http://qa-reports.meego.com/1.2/SDK
* Latest results from week 18 using armv7hl handset sysroot [[http://repo.meego.com/MeeGo/builds/1.1.99/1.1.99.5.20110503.6/images/meego-handset-armv7hl-madde-sysroot/ 1.1.99.6.20110503.6]] and N900DE image.
** 17201 - libcloog package is missing
** 12853 - libattica is not installed with QtCreator
** 13171 - Qt-creator OBS plugin gives network error
** 14910 - Application launch on device is success but screen is distorded

=== QA-tools Update ===
2011.06.07
* testrunner-lite 1.7.0 released (closes 11 bugs)
* corelysis (the core processing backend) packaged and available from Tools:Testing
* [https://gitorious.org/meego-developer-edition-for-n900/mg-package-manager package manager] for the n900 DE released
2011.05.30
* OTS version 0.8.5 released
* corelysis open sourced
** Core dump processing component
* Core processing plugin for OTS
** Included in OTS 0.8.5 release
* Min test framework version 2011w20 released
* test-definition version 1.3.2 released
* Development focusing on improving test automation tool chain (OTS, testrunner-lite, etc.) and enabling automated core dump processing with OTS test runs.

Quality/Meetings/QA leads update 1.2

2011-06-07T06:46:34Z

Samposa: /* QA-tools Update */

=== Core Update ===
31.05.2011
* Link to latest daily testing results: http://qa-reports.meego.com/1.2/Core/Acceptance/
* QA kicked off 1.3 daily validation on Netbook, the latest run rate is 78%, pass rate of total is 65%, pass rate of executed is 84%.
* MeeGo 1.2.80 image can be installed and live boot, bug(17737) has been fixed
** ''Key Issues:''
**17838 - [Trunk:Testing][REG] No help information print out when execute powertop -h o powertop --help
**13537 - A black screen displayed when running OpenGL-Blit case
**17835 - [Trunk:Testing][REG]Fail to run ut_qtcontacts_trackerplugin

10.05.2011
* Link to latest daily testing results: http://qa-reports.meego.com/1.2/Core/Acceptance/
Latest Acceptance iCDK:RR 95% PR:75; Netbook:RR:78% PR:68;IVI testing blocked by #17001
*Some regression issues were found in this week.
**17001 - [IVI][REG] missing meego-ux-ivi packages in ivi images（fixed in T:T）
**17065 - [REG]Error initializing garage client services library
**17248 - [IVI][REG][1.2:Testing] tear screen showed when unlocked the screen

06.06.2011
*Quality summary:
* No 1.3 images available during wk22.
* 1.2 Status:
* Acceptance and sanity tests done for 1.2
* Some test case tuning in acceptance test asset
*Test results for Core: http://qa-reports.meego.com/1.2/Core
*Latest N900 basic feature test result: http://qa-reports.meego.com/1.2/Core/Basic%20Feature%20Testing/N900/2635
*Latest acceptance results last Friday:Run Rate / Pass Rate of total: 60% / 42%

*DE testing:
** Not much testing done since no images available last week.
** Improved basic feature set taken into use.

=== Handset Update ===
06.06.2011
** Test results for Handset: http://qa-reports.meego.com/1.2/Handset/
* No images available last week for public Meego.
* In wk21 we tested upgrade release 1.2.1. Results in the QA reports.
* Open itm is what is the plan with these upgrade images. Was this one shot only or is there more to come?
* Blocking issues
**14135- [testing:daily]meegophotos crash (FIXED)
**15947- [META][REG][MTF]People, SMS, Dialer, Chat application fail to launch. (ASSIGNED)
**16664- No response when tapping "send a new message" button in sms application. (FIXED)
**17501- [Trunk:Testing] there is no audio in voice call. (FIXED)
**16943- [Trunk:Testing] Bluetooth cannot pair with headset. (NEW)
* Feature Verified status Meego 1.2:
** 11 Nokia committed features total
*** 3 Verified
*** 3 blocked by bugs
*** 3 blocked by missing implementation
*** 2 Failed

* Latest DE N900 test results:
** No DE images available last week.
** Next thing is to start concentrate and plan summer release testing.
** SF release tested wk21. Results are in QA reports. Also ET testing done for that.
*** Maturity status table updated according to results: http://wiki.meego.com/ARM/N900/Status
*** Blocking issues:
****16857- [DE] Email application is missing (FIXED)
****17844- [DE] there is no call duration while outgoing/incoming call. (NEW)
****12149- [REG]when adding more than one events,calendar application crashes (WAITING FOR UPSTREAM)
****17751- [DE] Unable to make conference call (NEW)
****18060- [DE]No browser shortcut on quicklaunch bar (WONTFIX)

=== Tablet Update ===
2011.05.24
* Weekly
** QA complete key feature testing for Pinetrail platform
*** Pass rate is 77%;
*** Improvement
**** Panels: fix four bug Bug 15154, Bug 16739, Bug 15502, Bug 16737;
**** Meego-ux-Components: for 0512 image, unable reproduce Bug 15835 - Unable to switch time format between 12hr and 24hr mode;
**** Setting: fix two Bug 16028 and Bug 15728;
*** Key Issue
**** Bug 15208 - Pinch zoom does not work
**** Bug 16735 - Unable to delete bookmark item
**** Bug 17670 - [Reg]Sometime crashed after input incorrect username and password via IM app
* Pinetrail Weekly Test Report: http://qa-reports.meego.com/1.2/Tablet/Key%20Feature/Pinetrail

=== Netbook Update ===
2011.05.24
* MeeGo 1.0 SW Update: No update yet.
* MeeGo 1.1 SW Update: QA Complete one round of sanity testing for 1.1 SW Update, and verified 15 bugs and reported 2 regression bugs.
* MeeGo 1.2 Netbook:
** MeeGo 1.2 is released in 19th May, 2011;
** To align with 1.2 release, QA started weekly system functional testing for Netbook until 1.2 released and completed 3 rounds of system functional testing, for 0503 image, pass rate is 86.78%, compared to 89% for 1.1 release and 92% for 1.1 Update 4 release; QA also completed one round of Chinese L10N testing on 0426 image and report 9 translation bugs.
** All approved release blocker bugs (19) are fixed and verified
* 1.2 System Functioanl test report for Netbook: http://qa-reports.meego.com/1.2/Netbook/System%20Function/Pinetrail
* Trunk test report for Netbook UX: http://qa-reports.meego.com/1.2/Netbook/Sanity/Pinetrail
* Trunk-tetsing report for Netbook UX: http://qa-reports.meego.com/1.2/Netbook/Acceptance/Pinetrail

=== IVI Update ===
31.05.2011
* For IVI, update for 1.2.0 will branch from 1.2.1, with limited bug fixing, but not all codes in 1.2.1.
* QA will focus on 1.2.0 update till end of Jun, and defer QA effort on 1.3 images before Jun 1.2.0.1 update
** ''Top Issues:''
*** [major] 17467 - IVI/Crossville-OKI: touchscreen NOT function
*** [major] 17502 - [IVI] boot time is too long ~35 second
*** [major] 16929 - IVI/Russellville: hang when switch from command line to X
*** [major] 17177 - [IVI]tear/black screen showed sometimes on LVDS'13
*** [major] 16997 - IVI: meego-ux-daemon crashes QT when screen is locked with EMGD
*** [major] 16760 - Russellville: /dev/radio0 not found
*** [major] 17135 - CrossvilleOKI: ioh_video_in initialize failed
*** [major] 16876 - [IVI] RV: GUI Installer takes a long time to launch
** See all open bugs about MeeGo 1.2 IVI: http://bugs.meego.com/buglist.cgi?emailcc1=1&classification=MeeGo%20Platform&emailtype1=substring&query_format=advanced&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&version=1.2&email1=ivi-bugs%40meego.com
** See all wontfix bugs of MeeGo 1.2 IVI: https://bugs.meego.com/buglist.cgi?resolution=WONTFIX&classification=MeeGo%20Platform&query_format=advanced&rep_platform=Automotive&version=1.2

=== SDK Update ===
10.05.2011

ARM -target:
* Test results for SDK : http://qa-reports.meego.com/1.2/SDK
* Latest results from week 18 using armv7hl handset sysroot [[http://repo.meego.com/MeeGo/builds/1.1.99/1.1.99.5.20110503.6/images/meego-handset-armv7hl-madde-sysroot/ 1.1.99.6.20110503.6]] and N900DE image.
** 17201 - libcloog package is missing
** 12853 - libattica is not installed with QtCreator
** 13171 - Qt-creator OBS plugin gives network error
** 14910 - Application launch on device is success but screen is distorded

=== QA-tools Update ===
2011.06.07
* testrunner-lite 1.7.0 released (closes 11 bugs)
* corelysis (the core processing backend) packaged and available from Tools:Testing
* [https://gitorious.org/meego-developer-edition-for-n900/mg-package-manager package manager] package manager for the n900 DE released
2011.05.30
* OTS version 0.8.5 released
* corelysis open sourced
** Core dump processing component
* Core processing plugin for OTS
** Included in OTS 0.8.5 release
* Min test framework version 2011w20 released
* test-definition version 1.3.2 released
* Development focusing on improving test automation tool chain (OTS, testrunner-lite, etc.) and enabling automated core dump processing with OTS test runs.

Quality/Meetings/QA leads update 1.2

2011-06-07T06:46:14Z

Samposa: /* QA-tools Update */

=== Core Update ===
31.05.2011
* Link to latest daily testing results: http://qa-reports.meego.com/1.2/Core/Acceptance/
* QA kicked off 1.3 daily validation on Netbook, the latest run rate is 78%, pass rate of total is 65%, pass rate of executed is 84%.
* MeeGo 1.2.80 image can be installed and live boot, bug(17737) has been fixed
** ''Key Issues:''
**17838 - [Trunk:Testing][REG] No help information print out when execute powertop -h o powertop --help
**13537 - A black screen displayed when running OpenGL-Blit case
**17835 - [Trunk:Testing][REG]Fail to run ut_qtcontacts_trackerplugin

10.05.2011
* Link to latest daily testing results: http://qa-reports.meego.com/1.2/Core/Acceptance/
Latest Acceptance iCDK:RR 95% PR:75; Netbook:RR:78% PR:68;IVI testing blocked by #17001
*Some regression issues were found in this week.
**17001 - [IVI][REG] missing meego-ux-ivi packages in ivi images（fixed in T:T）
**17065 - [REG]Error initializing garage client services library
**17248 - [IVI][REG][1.2:Testing] tear screen showed when unlocked the screen

06.06.2011
*Quality summary:
* No 1.3 images available during wk22.
* 1.2 Status:
* Acceptance and sanity tests done for 1.2
* Some test case tuning in acceptance test asset
*Test results for Core: http://qa-reports.meego.com/1.2/Core
*Latest N900 basic feature test result: http://qa-reports.meego.com/1.2/Core/Basic%20Feature%20Testing/N900/2635
*Latest acceptance results last Friday:Run Rate / Pass Rate of total: 60% / 42%

*DE testing:
** Not much testing done since no images available last week.
** Improved basic feature set taken into use.

=== Handset Update ===
06.06.2011
** Test results for Handset: http://qa-reports.meego.com/1.2/Handset/
* No images available last week for public Meego.
* In wk21 we tested upgrade release 1.2.1. Results in the QA reports.
* Open itm is what is the plan with these upgrade images. Was this one shot only or is there more to come?
* Blocking issues
**14135- [testing:daily]meegophotos crash (FIXED)
**15947- [META][REG][MTF]People, SMS, Dialer, Chat application fail to launch. (ASSIGNED)
**16664- No response when tapping "send a new message" button in sms application. (FIXED)
**17501- [Trunk:Testing] there is no audio in voice call. (FIXED)
**16943- [Trunk:Testing] Bluetooth cannot pair with headset. (NEW)
* Feature Verified status Meego 1.2:
** 11 Nokia committed features total
*** 3 Verified
*** 3 blocked by bugs
*** 3 blocked by missing implementation
*** 2 Failed

* Latest DE N900 test results:
** No DE images available last week.
** Next thing is to start concentrate and plan summer release testing.
** SF release tested wk21. Results are in QA reports. Also ET testing done for that.
*** Maturity status table updated according to results: http://wiki.meego.com/ARM/N900/Status
*** Blocking issues:
****16857- [DE] Email application is missing (FIXED)
****17844- [DE] there is no call duration while outgoing/incoming call. (NEW)
****12149- [REG]when adding more than one events,calendar application crashes (WAITING FOR UPSTREAM)
****17751- [DE] Unable to make conference call (NEW)
****18060- [DE]No browser shortcut on quicklaunch bar (WONTFIX)

=== Tablet Update ===
2011.05.24
* Weekly
** QA complete key feature testing for Pinetrail platform
*** Pass rate is 77%;
*** Improvement
**** Panels: fix four bug Bug 15154, Bug 16739, Bug 15502, Bug 16737;
**** Meego-ux-Components: for 0512 image, unable reproduce Bug 15835 - Unable to switch time format between 12hr and 24hr mode;
**** Setting: fix two Bug 16028 and Bug 15728;
*** Key Issue
**** Bug 15208 - Pinch zoom does not work
**** Bug 16735 - Unable to delete bookmark item
**** Bug 17670 - [Reg]Sometime crashed after input incorrect username and password via IM app
* Pinetrail Weekly Test Report: http://qa-reports.meego.com/1.2/Tablet/Key%20Feature/Pinetrail

=== Netbook Update ===
2011.05.24
* MeeGo 1.0 SW Update: No update yet.
* MeeGo 1.1 SW Update: QA Complete one round of sanity testing for 1.1 SW Update, and verified 15 bugs and reported 2 regression bugs.
* MeeGo 1.2 Netbook:
** MeeGo 1.2 is released in 19th May, 2011;
** To align with 1.2 release, QA started weekly system functional testing for Netbook until 1.2 released and completed 3 rounds of system functional testing, for 0503 image, pass rate is 86.78%, compared to 89% for 1.1 release and 92% for 1.1 Update 4 release; QA also completed one round of Chinese L10N testing on 0426 image and report 9 translation bugs.
** All approved release blocker bugs (19) are fixed and verified
* 1.2 System Functioanl test report for Netbook: http://qa-reports.meego.com/1.2/Netbook/System%20Function/Pinetrail
* Trunk test report for Netbook UX: http://qa-reports.meego.com/1.2/Netbook/Sanity/Pinetrail
* Trunk-tetsing report for Netbook UX: http://qa-reports.meego.com/1.2/Netbook/Acceptance/Pinetrail

=== IVI Update ===
31.05.2011
* For IVI, update for 1.2.0 will branch from 1.2.1, with limited bug fixing, but not all codes in 1.2.1.
* QA will focus on 1.2.0 update till end of Jun, and defer QA effort on 1.3 images before Jun 1.2.0.1 update
** ''Top Issues:''
*** [major] 17467 - IVI/Crossville-OKI: touchscreen NOT function
*** [major] 17502 - [IVI] boot time is too long ~35 second
*** [major] 16929 - IVI/Russellville: hang when switch from command line to X
*** [major] 17177 - [IVI]tear/black screen showed sometimes on LVDS'13
*** [major] 16997 - IVI: meego-ux-daemon crashes QT when screen is locked with EMGD
*** [major] 16760 - Russellville: /dev/radio0 not found
*** [major] 17135 - CrossvilleOKI: ioh_video_in initialize failed
*** [major] 16876 - [IVI] RV: GUI Installer takes a long time to launch
** See all open bugs about MeeGo 1.2 IVI: http://bugs.meego.com/buglist.cgi?emailcc1=1&classification=MeeGo%20Platform&emailtype1=substring&query_format=advanced&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&version=1.2&email1=ivi-bugs%40meego.com
** See all wontfix bugs of MeeGo 1.2 IVI: https://bugs.meego.com/buglist.cgi?resolution=WONTFIX&classification=MeeGo%20Platform&query_format=advanced&rep_platform=Automotive&version=1.2

=== SDK Update ===
10.05.2011

ARM -target:
* Test results for SDK : http://qa-reports.meego.com/1.2/SDK
* Latest results from week 18 using armv7hl handset sysroot [[http://repo.meego.com/MeeGo/builds/1.1.99/1.1.99.5.20110503.6/images/meego-handset-armv7hl-madde-sysroot/ 1.1.99.6.20110503.6]] and N900DE image.
** 17201 - libcloog package is missing
** 12853 - libattica is not installed with QtCreator
** 13171 - Qt-creator OBS plugin gives network error
** 14910 - Application launch on device is success but screen is distorded

=== QA-tools Update ===
2011.06.07
* testrunner-lite 1.7.0 released (closes 11 bugs)
* corelysis (the core processing backend) packaged and available from Tools:Testing
* [https://gitorious.org/meego-developer-edition-for-n900/mg-package-manager package manager]package manager for the n900 DE released
2011.05.30
* OTS version 0.8.5 released
* corelysis open sourced
** Core dump processing component
* Core processing plugin for OTS
** Included in OTS 0.8.5 release
* Min test framework version 2011w20 released
* test-definition version 1.3.2 released
* Development focusing on improving test automation tool chain (OTS, testrunner-lite, etc.) and enabling automated core dump processing with OTS test runs.

Quality/Meetings/QA leads update 1.2

2011-06-07T06:26:51Z

Samposa: /* QA-tools Update */

=== Core Update ===
31.05.2011
* Link to latest daily testing results: http://qa-reports.meego.com/1.2/Core/Acceptance/
* QA kicked off 1.3 daily validation on Netbook, the latest run rate is 78%, pass rate of total is 65%, pass rate of executed is 84%.
* MeeGo 1.2.80 image can be installed and live boot, bug(17737) has been fixed
** ''Key Issues:''
**17838 - [Trunk:Testing][REG] No help information print out when execute powertop -h o powertop --help
**13537 - A black screen displayed when running OpenGL-Blit case
**17835 - [Trunk:Testing][REG]Fail to run ut_qtcontacts_trackerplugin

10.05.2011
* Link to latest daily testing results: http://qa-reports.meego.com/1.2/Core/Acceptance/
Latest Acceptance iCDK:RR 95% PR:75; Netbook:RR:78% PR:68;IVI testing blocked by #17001
*Some regression issues were found in this week.
**17001 - [IVI][REG] missing meego-ux-ivi packages in ivi images（fixed in T:T）
**17065 - [REG]Error initializing garage client services library
**17248 - [IVI][REG][1.2:Testing] tear screen showed when unlocked the screen

06.06.2011
*Quality summary:
* No 1.3 images available during wk22.
* 1.2 Status:
* Acceptance and sanity tests done for 1.2
* Some test case tuning in acceptance test asset
*Test results for Core: http://qa-reports.meego.com/1.2/Core
*Latest N900 basic feature test result: http://qa-reports.meego.com/1.2/Core/Basic%20Feature%20Testing/N900/2635
*Latest acceptance results last Friday:Run Rate / Pass Rate of total: 60% / 42%

*DE testing:
** Not much testing done since no images available last week.
** Improved basic feature set taken into use.

=== Handset Update ===
06.06.2011
** Test results for Handset: http://qa-reports.meego.com/1.2/Handset/
* No images available last week for public Meego.
* In wk21 we tested upgrade release 1.2.1. Results in the QA reports.
* Open itm is what is the plan with these upgrade images. Was this one shot only or is there more to come?
* Blocking issues
**14135- [testing:daily]meegophotos crash (FIXED)
**15947- [META][REG][MTF]People, SMS, Dialer, Chat application fail to launch. (ASSIGNED)
**16664- No response when tapping "send a new message" button in sms application. (FIXED)
**17501- [Trunk:Testing] there is no audio in voice call. (FIXED)
**16943- [Trunk:Testing] Bluetooth cannot pair with headset. (NEW)
* Feature Verified status Meego 1.2:
** 11 Nokia committed features total
*** 3 Verified
*** 3 blocked by bugs
*** 3 blocked by missing implementation
*** 2 Failed

* Latest DE N900 test results:
** No DE images available last week.
** Next thing is to start concentrate and plan summer release testing.
** SF release tested wk21. Results are in QA reports. Also ET testing done for that.
*** Maturity status table updated according to results: http://wiki.meego.com/ARM/N900/Status
*** Blocking issues:
****16857- [DE] Email application is missing (FIXED)
****17844- [DE] there is no call duration while outgoing/incoming call. (NEW)
****12149- [REG]when adding more than one events,calendar application crashes (WAITING FOR UPSTREAM)
****17751- [DE] Unable to make conference call (NEW)
****18060- [DE]No browser shortcut on quicklaunch bar (WONTFIX)

=== Tablet Update ===
2011.05.24
* Weekly
** QA complete key feature testing for Pinetrail platform
*** Pass rate is 77%;
*** Improvement
**** Panels: fix four bug Bug 15154, Bug 16739, Bug 15502, Bug 16737;
**** Meego-ux-Components: for 0512 image, unable reproduce Bug 15835 - Unable to switch time format between 12hr and 24hr mode;
**** Setting: fix two Bug 16028 and Bug 15728;
*** Key Issue
**** Bug 15208 - Pinch zoom does not work
**** Bug 16735 - Unable to delete bookmark item
**** Bug 17670 - [Reg]Sometime crashed after input incorrect username and password via IM app
* Pinetrail Weekly Test Report: http://qa-reports.meego.com/1.2/Tablet/Key%20Feature/Pinetrail

=== Netbook Update ===
2011.05.24
* MeeGo 1.0 SW Update: No update yet.
* MeeGo 1.1 SW Update: QA Complete one round of sanity testing for 1.1 SW Update, and verified 15 bugs and reported 2 regression bugs.
* MeeGo 1.2 Netbook:
** MeeGo 1.2 is released in 19th May, 2011;
** To align with 1.2 release, QA started weekly system functional testing for Netbook until 1.2 released and completed 3 rounds of system functional testing, for 0503 image, pass rate is 86.78%, compared to 89% for 1.1 release and 92% for 1.1 Update 4 release; QA also completed one round of Chinese L10N testing on 0426 image and report 9 translation bugs.
** All approved release blocker bugs (19) are fixed and verified
* 1.2 System Functioanl test report for Netbook: http://qa-reports.meego.com/1.2/Netbook/System%20Function/Pinetrail
* Trunk test report for Netbook UX: http://qa-reports.meego.com/1.2/Netbook/Sanity/Pinetrail
* Trunk-tetsing report for Netbook UX: http://qa-reports.meego.com/1.2/Netbook/Acceptance/Pinetrail

=== IVI Update ===
31.05.2011
* For IVI, update for 1.2.0 will branch from 1.2.1, with limited bug fixing, but not all codes in 1.2.1.
* QA will focus on 1.2.0 update till end of Jun, and defer QA effort on 1.3 images before Jun 1.2.0.1 update
** ''Top Issues:''
*** [major] 17467 - IVI/Crossville-OKI: touchscreen NOT function
*** [major] 17502 - [IVI] boot time is too long ~35 second
*** [major] 16929 - IVI/Russellville: hang when switch from command line to X
*** [major] 17177 - [IVI]tear/black screen showed sometimes on LVDS'13
*** [major] 16997 - IVI: meego-ux-daemon crashes QT when screen is locked with EMGD
*** [major] 16760 - Russellville: /dev/radio0 not found
*** [major] 17135 - CrossvilleOKI: ioh_video_in initialize failed
*** [major] 16876 - [IVI] RV: GUI Installer takes a long time to launch
** See all open bugs about MeeGo 1.2 IVI: http://bugs.meego.com/buglist.cgi?emailcc1=1&classification=MeeGo%20Platform&emailtype1=substring&query_format=advanced&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&version=1.2&email1=ivi-bugs%40meego.com
** See all wontfix bugs of MeeGo 1.2 IVI: https://bugs.meego.com/buglist.cgi?resolution=WONTFIX&classification=MeeGo%20Platform&query_format=advanced&rep_platform=Automotive&version=1.2

=== SDK Update ===
10.05.2011

ARM -target:
* Test results for SDK : http://qa-reports.meego.com/1.2/SDK
* Latest results from week 18 using armv7hl handset sysroot [[http://repo.meego.com/MeeGo/builds/1.1.99/1.1.99.5.20110503.6/images/meego-handset-armv7hl-madde-sysroot/ 1.1.99.6.20110503.6]] and N900DE image.
** 17201 - libcloog package is missing
** 12853 - libattica is not installed with QtCreator
** 13171 - Qt-creator OBS plugin gives network error
** 14910 - Application launch on device is success but screen is distorded

=== QA-tools Update ===
2011.06.07
* testrunner-lite 1.7.0 released (closes 11 bugs)
* corelysis (the core processing backend) packaged and available from Tools:Testing
2011.05.30
* OTS version 0.8.5 released
* corelysis open sourced
** Core dump processing component
* Core processing plugin for OTS
** Included in OTS 0.8.5 release
* Min test framework version 2011w20 released
* test-definition version 1.3.2 released
* Development focusing on improving test automation tool chain (OTS, testrunner-lite, etc.) and enabling automated core dump processing with OTS test runs.

ARM/N900/QA

2011-05-16T11:06:39Z

Samposa: /* Crash analysis support for ARM core dumps */

= General info =
Quality Assurance for Meego Developers Edition.

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Data%20Flow%20For%20Developer%20Edition Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Use%20Case%20Testing%20For%20Developer%20Edition Use case reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== OTS setup and automated hourly testing ==

=== OTS setup ===
* [http://194.136.64.78/logger/view/ OTS server] - Ville Ilvonen/Riku Halonen, DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
* Minimize automatic installation time to 10mins - Timo Makimattila, DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)

* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)

* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - ONGOING (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- ONGOING (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - NOT STARTED
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - NOT STARTED

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]

= Core Testing =
== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* Bug verification

==== Backlog ====
* Testcase automation list

==== In progress ====
* DE Hourly Automation improvement
* Wiki clean/update (waiting for comments)

==== Done ====
* * Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.2(Tu Qingqing)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
|Monday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Monday, Tuesday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Wednesday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Wednesday, Thursday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.2 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script

==== In progress ====
* MeeGo1.2 feature verification
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-05-16T11:05:46Z

Samposa: /* Crash analysis support for ARM core dumps */

= General info =
Quality Assurance for Meego Developers Edition.

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Data%20Flow%20For%20Developer%20Edition Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Use%20Case%20Testing%20For%20Developer%20Edition Use case reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== OTS setup and automated hourly testing ==

=== OTS setup ===
* [http://194.136.64.78/logger/view/ OTS server] - Ville Ilvonen/Riku Halonen, DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
* Minimize automatic installation time to 10mins - Timo Makimattila, DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)

* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)

* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - ONGOING (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- ONGOING (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - NOT STARTED
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - Tested manually, untested python script exists (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - NOT STARTED

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]

= Core Testing =
== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* Bug verification

==== Backlog ====
* Testcase automation list

==== In progress ====
* DE Hourly Automation improvement
* Wiki clean/update (waiting for comments)

==== Done ====
* * Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.2(Tu Qingqing)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
|Monday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Monday, Tuesday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Wednesday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Wednesday, Thursday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.2 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script

==== In progress ====
* MeeGo1.2 feature verification
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-05-16T11:04:33Z

Samposa: /* Crash analysis support for ARM core dumps */

= General info =
Quality Assurance for Meego Developers Edition.

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Data%20Flow%20For%20Developer%20Edition Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Use%20Case%20Testing%20For%20Developer%20Edition Use case reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== OTS setup and automated hourly testing ==

=== OTS setup ===
* [http://194.136.64.78/logger/view/ OTS server] - Ville Ilvonen/Riku Halonen, DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
* Minimize automatic installation time to 10mins - Timo Makimattila, DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)

* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)

* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - ONGOING (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- ONGOING (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - NOT STARTED
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.

<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

[[File:backend.png|350px|thumb|right|Core Processing Backend]]
* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - Tested manually, untested python script exists (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - NOT STARTED

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]

= Core Testing =
== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* Bug verification

==== Backlog ====
* Testcase automation list

==== In progress ====
* DE Hourly Automation improvement
* Wiki clean/update (waiting for comments)

==== Done ====
* * Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.2(Tu Qingqing)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
|Monday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Monday, Tuesday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Wednesday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Wednesday, Thursday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.2 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script

==== In progress ====
* MeeGo1.2 feature verification
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-05-16T11:04:00Z

Samposa: /* Crash analysis support for ARM core dumps */

= General info =
Quality Assurance for Meego Developers Edition.

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Data%20Flow%20For%20Developer%20Edition Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Use%20Case%20Testing%20For%20Developer%20Edition Use case reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== OTS setup and automated hourly testing ==

=== OTS setup ===
* [http://194.136.64.78/logger/view/ OTS server] - Ville Ilvonen/Riku Halonen, DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
* Minimize automatic installation time to 10mins - Timo Makimattila, DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)

* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)

* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - ONGOING (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- ONGOING (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - NOT STARTED
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.

<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - Tested manually, untested python script exists (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - NOT STARTED

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]

= Core Testing =
== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* Bug verification

==== Backlog ====
* Testcase automation list

==== In progress ====
* DE Hourly Automation improvement
* Wiki clean/update (waiting for comments)

==== Done ====
* * Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.2(Tu Qingqing)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
|Monday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Monday, Tuesday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Wednesday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Wednesday, Thursday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.2 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script

==== In progress ====
* MeeGo1.2 feature verification
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

File:Backend.png

2011-05-16T11:00:32Z

Samposa: Core processing backend functionality

Core processing backend functionality

ARM/N900/QA

2011-04-19T05:55:17Z

Samposa: /* OTS setup */

= QA TODOs (in priority order) =

== MeeGo.com IRC meetings ==

There are no special QA meeting, other DE team meetings can be found from here:
* [[MeeGo-Meeting_IRC_Schedule|MeeGo-Meeting IRC Schedule]]

== OTS setup and automated hourly testing ==

=== OTS setup ===
* [http://194.136.64.78/logger/view/ OTS server] - Ville Ilvonen/Riku Halonen, DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
* Minimize automatic installation time to 10mins - Timo Makimattila, ONGOING
* OTS worker for UX tests - NOTSTARTED
* Worker(s) for DE tests - NOTSTARTED
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)

* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)

* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

Core dump processing and backtraces from crashing ARM processes.

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (Something goes wrong, when reducer processes coredump.) - DONE (sampos)
** Oopslog (and lifelog) functionality - NOT STARTED

* Back-end server
** Set up back-end server for core processing - ONGOING (rikhalon, sampos)

* Script to produce backtraces locally
** Extend rich-core-extract to proceduce backtrace from rich-core file - NOT STARTED
** Write howto guide in wiki - NOT STARTED

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

== Test Execution Schedule ==
* Core (Teivas)
* Handset UX weekly testing schedule (Rajala)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Key feature (OK)
| Preview
|-
| Monday
| Acceptance (OK)
| Testing trunk
|-
|Tuesday
|Acceptance (Ok)
|Testing trunk
|-
|Tuesday
|Sanity (Ok)
|Daily trunk
|-
|Tuesday
|DE Dataflow (Ok)
|Preview
|-
|Tuesday
|DE use cases (Ok)
|Preview
|-
|Wednesday
|Acceptance (Ok)
|Testing trunk
|-
|Wednesday
|Key feature (Ok)
|Weekly
|-
|Thursday
|DE Dataflow (Ok)
|Weekly
|-
|Thursday
|DE use cases (Ok)
|Weekly
|-
|Thursday
|Acceptance (Ok)
|Testing trunk
|-
|Thursday
|Sanity Ok
|Daily trunk
|-
|Thursday
|DE Reliability (Ongoing)
|Weekly
|-
|Thursday
|DE Performance (Ongoing)
|Weekly
|-
|Friday
|Acceptance (Ok)
|Testing trunk
|-
|Friday
|Sanity (Ok)
|Daily trunk
|}

== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* Bug verification

==== Backlog ====
* Maturity statement of Alpha RELEASE
* Wiki clean/update
* Plan Feature testset

==== In progress ====
* Alpha RELEASE testing
* Create weekly schedule for MRT (currently in draft form)
* DE Hourly Automation improvement

==== Done ====
* Week 15 DE Sanity Testing

== QA Organization ==

== Error Management ==
* Error Manager Iekku Huttunen

N900 DE Blocker Bug Triage meeting minutes:
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.2(Tu Qingqing)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Monday
| Basic feature test(OK)
| Preview
| P2
|-
| Monday
| Sanity (OK)
| Trunk
| P3
|-
| Tuesday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Tuesday
| Sanity (OK)
| Trunk
| P2
|-
| Wednesday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Wednesday
| Data Flow(OK)
| Weekly
| P2
|-
| Wednesday
| Sanity (OK)
| Trunk
| P3
|-
| Thursday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Thursday
| Sanity (OK)
| Trunk
| P2
|-
| Friday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Friday
| Sanity (OK)
| Trunk
| P2
|-
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.2 N900 daily validation

==== Backlog ====
* Automation testing for trunk:test and trunk image

==== In progress ====
* MeeGo1.2 feature verification
* Publish automation test result

==== Done ====

== Usefull links ==

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-04-04T09:09:00Z

Samposa: /* Crashdb support for ARM core dumps */

User:Samposa

2011-03-24T10:39:03Z

Samposa: /* TODO */

== Getting sp-rich-core running in meego/n900==

=== get packages ===
* from devel:quality
** sp-endurance-2.2.1-1.1.armv7l.rpm
** lzop-1.03-1.29.armv7l.rpm
* from http://download.meego.com/live/home:/samposa/testing/armv7l/
** sp-rich-core-1.65.1-66.1.armv7l.rpm
** core-reducer-1.65.1-66.1.armv7l.rpm

=== install ===
* zypper install lzo
* zypper install libXres
* rpm -i sp-endurance-2.2.1-1.1.armv7l.rpm
* rpm -i sp-rich-core-1.65.1-66.1.armv7l.rpm core-reducer-1.65.1-66.1.armv7l.rpm lzop-1.03-1.29.armv7l.rpm

=== reboot ===
* we have our core_pattern in place \o/
* mkdir /home/meego/core-dumps

=== make it crash ===
* email->settings->edit sms account
-> bum -> we have rich-core in /home/meego/core-dumps

== TODO ==
* conflicts with corewatcher
* did not try core-reducing (imo can leave that ffs)
* only rich_core_pattern called from init atm (hence needed to create /home/meego/coredumps manually). IMO we could add the folder creation to rich-core-pattern and live with only one init script.
* imei obtaining (ffs also)
* did some dirtyish things in the spec file to get it working ...

User:Samposa

2011-03-24T10:38:26Z

Samposa: Created page with "== Getting sp-rich-core running in meego/n900== === get packages === * from devel:quality ** sp-endurance-2.2.1-1.1.armv7l.rpm ** lzop-1.03-1.29.armv7l.rpm * from http://downloa…"

ARM/N900/QA

2011-03-23T08:39:30Z

Samposa: /* Crashdb support for ARM core dumps */

= QA TODOs (in priority order) =

== OTS setup and automated hourly testing ==

=== OTS setup ===
* OTS server - Ville Ilvonen/Riku Halonen, DONE
* OTS worker(s) for core tests - Ville Ilvonen/Riku Halonen/Timo Harkonen - ONGOING (2 workers up)
* OTS worker for UX tests
* Power consumption measurements
* Reporting to QA-reports
* Minimize automatic installation time to 10mins - Timo Makimattila, DONE

=== Test automation images ===
* Setup hourly image building for autotest image on own setup (maybe later with BOSS) - Riku Halonen ONGOING (setting up F13 machine for image creation)
* We need to be able to control included test packages

If you need something else from QA please tell it to us :)

== Applications for testing ==
* Package QML demo/example applications from http://doc.qt.nokia.com/latest/qdeclarativeexamples.html

== Test asset ==
* Define what the asset includes
* Automating as many core/ux tests as possible
* Call/SMS cases will require parallel test execution support (If we want to automate those kind of tests)
* WLAN cases
* Audio policy framework cases (lower priority)
* Camera cases (lower priority)
* Sensor data cases (Qt Mobility, lower priority)

== Crashdb support for ARM core dumps ==

* Core dump processing and backtraces from crashing ARM processes.
** looking into rich-core use in MeeGo (sampos)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Test Execution Schedule ==
* Core

* Handset UX weekly testing schedule
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Key feature (OK)
| Preview
|-
| Monday
| Acceptance (OK)
| Testing trunk
|-
|Tuesday
|Acceptance (Ok)
|Testing trunk
|-
|Tuesday
|Sanity (Ok)
|Daily trunk
|-
|Tuesday
|DE Dataflow (Ok)
|Preview
|-
|Tuesday
|DE use cases (Ok)
|Preview
|-
|Wednesday
|Acceptance (Ok)
|Testing trunk
|-
|Wednesday
|Key feature (Ok)
|Weekly
|-
|Thursday
|DE Dataflow (Ok)
|Weekly
|-
|Thursday
|DE use cases (Ok)
|Weekly
|-
|Thursday
|Acceptance (Ok)
|Testing trunk
|-
|Thursday
|Sanity Ok
|Daily trunk
|-
|Thursday
|DE Reliability (Ongoing)
|Weekly
|-
|Thursday
|DE Performance (Ongoing)
|Weekly
|-
|Friday
|Acceptance (Ok)
|Testing trunk
|-
|Friday
|Sanity (Ok)
|Daily trunk
|}

== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Sanity Test Set ===
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features

=== Feature Test Set ===
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

== Error Management ==
* Error Manager Iekku Huttunen

== Usefull links ==

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

Local MeeGo Networks/MeeGo Network Finland/MeeGo Summit FI

2011-03-22T10:01:31Z

Samposa: /* Hack3r Team */

<div style="float:right;background-color:#fff;width:400px;height:520px;border: solid 1px #ddd;padding:15px;font-size:1.3em;"><b>Sponsors:</b><ul style="list-style:none;"><li>[[File:Ixonos-logo.gif]] (Platinum)</li><li>[[File:Tieto.jpg]] (Gold)</li><li>[[File:Intel-logo.png‎]] (Gold)</li><li>[[File:futurice_logo.png]] (Silver)</li><li>[[File:ICS-logo.gif‎]] (Silver)</li><li>[[File:Nomovok.png‎]] (Silver)</li><li>[[File:Symbio.png‎]] (Silver)</li></ul></div>

= Developer to developer =
Tampere initiated the first [[Local_MeeGo_Networks|Local Meego Network]] in Europe, [http://www.meetup.com/Tampere-MeeGo-Network/ Tampere Meego Network]. This is annual summit which is held in different locations around Finland. Tampere will host the first Regional Meego Summit in the world. Although the first Summit is at Tampere, this is a joined effort.

<div style="float:right;background-color:#fff;width:280px;height:160px;border: solid 1px #ddd;padding:15px;font-size:1.3em;">[[Image:MeeGoSummitNetworkLogo-RGB-72_Summit-01.jpg‎|link=http://summit.meegonetwork.fi]]</div>

'''[http://www.meetup.com/Tampere-MeeGo-Network/ Tampere] and [http://www.meetup.com/Helsinki-MeeGo-Network/ Helsinki] local networks have joined forces and other local networks are also welcome to participate! Furthermore, this conference will host Finnish hacker & OSS event ''Finhack'' (16th Apr) '''

The summit is free of charge, but the amount of seats is limited! Program includes:
* Keynotes
* Three tracks
* Meegathon competition
* Party
* BoFs & Workshops
* Quadricopter fun

== Details ==
* Location: Tampere, Finland. Finlayson area (Plevna, [http://www.uusitehdas.fi/en/ New Factory])
* Time: 15.-16. April 2011
* Duration: 2 days (Fri-Sat)
* Registration fee: Free!
* Seat amount: max 300
* Registration required: yes (address here)
* Web: http://summit.meegonetwork.fi
* '''[[Local_MeeGo_Networks/MeeGo_Network_Finland/roadmap | Roadmap/important dates]]'''

== Organizers ==
'''Main organizer:'''
{| class="wikitable sortable" border="1" cellpadding="3" cellspacing="0" style="width:800px;"
|-
!width="25%"|Contact Person(s)
!width="25%"| Sector
!width="25%"| Role
!width="25%"| Organization
|-
| Ilkka Lehtinen
| General, sponsors
| Head
| [http://www.coss.fi/en COSS]
|}

'''Co-Organizers:'''

{| class="wikitable sortable" border="1" cellpadding="3" cellspacing="0" style="width:800px;"
|-
!width="25%"| Contact Person(s)
!width="25%"| Sector
!width="25%"| Role
!width="25%"| Organization
|-
| Jarkko Moilanen
| General + Program (Community & hackers)
| Organizer
| [http://www.oske.net/en/ OSKE]
|-
| Jukka Matikainen
| ?
| Organizer
| [http://www.oske.net/en/ OSKE]
|-
| Matti Saastamoinen
| General, sponsors
| Organizer
| [http://www.coss.fi/en COSS]
|-
| Ville Kairamo
| Facilities
| Organizer
| [http://www.uusitehdas.fi/en NewFactory]
|}

== Teams ==
MNFI community needs different type of teams and task-forces to arrange activities. Members of the teams should be from the member networks but foreign members are also welcome. Some of the teams are active throughout the year, but some are active only before and during our annual Summits.

All teams (should) have a lead which coordinates the team efforts. To join a team is simple:
* add your name to the team member list and
* if possible contact the lead in IRC-channel (#meego-fi)

=== Program Task-force ===
The committee has been formed by inviting suitable MeeGo Network Finland -community members to it.

'''Members'''
* [http://meego.com/users/bergie Henri Bergius] (bergie)
* [http://meego.com/users/jukkaeklund Jukka Eklund] (jukkaeklund)
* [http://meego.com/users/timoph Timo Härkönen] (timoph)
* (lead) [[User:Kyber | Jarkko Moilanen ]] (kyb3R)



=== Marketing Team ===
Team description...(team lead, please add)

* More information, tasks and how to join can be found [[Local_MeeGo_Networks/MeeGo_Network_Finland/Marketing_Team|here]]

=== Tech Team ===
Effective and open communication has two sides: internal and external. This team is responsible for internal communication tools which include methods to share information among local networks and members. Working and transparent communication is vital to our community since our activities are coordinated with virtual tools such as email, IRC and portal.

External communication refers to the different tools we use to communicate with the public and global MeeGo community. For these purposes we use various social media services such as [http://twitter.com/meegofi twitter.com/meegofi], [http://facebook.com/meegofi facebook.com/meegofi] and [http://identi.ca/meegofi identi.ca/meegofi].

* More information, tasks and how to join can be found [[Local_MeeGo_Networks/MeeGo_Network_Finland/Communication_Team|here]]

=== Hosting Team ===
Team is active mostly before and during the annual MeeGo Summit FI events. Team's duties include comprehensive management of items which affect speaker's enjoyment. Such items are for example arranging hotel rooms, providing information and assisting in session preparations. Managing pre-registration and registration (including portal) is one of the key responsibilities of this team.

Team also manages the information desk at the summit venue. They are the people to go to when you have lost your precious laptop, Tux/MeeGon figure or hacking buddy. Information desk is also the correct place to inform the organizers about different activities which summit participants are encouraged to arrange at the venue. Such activities include hands-on sessions for small groups, unconference/barcamp sessions and workhops (or in this case hack-shops). When in doubt, turn to these people.

* More information, tasks and how to join can be found [[Local_MeeGo_Networks/MeeGo_Network_Finland/MeeGo_Summit_FI_Hosting_Team | here]]

=== Party MeeGons ===
This is probably 'teh' team for all party-MeeGons. If you have talent in being a VJ, DJ or you have ideas for stunning laser shows, this team is for you. Parties are important part of any summit or conference. We do not want to offer our summit participants fancy dinners at mansions or rigid and boring wine tasting. We prefer to have more casual and relaxed parties, which can include ad hoc competitions and shows. MeeGo Summit FI evening party is on Friday night. Join the party-MeeGons!

While bouncers are handy to control the crowd, we believe that they are not needed in this case :)

Responsible for:
* Evening party at MeeGo Summit FI
* DJ/VJ (1+1 volunteers)
* ...

'''Members:'''
* Juha "[[User:Riussi|Riussi]]" Ristolainen. I can also help in organizing the party but we need an experienced team lead for this team.
* Jukka Eklund ([http://meego.com/users/jukkaeklund jukkaeklund]). I can't VJ or anything but will help to get the party going (won't be a bouncer however)
* [http://meego.com/users/smoinen Matti Saastamoinen] (smoinen) - [http://www.coss.fi/en/contact contact] // I can be the interface to party sponsors and bring my knowledge from our past event parties

''' Ideas & Tasks'''
* Invent a leet name for the party!

=== Hack3r Team ===
To be labelled as Hacker is an honor among us. The word hacker means to us the same as 'Sir' does for the royalty. Members of this team are
talented in different technical areas and posses deep knowledge. If you're not overwhelmed in front of technical problems, this is your team.
Ability to think outside-the-box and to find a solution is important. Your solution to problems do not need to be most
elegant and polished, it just needs to work.

If you want to join the hack3r team, send an e-mail to Topi Santakivi (first_nameD0Tlastname _AT_ gmail.com)

This team will coordinate efforts related to:
* '''Meegathon''' competition which is part of MeeGo Summit FI 2011
* Helps competition teams in technical issues. (4-5 volunteers)
* Responsible for the USB-sticks ('Developer stick') which is given to all participants (contains latest developer tools) (1-2 volunteers)
* Coordinates tool-shop usage (1 volunteer)
* Manages our (meego-fi) Intel AppUp developer program services and members.

'''Members:'''
* Laszlo Papp (djszapi)
* (lead) [http://meego.com/users/sandst1 Topi Santakivi] (sandst1) first_nameD0Tlastname _AT_ gmail.com
* [http://meego.com/users/timoph Timo Härkönen] (timoph)
* [http://meego.com/users/samposa Sampo Saaristo] (sampos)
...

''' Ideas & Tasks: '''
* Sign to be attached in the entry-point of Meegathon teams: [http://dl.dropbox.com/u/7717672/Meegonetwork.fi/hackland-sign.png hackland-sign.png]
* Polish/Define the competition (Meegathon) [[Local_MeeGo_Networks/MeeGo_Network_Finland/MeeGo_Summit_FI#Meegathon |rules]].
* Define the content of 'Developer USB-stick'
* Create new LAN/WLAN solutions which might help.

== Program ==
The summit program includes keynotes, three tracks, workshops, evening party on Friday and Meegathon competition. On Saturday we have Finhack (http://www.finhack.org) annual conference on the second floor.

=== Confirmed Speakers ===
Topics will be set later, in February - March.
* [http://meego.com/users/stskeeps Carsten Munk] (keynote)
* [http://fi.linkedin.com/in/valhalla Valtteri Halla] (keynote)
* [http://conference2010.meego.com/people/lbt David Greaves]
* [http://conference2010.meego.com/people/vilvo Ville Ilvonen]
* [http://meego.com/users/kate Kate Alhola]
* [http://fi.linkedin.com/pub/jan-krebber/5/957/2b Jan Krebber]
* [http://www.linkedin.com/pub/justin-noel/5/a5b/95a Justin Noel]
* [http://meego.com/users/abustany Adrien Bustany] & [http://fi.linkedin.com/pub/ivan-frade/4/984/903 Ivan Frade]

=== Tracks ===
In every organization, there are people who just have the knack for getting the most out of certain software and tools. These tracks are where those people shine -- it is the place to show off the newest tools, share creative uses for old favorites, and of course pass on hacks, tips and tricks. Sessions may include demonstrations, pilot projects, examples, techniques, and lessons learned.

# '''Core & Future''' - Apps and possibilities with latest tools (Fri - Sat)
# '''Hacks & Tricks''' - HW inventions, unconventional experiments (Fri - Sat)
# '''Biz Buzz''' - MeeGo Ecosystem and Open Source (Fri)

=== Hackshops and BoFs ===
Also known as lame Workshops. Participating to Hackshops requires sufficient amount of hack attitude and enthusiasm. We have some facilities (small meeting rooms) for hackshops. If you get an idea about what to teach/show to others (preferably hands-on stuff), book yourself a hackshop space and let the organizers know about it by dropping at the information desk! Your hackshop will be announced in twitter and on the venue screens. If you already know a topic, put it below.

'''Ideas for Workshops and BoFs'''

Add your ideas for workshops below. Also add your name if willing to run the workshop.
* UX helpdesk (whole day)
* MeeGo SDK (getting started)
* ...

=== Meegathon ===
Meegathon is a hacker type of competition, much like a hack-a-thon. The following competition will take place between Fri 12:00am and Sat 12:00am. In other words, teams have 24 hours time to finish their project. You have the possibility to sleep in hackerspace (big hall), so take your sleeping bag with you.

Meegathon is both a software development and hardware innovation competition, with the emphasis on software. Teams participating in the competition can choose their own topic or use the one provided below. The purpose of the competition is to produce new open innovative software or improvements to existing open source projects. The judges will expect the projects to be targeted at MeeGo but if the project is useful for the larger open source community - even better.

NOTE: We need people who will do shifts in supervising the Meegathon. This doesn't have to be technical support, there just needs to be somebody present with a key who can let people in and make sure that things are going smoothly.

Put your name on the task list if you want to help:
[http://doodle.com/n8dbwzzis7iawwty Meegathon shifts in doodle]

See different page for full details & rules: [[Local_MeeGo_Networks/MeeGo_Network_Finland/meegathon | Meegathon Competetion]]

=== Party ===
* Friday night, detailed time will be announced later
* Location: coordinates will be announced later
* name of the party?



=== Quadricopter/MeeGo games ===
We might be able to arrange a possibility to hold small-scale playful games with couple of Quadricopters. More will be revealed later. Needs 1-3 volunteers.



== Sponsoring ==
Sponsoring opportunities and packages, contact Ilkka Lehtinen (ilkka dot lehtinen @ hermia dot fi)

'''Sponsors:'''

* Ixonos (Platinum)
* Tieto (Gold)
* Futurice (Silver)
* Sasken (Silver)
* ICS (Silver)
* Nomovok (Silver)

== How to contribute? ==
Interested in helping? Great! Contributing is easy and you can give as little or as much time as you want.

Here are just a few ways you can get started:
* Check the TODO list below to find something that needs to be done.
* Join discussion at our IRC channel: #meego-fi (irc.freenode.org) and ask what you can do. If you don't have a IRC client, you can access the channel from this free service: http://webchat.freenode.net/
* Join our mailing list (add here)

Here are some of the volunteer opportunities:
* IRC Chat Moderators
* Information Desk Helpers
* Session Tech Support
* Conference Setup - 14th Apr.
* Conference Cleanup - 16th Apr
* Speaker Assistants
* Hacking Session Helpers

Not sure where you want to volunteer yet until the conference schedule is finalized? Put your name here and we'll add more detailed volunteer tables in the months to come.

'''Volunteers:'''

* Juha "[[User:Riussi|Riussi]]" Ristolainen
* Jukka [http://twitter.com/jukkaeklund @jukkaeklund] Eklund
* Carol [http://twitter.com/cybette @cybette] Chen
* Ville [http://twitter.com/jykae_ @jykae] Jyrkkä -- tha 'all-around' aMeeGo for the Summit arrangements, kick me in the IRC!

== TODO ==
List of tasks etc that needs to be done by organizers or volunteers. Add your name after a task if you want to do it.
* Define and collect software package (collected about a week before Summit) given for every attendee in a USB stick. 300 pcs. Cuts back the need to download all software from the net. USB-sticks will be provided by New Factory. - Topi Santakivi (sandst1)
* Marketing material: t-shirts, stickers, flyers....
** Matti Saastamoinen (smoinen) and Ilkka Lehtinen (among other guys from the community and sponsors) will attend [http://fosdem.org/ FOSDEM] on Feb 5-6, 2011. Leaflets and summit t-shirts for marketing should be ready before that.

* Activate MeeGo bloggers to write about this community and MeeGo Summit FI 2011. This could be done be sending 'inside' information to key bloggers once in a while. That way they get the feeling of being 'special' and in the frontline.
* Photographs and/or video from the venue site. Sneak-peak about the facilities which will be transformed to for Summit needs. Place the images somewhere public so they can be used in tweets.

ARM/N900/QA

2011-03-22T07:14:39Z

Samposa: /* Crashdb support for ARM core dumps */

= QA TODOs (in priority order) =

== OTS setup and automated hourly testing ==

=== OTS setup ===
* OTS server - Ville Ilvonen/Riku Halonen, DONE
* OTS worker(s) for core tests - Ville Ilvonen/Riku Halonen/Timo Harkonen - ONGOING (2 workers up)
* OTS worker for UX tests
* Power consumption measurements
* Reporting to QA-reports
* Minimize automatic installation time to 10mins - Timo Makimattila, ONGOING

=== Test automation images ===
* Setup hourly image building for autotest image on own setup (maybe later with BOSS) - Riku Halonen ONGOING (setting up F13 machine for image creation)
* We need to be able to control included test packages

If you need something else from QA please tell it to us :)

== Applications for testing ==
* Package QML demo/example applications from http://doc.qt.nokia.com/latest/qdeclarativeexamples.html

== Test asset ==
* Define what the asset includes
* Automating as many core/ux tests as possible
* Call/SMS cases will require parallel test execution support (If we want to automate those kind of tests)

== Crashdb support for ARM core dumps ==

* Core dump processing and backtraces from crashing ARM processes. - sampos, ONGOING

== Test Execution Schedule ==
* Core

* Handset UX weekly testing schedule
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Key feature (OK)
| Preview
|-
| Monday
| Acceptance (OK)
| Testing trunk
|-
|Tuesday
|Acceptance (Ok)
|Testing trunk
|-
|Tuesday
|Sanity (Ok)
|Daily trunk
|-
|Tuesday
|DE Dataflow (Ok)
|Preview
|-
|Tuesday
|DE use cases (Ok)
|Preview
|-
|Wednesday
|Acceptance (Ok)
|Testing trunk
|-
|Wednesday
|Key feature (Ok)
|Weekly
|-
|Thursday
|DE Dataflow (Ok)
|Weekly
|-
|Thursday
|DE use cases (Ok)
|Weekly
|-
|Thursday
|Acceptance (Ok)
|Testing trunk
|-
|Thursday
|Sanity Ok
|Daily trunk
|-
|Thursday
|DE Reliability (Ongoing)
|Weekly
|-
|Thursday
|DE Performance (Ongoing)
|Weekly
|-
|Friday
|Acceptance (Ok)
|Testing trunk
|-
|Friday
|Sanity (Ok)
|Daily trunk
|}

== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Sanity Test Set ===
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features

=== Feature Test Set ===
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

== Error Management ==
* Error Manager Iekku Huttunen

== Usefull links ==

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-03-22T07:14:12Z

Samposa: /* Crashdb support for ARM core dumps */

= QA TODOs (in priority order) =

== OTS setup and automated hourly testing ==

=== OTS setup ===
* OTS server - Ville Ilvonen/Riku Halonen, DONE
* OTS worker(s) for core tests - Ville Ilvonen/Riku Halonen/Timo Harkonen - ONGOING (2 workers up)
* OTS worker for UX tests
* Power consumption measurements
* Reporting to QA-reports
* Minimize automatic installation time to 10mins - Timo Makimattila, ONGOING

=== Test automation images ===
* Setup hourly image building for autotest image on own setup (maybe later with BOSS) - Riku Halonen ONGOING (setting up F13 machine for image creation)
* We need to be able to control included test packages

If you need something else from QA please tell it to us :)

== Applications for testing ==
* Package QML demo/example applications from http://doc.qt.nokia.com/latest/qdeclarativeexamples.html

== Test asset ==
* Define what the asset includes
* Automating as many core/ux tests as possible
* Call/SMS cases will require parallel test execution support (If we want to automate those kind of tests)

== Crashdb support for ARM core dumps ==

* Core dump processing and backtraces from crashing ARM processes. -sampos

== Test Execution Schedule ==
* Core

* Handset UX weekly testing schedule
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Key feature (OK)
| Preview
|-
| Monday
| Acceptance (OK)
| Testing trunk
|-
|Tuesday
|Acceptance (Ok)
|Testing trunk
|-
|Tuesday
|Sanity (Ok)
|Daily trunk
|-
|Tuesday
|DE Dataflow (Ok)
|Preview
|-
|Tuesday
|DE use cases (Ok)
|Preview
|-
|Wednesday
|Acceptance (Ok)
|Testing trunk
|-
|Wednesday
|Key feature (Ok)
|Weekly
|-
|Thursday
|DE Dataflow (Ok)
|Weekly
|-
|Thursday
|DE use cases (Ok)
|Weekly
|-
|Thursday
|Acceptance (Ok)
|Testing trunk
|-
|Thursday
|Sanity Ok
|Daily trunk
|-
|Thursday
|DE Reliability (Ongoing)
|Weekly
|-
|Thursday
|DE Performance (Ongoing)
|Weekly
|-
|Friday
|Acceptance (Ok)
|Testing trunk
|-
|Friday
|Sanity (Ok)
|Daily trunk
|}

== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Sanity Test Set ===
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features

=== Feature Test Set ===
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

== Error Management ==
* Error Manager Iekku Huttunen

== Usefull links ==

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

Quality/QA-tools

2011-03-09T07:19:05Z

Samposa: /* Tools, test suites and other utilities */

= Quality assurance (QA) tools =

[[File:Tool_meegon.png|400px]]

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for quality assurance.

* "Open source tools – available for all, available for development and contributions. Make people accountable for quality."

'''Please note''' that developer-oriented content has been moved to [[Quality/QA tools development|QA tools development]] page. We apologize for any inconvenience. The target is to make this page clearer and more informative for the actual users of QA tools.

== Overview ==

QA tools support test planning, test execution and test reporting. The most important tool offering relating to this is illustrated in the figure below. In addition, there is also a [http://www.youtube.com/watch?v=WdXBqGdLz8I concept video] available on manual testing tool offering on our [http://www.youtube.com/user/meegoqatools YouTube channel].

[[File:Meego-qa-tools.png|600px]]

== Tools, test suites and other utilities ==

If you want to contribute to the development of the following tools, test suites and utilities, please visit [[Quality/QA_tools_development|QA tools development page]].

'''Test tools'''

Each wiki page will contain more detailed description, installation instructions and tutorial on the basic usage. For now, please [[Quality/QA-tools#Collaboration spaces|contact us]] if you need help. Currently, you can install Testrunner, testrunner-lite, test-definition, Testplanner, OTS, Meego-ai, libcqpid, eat and MIN from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Short description'''
|-
| Crash Reporter || On-device tool for sending core dumps to core dump data base
|-
| [[Quality/QA-tools/MeeGo_Fast_Feedback_Testing|MeeGo Fast Feedback Testing (MeeGo-FFT)]] ||
|-
| [http://min.sf.net/ MIN Test Framework] || Advanced test framework
|-
| [[Quality/QA-tools/OTS|OTS - open testing system]] || Test automation system
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] || Simple tool for creating and editing test plan files
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||Command line tool for test execution
|-
| [[Quality/QA-tools/Testrunner|Testrunner]] || Test execution tool with graphical UI
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || Comprehensive tool for UI testing
|-
| [[Quality/QA-tools/QAReports|QA Reports]] || Reporting tool for publishing test reports
|}

'''Test suites'''

{| border="1" cellspacing="0"
|+
|'''Test suite (link to wiki page)''' ||| '''Test plans'''
|-
| [[Quality/TestSuite/MCTS|MeeGo Core Test Suite]] ||[http://testplans.meego.com/mcts/ Link]
|-
| [[Quality/TestSuite/handset-test-suite/handset-ux-test_Releases |MeeGo Handset UX Test Suite]] ||[http://testplans.meego.com/handset-ux-tests/ Link]
|-
| [[Quality/Netbook Test Suite and Utilities|MeeGo Netbook Test Suite]] ||[http://testplans.meego.com/mnts/ Link]
|-
| MeeGo SDK Test Suite ||[http://testplans.meego.com/meego-sdk-tests/ Link]
|}

'''Other utilities'''

{| border="1" cellspacing="0"
|+
|'''Name (link to wiki page)''' ||'''Short description'''
|-
|[[Quality/QA-tools/Test-definition|test-definition]]|| XML schema for test plan and result files
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] || Test automation configuration packages
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] ||
|-
| Rich Core dumper ||
|-
| Crash Reporter settings || Settings package for Crash Reporter
|-
| [[Quality/QA-tools/hat-control|Hardware Accessory for Testing (HAT)]] ||
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] ||
|-
| Service OS based Flasher ||
|-
| Qpid C wrapper library - libcqpid ||
|}

== Feature requests and bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=MeeGo%20Quality%20Assurance Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance All open features and bugs in priority order]

Bugzilla workflow: [[Quality/How To Report Bugs]]

== Collaboration spaces ==

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
** [http://timoph.fi/qa-tools-logs/ #meego-qa-tools irc logs]
* [http://www.youtube.com/user/meegoqatools YouTube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]

<br />
[[Category:QA]]

Quality/QA-tools

2011-02-08T11:28:22Z

Samposa: /* Roadmap */

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://meego.gitorious.org/meego-quality-assurance/ git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner|Testrunner]] || [http://meego.gitorious.org/meego-quality-assurance/testrunner Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://meego.gitorious.org/meego-quality-assurance/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://meego.gitorious.org/meego-quality-assurance/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| [[Quality/TestSuite/MCTS|MeeGo Core Test Suite]] || [http://meego.gitorious.org/meego-quality-assurance/mcts/ Gitorious] ||Matti Salmi || Jeff Zheng
|-
| [[Quality/Netbook Test Suite and Utilities|MeeGo Netbook Test Suite]] || [http://gitorious.org/qa-tools/mnts/ Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://meego.gitorious.org/meego-quality-assurance/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://meego.gitorious.org/meego-quality-assurance/rich-core Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter settings || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter-settings-public Gitorious] || Carol Rus || Raimo Gratseff
|-
| [[Quality/QA-tools/hat-control|Hardware Accessory for Testing (HAT)]] || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| [[Quality/QA-tools/QAReports|QA Reports]] || [http://meego.gitorious.org/meego-quality-assurance/qa-reports Gitorious] || Sami Hangaslammi || Jarno Keskikangas
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] || [http://meego.gitorious.com/meego-quality-assurance/scripts-and-utils Gitorious] || N/A || N/A
|-
| [[Quality/TestSuite/handset-test-suite/handset-ux-test_Releases | handset_ux_tests]] || [http://meego.gitorious.com/meego-quality-assurance/handset-ux-tests Gitorious] || JessicaJi || N/A
|-
| [[Quality/QA-tools/MeeGo_Fast_Feedback_Testing|MeeGo Fast Feedback Testing (MeeGo-FFT)]] || [http://meego.gitorious.org/meego-quality-assurance/meego-testing-hudson-plugin Gitorious] || Alexey Kuznetsov || Timo Härkönen
|-
| Service OS based Flasher || [http://meego.gitorious.org/meego-quality-assurance/meego-ai-serviceos Gitorious] || Jing Wang || N/A
|-
| Qpid C wrapper library - libcqpid || [http://meego.gitorious.org/meego-quality-assurance/libcqpid Gitorious] || Sami Lahtinen || N/A
|}

You can install Testrunner, testrunner-lite, test-definition, Testplanner, OTS, Meego-ai, libcqpid, eat and MIN from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here is the workflow for QA tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file: [[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Roadmap ==
(We need still rough estimates for releases -timakima)

The release dates defined in MeeGo Release plans [[Release_Engineering/Plans/1.1|1.1]] and [[Release_Engineering/Plans/1.2|1.2]]. These dates are the latest estimation. They will be updated as work progresses.

The features in the roadmap are followed with META FEA bugs. The features are split to small tool specific FEA:s that block the feature META bug. The META bug is then the last bug to be closed when the feature is finished. You can also follow the tool bug progression from the dependency trees of the META bugs.

Adding a new feature to the roadmap:
# Add a feature bug describing the main purpose of the feature with META and FEA tags.
# Add separate bugs for each tool specific change
# Add correct dependencies between tool specific bugs (e.g. testrunner bug depends on testrunner-lite bug that depends on test-definition bug...)
# Make all of the bugs block the META bug

{| border="1" cellspacing="0"
|+
|'''Feature''' || '''Release'''
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12980&hide_resolved=0 Measurement support] || 1.1.90.5
|-
| [https://bugs.meego.com/show_bug.cgi?id=12810 Easy install] || 1.1.90.5
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12454&hide_resolved=0 Parallel testing] || 1.1.90.7
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12981&hide_resolved=0 Events feature in automatic testing] || 1.1.90.8
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12937&hide_resolved=0 MCTS coverage support] || 1.2.0.0
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12833&hide_resolved=0 Test environment validation] || 1.2.x
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=13092&hide_resolved=0 Test equipment control] || 1.2.x
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12971&hide_resolved=0 Qt Creator integration] || 1.2.x
|}

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=MeeGo%20Quality%20Assurance Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

We'd like to have the best guess of the moment about the delivery of features. Remember to set the target build of the bug you're working with according to: [[Release Engineering/Plans/1.1]] and [[Release Engineering/Plans/1.2]]

When you add a new bug, add correct dependencies to the corresponding [[Quality/QA-tools#Roadmap|roadmap]] meta bug.

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS test automation design|MCTS Test Automation]] (QA Tools support for MCTS)

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

* Team meetings
** QA tools team meetings will be held on need basis for specific topics. This was agreed in [http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-07-11.59.html the last weekly meeting held on December 7th 2010]
** [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

* Architecture meetings
** Architecture meetings will be held on-demand basis.
** [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

* Workshops
** Face-2-face meetings within the team.
** [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarmo Savinen || Team member || Digia || jasavi
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
| Sami Hangaslammi || Team member || Leonidas || sahangas
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
** [http://timoph.fi/qa-tools-stats/ #meego-qa-tools statistics]
** [http://timoph.fi/qa-tools-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools

2011-02-08T11:23:31Z

Samposa: /* Roadmap */

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://meego.gitorious.org/meego-quality-assurance/ git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner|Testrunner]] || [http://meego.gitorious.org/meego-quality-assurance/testrunner Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://meego.gitorious.org/meego-quality-assurance/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://meego.gitorious.org/meego-quality-assurance/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| [[Quality/TestSuite/MCTS|MeeGo Core Test Suite]] || [http://meego.gitorious.org/meego-quality-assurance/mcts/ Gitorious] ||Matti Salmi || Jeff Zheng
|-
| [[Quality/Netbook Test Suite and Utilities|MeeGo Netbook Test Suite]] || [http://gitorious.org/qa-tools/mnts/ Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://meego.gitorious.org/meego-quality-assurance/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://meego.gitorious.org/meego-quality-assurance/rich-core Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter settings || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter-settings-public Gitorious] || Carol Rus || Raimo Gratseff
|-
| [[Quality/QA-tools/hat-control|Hardware Accessory for Testing (HAT)]] || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| [[Quality/QA-tools/QAReports|QA Reports]] || [http://meego.gitorious.org/meego-quality-assurance/qa-reports Gitorious] || Sami Hangaslammi || Jarno Keskikangas
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] || [http://meego.gitorious.com/meego-quality-assurance/scripts-and-utils Gitorious] || N/A || N/A
|-
| [[Quality/TestSuite/handset-test-suite/handset-ux-test_Releases | handset_ux_tests]] || [http://meego.gitorious.com/meego-quality-assurance/handset-ux-tests Gitorious] || JessicaJi || N/A
|-
| [[Quality/QA-tools/MeeGo_Fast_Feedback_Testing|MeeGo Fast Feedback Testing (MeeGo-FFT)]] || [http://meego.gitorious.org/meego-quality-assurance/meego-testing-hudson-plugin Gitorious] || Alexey Kuznetsov || Timo Härkönen
|-
| Service OS based Flasher || [http://meego.gitorious.org/meego-quality-assurance/meego-ai-serviceos Gitorious] || Jing Wang || N/A
|-
| Qpid C wrapper library - libcqpid || [http://meego.gitorious.org/meego-quality-assurance/libcqpid Gitorious] || Sami Lahtinen || N/A
|}

You can install Testrunner, testrunner-lite, test-definition, Testplanner, OTS, Meego-ai, libcqpid, eat and MIN from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here is the workflow for QA tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file: [[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Roadmap ==
(We need still rough estimates for releases -timakima)

The release dates defined in MeeGo Release plans [[Release_Engineering/Plans/1.1|1.1]] and [[Release_Engineering/Plans/1.2|1.2]]. These dates are the latest estimation. They will be updated as work progresses.

The features in the roadmap are followed with META FEA bugs. The features are split to small tool specific FEA:s that block the feature META bug. The META bug is then the last bug to be closed when the feature is finished. You can also follow the tool bug progression from the dependency trees of the META bugs.

Adding a new feature to the roadmap:
# Add a feature bug describing the main purpose of the feature with META and FEA tags.
# Add separate bugs for each tool specific change
# Add correct dependencies between tool specific bugs (e.g. testrunner bug depends on testrunner-lite bug that depends on test-definition bug...)
# Make all of the bugs block the META bug

{| border="1" cellspacing="0"
|+
|'''Feature''' || '''Release'''
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12980&hide_resolved=0 Measurement support] || 1.1.90.5
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12937&hide_resolved=0 MCTS coverage support] || 1.2.0.0
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12833&hide_resolved=0 Test environment validation] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=13092&hide_resolved=0 Test equipment control] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12454&hide_resolved=0 Parallel testing] || 1.1.90.7
|-
| [https://bugs.meego.com/show_bug.cgi?id=12810 Easy install] || 1.1.90.5
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12971&hide_resolved=0 Qt Creator integration] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12981&hide_resolved=0 Events feature in automatic testing] || 1.1.90.8
|}

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=MeeGo%20Quality%20Assurance Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

We'd like to have the best guess of the moment about the delivery of features. Remember to set the target build of the bug you're working with according to: [[Release Engineering/Plans/1.1]] and [[Release Engineering/Plans/1.2]]

When you add a new bug, add correct dependencies to the corresponding [[Quality/QA-tools#Roadmap|roadmap]] meta bug.

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS test automation design|MCTS Test Automation]] (QA Tools support for MCTS)

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

* Team meetings
** QA tools team meetings will be held on need basis for specific topics. This was agreed in [http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-07-11.59.html the last weekly meeting held on December 7th 2010]
** [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

* Architecture meetings
** Architecture meetings will be held on-demand basis.
** [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

* Workshops
** Face-2-face meetings within the team.
** [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarmo Savinen || Team member || Digia || jasavi
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
| Sami Hangaslammi || Team member || Leonidas || sahangas
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
** [http://timoph.fi/qa-tools-stats/ #meego-qa-tools statistics]
** [http://timoph.fi/qa-tools-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools

2011-02-08T11:21:18Z

Samposa: /* Roadmap */

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://meego.gitorious.org/meego-quality-assurance/ git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner|Testrunner]] || [http://meego.gitorious.org/meego-quality-assurance/testrunner Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://meego.gitorious.org/meego-quality-assurance/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://meego.gitorious.org/meego-quality-assurance/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| [[Quality/TestSuite/MCTS|MeeGo Core Test Suite]] || [http://meego.gitorious.org/meego-quality-assurance/mcts/ Gitorious] ||Matti Salmi || Jeff Zheng
|-
| [[Quality/Netbook Test Suite and Utilities|MeeGo Netbook Test Suite]] || [http://gitorious.org/qa-tools/mnts/ Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://meego.gitorious.org/meego-quality-assurance/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://meego.gitorious.org/meego-quality-assurance/rich-core Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter settings || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter-settings-public Gitorious] || Carol Rus || Raimo Gratseff
|-
| [[Quality/QA-tools/hat-control|Hardware Accessory for Testing (HAT)]] || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| [[Quality/QA-tools/QAReports|QA Reports]] || [http://meego.gitorious.org/meego-quality-assurance/qa-reports Gitorious] || Sami Hangaslammi || Jarno Keskikangas
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] || [http://meego.gitorious.com/meego-quality-assurance/scripts-and-utils Gitorious] || N/A || N/A
|-
| [[Quality/TestSuite/handset-test-suite/handset-ux-test_Releases | handset_ux_tests]] || [http://meego.gitorious.com/meego-quality-assurance/handset-ux-tests Gitorious] || JessicaJi || N/A
|-
| [[Quality/QA-tools/MeeGo_Fast_Feedback_Testing|MeeGo Fast Feedback Testing (MeeGo-FFT)]] || [http://meego.gitorious.org/meego-quality-assurance/meego-testing-hudson-plugin Gitorious] || Alexey Kuznetsov || Timo Härkönen
|-
| Service OS based Flasher || [http://meego.gitorious.org/meego-quality-assurance/meego-ai-serviceos Gitorious] || Jing Wang || N/A
|-
| Qpid C wrapper library - libcqpid || [http://meego.gitorious.org/meego-quality-assurance/libcqpid Gitorious] || Sami Lahtinen || N/A
|}

You can install Testrunner, testrunner-lite, test-definition, Testplanner, OTS, Meego-ai, libcqpid, eat and MIN from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here is the workflow for QA tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file: [[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Roadmap ==
(We need still rough estimates for releases -timakima)

The release dates defined in MeeGo Release plans [[Release_Engineering/Plans/1.1|1.1]] and [[Release_Engineering/Plans/1.2|1.2]]. These dates are the latest estimation. They will be updated as work progresses.

The features in the roadmap are followed with META FEA bugs. The features are split to small tool specific FEA:s that block the feature META bug. The META bug is then the last bug to be closed when the feature is finished. You can also follow the tool bug progression from the dependency trees of the META bugs.

Adding a new feature to the roadmap:
# Add a feature bug describing the main purpose of the feature with META and FEA tags.
# Add separate bugs for each tool specific change
# Add correct dependencies between tool specific bugs (e.g. testrunner bug depends on testrunner-lite bug that depends on test-definition bug...)
# Make all of the bugs block the META bug

{| border="1" cellspacing="0"
|+
|'''Feature''' || '''Release'''
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12980&hide_resolved=0 Measurement support] || 1.1.90.5
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12937&hide_resolved=0 MCTS coverage support] || 1.2.0.0
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12833&hide_resolved=0 Test environment validation] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=13092&hide_resolved=0 Test equipment control] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12454&hide_resolved=0 Parallel testing] || 1.1.90.7
|-
| [https://bugs.meego.com/show_bug.cgi?id=12810 Easy install] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12971&hide_resolved=0 Qt Creator integration] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12981&hide_resolved=0 Events feature in automatic testing] || 1.1.90.8
|}

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=MeeGo%20Quality%20Assurance Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

We'd like to have the best guess of the moment about the delivery of features. Remember to set the target build of the bug you're working with according to: [[Release Engineering/Plans/1.1]] and [[Release Engineering/Plans/1.2]]

When you add a new bug, add correct dependencies to the corresponding [[Quality/QA-tools#Roadmap|roadmap]] meta bug.

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS test automation design|MCTS Test Automation]] (QA Tools support for MCTS)

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

* Team meetings
** QA tools team meetings will be held on need basis for specific topics. This was agreed in [http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-07-11.59.html the last weekly meeting held on December 7th 2010]
** [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

* Architecture meetings
** Architecture meetings will be held on-demand basis.
** [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

* Workshops
** Face-2-face meetings within the team.
** [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarmo Savinen || Team member || Digia || jasavi
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
| Sami Hangaslammi || Team member || Leonidas || sahangas
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
** [http://timoph.fi/qa-tools-stats/ #meego-qa-tools statistics]
** [http://timoph.fi/qa-tools-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools

2011-02-08T11:18:20Z

Samposa: /* Roadmap */

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://meego.gitorious.org/meego-quality-assurance/ git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner|Testrunner]] || [http://meego.gitorious.org/meego-quality-assurance/testrunner Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://meego.gitorious.org/meego-quality-assurance/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://meego.gitorious.org/meego-quality-assurance/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| [[Quality/TestSuite/MCTS|MeeGo Core Test Suite]] || [http://meego.gitorious.org/meego-quality-assurance/mcts/ Gitorious] ||Matti Salmi || Jeff Zheng
|-
| [[Quality/Netbook Test Suite and Utilities|MeeGo Netbook Test Suite]] || [http://gitorious.org/qa-tools/mnts/ Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://meego.gitorious.org/meego-quality-assurance/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://meego.gitorious.org/meego-quality-assurance/rich-core Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter settings || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter-settings-public Gitorious] || Carol Rus || Raimo Gratseff
|-
| [[Quality/QA-tools/hat-control|Hardware Accessory for Testing (HAT)]] || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| [[Quality/QA-tools/QAReports|QA Reports]] || [http://meego.gitorious.org/meego-quality-assurance/qa-reports Gitorious] || Sami Hangaslammi || Jarno Keskikangas
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] || [http://meego.gitorious.com/meego-quality-assurance/scripts-and-utils Gitorious] || N/A || N/A
|-
| [[Quality/TestSuite/handset-test-suite/handset-ux-test_Releases | handset_ux_tests]] || [http://meego.gitorious.com/meego-quality-assurance/handset-ux-tests Gitorious] || JessicaJi || N/A
|-
| [[Quality/QA-tools/MeeGo_Fast_Feedback_Testing|MeeGo Fast Feedback Testing (MeeGo-FFT)]] || [http://meego.gitorious.org/meego-quality-assurance/meego-testing-hudson-plugin Gitorious] || Alexey Kuznetsov || Timo Härkönen
|-
| Service OS based Flasher || [http://meego.gitorious.org/meego-quality-assurance/meego-ai-serviceos Gitorious] || Jing Wang || N/A
|-
| Qpid C wrapper library - libcqpid || [http://meego.gitorious.org/meego-quality-assurance/libcqpid Gitorious] || Sami Lahtinen || N/A
|}

You can install Testrunner, testrunner-lite, test-definition, Testplanner, OTS, Meego-ai, libcqpid, eat and MIN from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here is the workflow for QA tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file: [[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Roadmap ==
(We need still rough estimates for releases -timakima)

The release dates defined in MeeGo Release plans [[Release_Engineering/Plans/1.1|1.1]] and [[Release_Engineering/Plans/1.2|1.2]]. These dates are the latest estimation. They will be updated as work progresses.

The features in the roadmap are followed with META FEA bugs. The features are split to small tool specific FEA:s that block the feature META bug. The META bug is then the last bug to be closed when the feature is finished. You can also follow the tool bug progression from the dependency trees of the META bugs.

Adding a new feature to the roadmap:
# Add a feature bug describing the main purpose of the feature with META and FEA tags.
# Add separate bugs for each tool specific change
# Add correct dependencies between tool specific bugs (e.g. testrunner bug depends on testrunner-lite bug that depends on test-definition bug...)
# Make all of the bugs block the META bug

{| border="1" cellspacing="0"
|+
|'''Feature''' || '''Release'''
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12980&hide_resolved=0 Measurement support] || 1.1.90.5
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12937&hide_resolved=0 MCTS coverage support] || 1.2.0.0
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12833&hide_resolved=0 Test environment validation] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=13092&hide_resolved=0 Test equipment control] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12454&hide_resolved=0 Parallel testing] || TBD
|-
| [https://bugs.meego.com/show_bug.cgi?id=12810 Easy install] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12971&hide_resolved=0 Qt Creator integration] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12981&hide_resolved=0 Events feature in automatic testing] || 1.1.90.8
|}

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=MeeGo%20Quality%20Assurance Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

We'd like to have the best guess of the moment about the delivery of features. Remember to set the target build of the bug you're working with according to: [[Release Engineering/Plans/1.1]] and [[Release Engineering/Plans/1.2]]

When you add a new bug, add correct dependencies to the corresponding [[Quality/QA-tools#Roadmap|roadmap]] meta bug.

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS test automation design|MCTS Test Automation]] (QA Tools support for MCTS)

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

* Team meetings
** QA tools team meetings will be held on need basis for specific topics. This was agreed in [http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-07-11.59.html the last weekly meeting held on December 7th 2010]
** [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

* Architecture meetings
** Architecture meetings will be held on-demand basis.
** [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

* Workshops
** Face-2-face meetings within the team.
** [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarmo Savinen || Team member || Digia || jasavi
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
| Sami Hangaslammi || Team member || Leonidas || sahangas
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
** [http://timoph.fi/qa-tools-stats/ #meego-qa-tools statistics]
** [http://timoph.fi/qa-tools-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools

2011-02-08T11:15:14Z

Samposa: /* Roadmap */

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://meego.gitorious.org/meego-quality-assurance/ git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner|Testrunner]] || [http://meego.gitorious.org/meego-quality-assurance/testrunner Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://meego.gitorious.org/meego-quality-assurance/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://meego.gitorious.org/meego-quality-assurance/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| [[Quality/TestSuite/MCTS|MeeGo Core Test Suite]] || [http://meego.gitorious.org/meego-quality-assurance/mcts/ Gitorious] ||Matti Salmi || Jeff Zheng
|-
| [[Quality/Netbook Test Suite and Utilities|MeeGo Netbook Test Suite]] || [http://gitorious.org/qa-tools/mnts/ Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://meego.gitorious.org/meego-quality-assurance/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://meego.gitorious.org/meego-quality-assurance/rich-core Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter settings || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter-settings-public Gitorious] || Carol Rus || Raimo Gratseff
|-
| [[Quality/QA-tools/hat-control|Hardware Accessory for Testing (HAT)]] || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| [[Quality/QA-tools/QAReports|QA Reports]] || [http://meego.gitorious.org/meego-quality-assurance/qa-reports Gitorious] || Sami Hangaslammi || Jarno Keskikangas
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] || [http://meego.gitorious.com/meego-quality-assurance/scripts-and-utils Gitorious] || N/A || N/A
|-
| [[Quality/TestSuite/handset-test-suite/handset-ux-test_Releases | handset_ux_tests]] || [http://meego.gitorious.com/meego-quality-assurance/handset-ux-tests Gitorious] || JessicaJi || N/A
|-
| [[Quality/QA-tools/MeeGo_Fast_Feedback_Testing|MeeGo Fast Feedback Testing (MeeGo-FFT)]] || [http://meego.gitorious.org/meego-quality-assurance/meego-testing-hudson-plugin Gitorious] || Alexey Kuznetsov || Timo Härkönen
|-
| Service OS based Flasher || [http://meego.gitorious.org/meego-quality-assurance/meego-ai-serviceos Gitorious] || Jing Wang || N/A
|-
| Qpid C wrapper library - libcqpid || [http://meego.gitorious.org/meego-quality-assurance/libcqpid Gitorious] || Sami Lahtinen || N/A
|}

You can install Testrunner, testrunner-lite, test-definition, Testplanner, OTS, Meego-ai, libcqpid, eat and MIN from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here is the workflow for QA tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file: [[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Roadmap ==
(We need still rough estimates for releases -timakima)

The release dates defined in MeeGo Release plans [[Release_Engineering/Plans/1.1|1.1]] and [[Release_Engineering/Plans/1.2|1.2]]. These dates are the latest estimation. They will be updated as work progresses.

The features in the roadmap are followed with META FEA bugs. The features are split to small tool specific FEA:s that block the feature META bug. The META bug is then the last bug to be closed when the feature is finished. You can also follow the tool bug progression from the dependency trees of the META bugs.

Adding a new feature to the roadmap:
# Add a feature bug describing the main purpose of the feature with META and FEA tags.
# Add separate bugs for each tool specific change
# Add correct dependencies between tool specific bugs (e.g. testrunner bug depends on testrunner-lite bug that depends on test-definition bug...)
# Make all of the bugs block the META bug

{| border="1" cellspacing="0"
|+
|'''Feature''' || '''Release'''
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12980&hide_resolved=0 Measurement support] || 1.1.90.5
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12937&hide_resolved=0 MCTS coverage support] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12833&hide_resolved=0 Test environment validation] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=13092&hide_resolved=0 Test equipment control] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12454&hide_resolved=0 Parallel testing] || TBD
|-
| [https://bugs.meego.com/show_bug.cgi?id=12810 Easy install] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12971&hide_resolved=0 Qt Creator integration] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12981&hide_resolved=0 Events feature in automatic testing] || 1.1.90.8
|}

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=MeeGo%20Quality%20Assurance Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

We'd like to have the best guess of the moment about the delivery of features. Remember to set the target build of the bug you're working with according to: [[Release Engineering/Plans/1.1]] and [[Release Engineering/Plans/1.2]]

When you add a new bug, add correct dependencies to the corresponding [[Quality/QA-tools#Roadmap|roadmap]] meta bug.

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS test automation design|MCTS Test Automation]] (QA Tools support for MCTS)

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

* Team meetings
** QA tools team meetings will be held on need basis for specific topics. This was agreed in [http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-07-11.59.html the last weekly meeting held on December 7th 2010]
** [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

* Architecture meetings
** Architecture meetings will be held on-demand basis.
** [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

* Workshops
** Face-2-face meetings within the team.
** [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarmo Savinen || Team member || Digia || jasavi
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
| Sami Hangaslammi || Team member || Leonidas || sahangas
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
** [http://timoph.fi/qa-tools-stats/ #meego-qa-tools statistics]
** [http://timoph.fi/qa-tools-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools

2011-02-08T11:07:01Z

Samposa: /* Roadmap */

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://meego.gitorious.org/meego-quality-assurance/ git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner|Testrunner]] || [http://meego.gitorious.org/meego-quality-assurance/testrunner Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://meego.gitorious.org/meego-quality-assurance/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://meego.gitorious.org/meego-quality-assurance/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| [[Quality/TestSuite/MCTS|MeeGo Core Test Suite]] || [http://meego.gitorious.org/meego-quality-assurance/mcts/ Gitorious] ||Matti Salmi || Jeff Zheng
|-
| [[Quality/Netbook Test Suite and Utilities|MeeGo Netbook Test Suite]] || [http://gitorious.org/qa-tools/mnts/ Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://meego.gitorious.org/meego-quality-assurance/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://meego.gitorious.org/meego-quality-assurance/rich-core Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter settings || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter-settings-public Gitorious] || Carol Rus || Raimo Gratseff
|-
| [[Quality/QA-tools/hat-control|Hardware Accessory for Testing (HAT)]] || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| [[Quality/QA-tools/QAReports|QA Reports]] || [http://meego.gitorious.org/meego-quality-assurance/qa-reports Gitorious] || Sami Hangaslammi || Jarno Keskikangas
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] || [http://meego.gitorious.com/meego-quality-assurance/scripts-and-utils Gitorious] || N/A || N/A
|-
| [[Quality/TestSuite/handset-test-suite/handset-ux-test_Releases | handset_ux_tests]] || [http://meego.gitorious.com/meego-quality-assurance/handset-ux-tests Gitorious] || JessicaJi || N/A
|-
| [[Quality/QA-tools/MeeGo_Fast_Feedback_Testing|MeeGo Fast Feedback Testing (MeeGo-FFT)]] || [http://meego.gitorious.org/meego-quality-assurance/meego-testing-hudson-plugin Gitorious] || Alexey Kuznetsov || Timo Härkönen
|-
| Service OS based Flasher || [http://meego.gitorious.org/meego-quality-assurance/meego-ai-serviceos Gitorious] || Jing Wang || N/A
|-
| Qpid C wrapper library - libcqpid || [http://meego.gitorious.org/meego-quality-assurance/libcqpid Gitorious] || Sami Lahtinen || N/A
|}

You can install Testrunner, testrunner-lite, test-definition, Testplanner, OTS, Meego-ai, libcqpid, eat and MIN from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here is the workflow for QA tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file: [[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Roadmap ==
(We need still rough estimates for releases -timakima)

The release dates defined in MeeGo Release plans [[Release_Engineering/Plans/1.1|1.1]] and [[Release_Engineering/Plans/1.2|1.2]]. These dates are the latest estimation. They will be updated as work progresses.

The features in the roadmap are followed with META FEA bugs. The features are split to small tool specific FEA:s that block the feature META bug. The META bug is then the last bug to be closed when the feature is finished. You can also follow the tool bug progression from the dependency trees of the META bugs.

Adding a new feature to the roadmap:
# Add a feature bug describing the main purpose of the feature with META and FEA tags.
# Add separate bugs for each tool specific change
# Add correct dependencies between tool specific bugs (e.g. testrunner bug depends on testrunner-lite bug that depends on test-definition bug...)
# Make all of the bugs block the META bug

{| border="1" cellspacing="0"
|+
|'''Feature''' || '''Release'''
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12980&hide_resolved=0 Measurement support] || 1.1.90.5
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12937&hide_resolved=0 MCTS coverage support] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12833&hide_resolved=0 Test environment validation] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=13092&hide_resolved=0 Test equipment control] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12454&hide_resolved=0 Parallel testing] || TBD
|-
| [https://bugs.meego.com/show_bug.cgi?id=12810 Easy install] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12971&hide_resolved=0 Qt Creator integration] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12981&hide_resolved=0 Events feature in automatic testing] || TBD
|}

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=MeeGo%20Quality%20Assurance Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

We'd like to have the best guess of the moment about the delivery of features. Remember to set the target build of the bug you're working with according to: [[Release Engineering/Plans/1.1]] and [[Release Engineering/Plans/1.2]]

When you add a new bug, add correct dependencies to the corresponding [[Quality/QA-tools#Roadmap|roadmap]] meta bug.

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS test automation design|MCTS Test Automation]] (QA Tools support for MCTS)

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

* Team meetings
** QA tools team meetings will be held on need basis for specific topics. This was agreed in [http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-07-11.59.html the last weekly meeting held on December 7th 2010]
** [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

* Architecture meetings
** Architecture meetings will be held on-demand basis.
** [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

* Workshops
** Face-2-face meetings within the team.
** [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarmo Savinen || Team member || Digia || jasavi
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
| Sami Hangaslammi || Team member || Leonidas || sahangas
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
** [http://timoph.fi/qa-tools-stats/ #meego-qa-tools statistics]
** [http://timoph.fi/qa-tools-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools/Arch-Meetings

2011-02-04T13:43:07Z

Samposa: /* Meeting Topics Backlog */

== Meeting Intent ==

Meeting purpose is to solve QA Tools related issues, which need an architectural level decision within the team. For example, a developer working in QA Tools team can propose topics to be reviewed. These meetings are being held on a need basis, approximately once or twice a month, in '''#meego-meeting'''. Depending the amount of topics.

'''Note:''' Primary channel for tools related issues is still '''#meego-qa-tools'''.

== Meeting Topics Backlog ==

[Add your topics here]

* TestCaseTemplate alignment with testdefinition (https://bugs.meego.com/show_bug.cgi?id=11253):
** The purpose of TC_ID, TC_TITLE vs. testdefinition name field. The uniqueness of TC_ID field.
** Description field syntax when supporting TestCaseTemplate.

* Is there really need for support for <environment> tag in Testplanner and Testrunner(ui). It's needed for backward
compatibility in testrunner-lite.

* N/A result interpretation
** How is the N/A handled by ots result judge; and on low level: N/A test step effect on case result (https://bugs.meego.com/show_bug.cgi?id=12378)
** Are filtered cases N/A ?(https://bugs.meego.com/show_bug.cgi?id=5978)

== Next Meeting ==

Not scheduled.

== Meeting Minutes ==

'''Wednesday November 24th 2010 12:00 UTC'''

'''Agenda:'''

* Current status
* Architecture proposal(s)
* OTS integration
* Plan forward

[[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-11-24-11.59.html Meeting minutes]]

Quality/QA-tools/Arch-Meetings

2011-02-04T13:42:23Z

Samposa: /* Meeting Topics Backlog */

== Meeting Intent ==

Meeting purpose is to solve QA Tools related issues, which need an architectural level decision within the team. For example, a developer working in QA Tools team can propose topics to be reviewed. These meetings are being held on a need basis, approximately once or twice a month, in '''#meego-meeting'''. Depending the amount of topics.

'''Note:''' Primary channel for tools related issues is still '''#meego-qa-tools'''.

== Meeting Topics Backlog ==

[Add your topics here]

* TestCaseTemplate alignment with testdefinition (https://bugs.meego.com/show_bug.cgi?id=11253):
** The purpose of TC_ID, TC_TITLE vs. testdefinition name field. The uniqueness of TC_ID field.
** Description field syntax when supporting TestCaseTemplate.

* Is there really need for support for <environment> tag in Testplanner and Testrunner(ui). It's needed for backward
compatibility in testrunner-lite.

* N/A result interpretation
** How is the N/A handled by ots result judge and on low level N/A test step effect on case result (https://bugs.meego.com/show_bug.cgi?id=12378)
** Are filtered cases N/A ?(https://bugs.meego.com/show_bug.cgi?id=5978)

== Next Meeting ==

Not scheduled.

== Meeting Minutes ==

'''Wednesday November 24th 2010 12:00 UTC'''

'''Agenda:'''

* Current status
* Architecture proposal(s)
* OTS integration
* Plan forward

[[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-11-24-11.59.html Meeting minutes]]

Quality/QA-tools

2011-02-04T08:49:10Z

Samposa: /* Tools and Maintainers */

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://meego.gitorious.org/meego-quality-assurance/ git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner|Testrunner]] || [http://meego.gitorious.org/meego-quality-assurance/testrunner Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://meego.gitorious.org/meego-quality-assurance/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://meego.gitorious.org/meego-quality-assurance/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| [[Quality/TestSuite/MCTS|MeeGo Core Test Suite]] || [http://meego.gitorious.org/meego-quality-assurance/mcts/ Gitorious] ||Matti Salmi || Jeff Zheng
|-
| [[Quality/Netbook Test Suite and Utilities|MeeGo Netbook Test Suite]] || [http://gitorious.org/qa-tools/mnts/ Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://meego.gitorious.org/meego-quality-assurance/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://meego.gitorious.org/meego-quality-assurance/rich-core Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter settings || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter-settings-public Gitorious] || Carol Rus || Raimo Gratseff
|-
| [[Quality/QA-tools/hat-control|Hardware Accessory for Testing (HAT)]] || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| [[Quality/QA-tools/QAReports|QA Reports]] || [http://meego.gitorious.org/meego-quality-assurance/qa-reports Gitorious] || Sami Hangaslammi || Jarno Keskikangas
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] || [http://meego.gitorious.com/meego-quality-assurance/scripts-and-utils Gitorious] || N/A || N/A
|-
| [[Quality/TestSuite/handset-test-suite/handset-ux-test_Releases | handset_ux_tests]] || [http://meego.gitorious.com/meego-quality-assurance/handset-ux-tests Gitorious] || JessicaJi || N/A
|-
| [[Quality/QA-tools/MeeGo_Fast_Feedback_Testing|MeeGo Fast Feedback Testing (MeeGo-FFT)]] || [http://meego.gitorious.org/meego-quality-assurance/meego-testing-hudson-plugin Gitorious] || Alexey Kuznetsov || Timo Härkönen
|-
| Service OS based Flasher || [http://meego.gitorious.org/meego-quality-assurance/meego-ai-serviceos Gitorious] || Jing Wang || N/A
|-
| Qpid C wrapper library - libcqpid || [http://meego.gitorious.org/meego-quality-assurance/libcqpid Gitorious] || Sami Lahtinen || N/A
|}

You can install Testrunner, testrunner-lite, test-definition, Testplanner, OTS, Meego-ai, libcqpid, eat and MIN from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here is the workflow for QA tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file: [[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Roadmap ==
(We need still rough estimates for releases -timakima)

The release dates defined in MeeGo Release plans [[Release_Engineering/Plans/1.1|1.1]] and [[Release_Engineering/Plans/1.2|1.2]]. These dates are the latest estimation. They will be updated as work progresses.

The features in the roadmap are followed with META FEA bugs. The features are split to small tool specific FEA:s that block the feature META bug. The META bug is then the last bug to be closed when the feature is finished. You can also follow the tool bug progression from the dependency trees of the META bugs.

Adding a new feature to the roadmap:
# Add a feature bug describing the main purpose of the feature with META and FEA tags.
# Add separate bugs for each tool specific change
# Add correct dependencies between tool specific bugs (e.g. testrunner bug depends on testrunner-lite bug that depends on test-definition bug...)
# Make all of the bugs block the META bug

{| border="1" cellspacing="0"
|+
|'''Feature''' || '''Release'''
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12980&hide_resolved=0 Measurement support] || 1.2
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12937&hide_resolved=0 MCTS coverage support] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12833&hide_resolved=0 Test environment validation] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=13092&hide_resolved=0 Test equipment control] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12454&hide_resolved=0 Parallel testing] || TBD
|-
| [https://bugs.meego.com/show_bug.cgi?id=12810 Easy install] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12971&hide_resolved=0 Qt Creator integration] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12981&hide_resolved=0 Events feature in automatic testing] || TBD
|}

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=MeeGo%20Quality%20Assurance Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

We'd like to have the best guess of the moment about the delivery of features. Remember to set the target build of the bug you're working with according to: [[Release Engineering/Plans/1.1]] and [[Release Engineering/Plans/1.2]]

When you add a new bug, add correct dependencies to the corresponding [[Quality/QA-tools#Roadmap|roadmap]] meta bug.

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS test automation design|MCTS Test Automation]] (QA Tools support for MCTS)

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

* Team meetings
** QA tools team meetings will be held on need basis for specific topics. This was agreed in [http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-07-11.59.html the last weekly meeting held on December 7th 2010]
** [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

* Architecture meetings
** Architecture meetings will be held on-demand basis.
** [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

* Workshops
** Face-2-face meetings within the team.
** [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarmo Savinen || Team member || Digia || jasavi
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
| Sami Hangaslammi || Team member || Leonidas || sahangas
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
** [http://timoph.fi/qa-tools-stats/ #meego-qa-tools statistics]
** [http://timoph.fi/qa-tools-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools

2011-02-04T08:47:31Z

Samposa: /* Tools and Maintainers */

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://meego.gitorious.org/meego-quality-assurance/ git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner|Testrunner]] || [http://meego.gitorious.org/meego-quality-assurance/testrunner Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://meego.gitorious.org/meego-quality-assurance/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://meego.gitorious.org/meego-quality-assurance/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| [[Quality/TestSuite/MCTS|MeeGo Core Test Suite]] || [http://meego.gitorious.org/meego-quality-assurance/mcts/ Gitorious] ||Matti Salmi || Jeff Zheng
|-
| [[Quality/Netbook Test Suite and Utilities|MeeGo Netbook Test Suite]] || [http://gitorious.org/qa-tools/mnts/ Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://meego.gitorious.org/meego-quality-assurance/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://meego.gitorious.org/meego-quality-assurance/rich-core Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter settings || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter-settings-public Gitorious] || Carol Rus || Raimo Gratseff
|-
| [[Quality/QA-tools/hat-control|Hardware Accessory for Testing (HAT)]] || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| [[Quality/QA-tools/QAReports|QA Reports]] || [http://meego.gitorious.org/meego-quality-assurance/qa-reports Gitorious] || Sami Hangaslammi || Jarno Keskikangas
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] || [http://meego.gitorious.com/meego-quality-assurance/scripts-and-utils Gitorious] || N/A || N/A
|-
| [[Quality/TestSuite/handset-test-suite/handset-ux-test_Releases | handset_ux_tests]] || [http://meego.gitorious.com/meego-quality-assurance/handset-ux-tests Gitorious] || JessicaJi || N/A
|-
| [[Quality/QA-tools/MeeGo_Fast_Feedback_Testing|MeeGo Fast Feedback Testing (MeeGo-FFT)]] || [http://meego.gitorious.org/meego-quality-assurance/meego-testing-hudson-plugin Gitorious] || Alexey Kuznetsov || Timo Härkönen
|-
| Service OS based Flasher || [http://meego.gitorious.org/meego-quality-assurance/meego-ai-serviceos Gitorious] || Jing Wang || N/A
|-
| Qpid C wrapper library - libcqpid || [http://meego.gitorious.org/meego-quality-assurance/libcqpid Gitorious] || Sami Lahtinen || N/A
|}

You can install Testrunner, testrunner-lite, test-definition, Testplanner, OTS, Meego-ai, libcqpid and eat from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here is the workflow for QA tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file: [[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Roadmap ==
(We need still rough estimates for releases -timakima)

The release dates defined in MeeGo Release plans [[Release_Engineering/Plans/1.1|1.1]] and [[Release_Engineering/Plans/1.2|1.2]]. These dates are the latest estimation. They will be updated as work progresses.

The features in the roadmap are followed with META FEA bugs. The features are split to small tool specific FEA:s that block the feature META bug. The META bug is then the last bug to be closed when the feature is finished. You can also follow the tool bug progression from the dependency trees of the META bugs.

Adding a new feature to the roadmap:
# Add a feature bug describing the main purpose of the feature with META and FEA tags.
# Add separate bugs for each tool specific change
# Add correct dependencies between tool specific bugs (e.g. testrunner bug depends on testrunner-lite bug that depends on test-definition bug...)
# Make all of the bugs block the META bug

{| border="1" cellspacing="0"
|+
|'''Feature''' || '''Release'''
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12980&hide_resolved=0 Measurement support] || 1.2
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12937&hide_resolved=0 MCTS coverage support] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12833&hide_resolved=0 Test environment validation] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=13092&hide_resolved=0 Test equipment control] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12454&hide_resolved=0 Parallel testing] || TBD
|-
| [https://bugs.meego.com/show_bug.cgi?id=12810 Easy install] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12971&hide_resolved=0 Qt Creator integration] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12981&hide_resolved=0 Events feature in automatic testing] || TBD
|}

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=MeeGo%20Quality%20Assurance Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

We'd like to have the best guess of the moment about the delivery of features. Remember to set the target build of the bug you're working with according to: [[Release Engineering/Plans/1.1]] and [[Release Engineering/Plans/1.2]]

When you add a new bug, add correct dependencies to the corresponding [[Quality/QA-tools#Roadmap|roadmap]] meta bug.

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS test automation design|MCTS Test Automation]] (QA Tools support for MCTS)

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

* Team meetings
** QA tools team meetings will be held on need basis for specific topics. This was agreed in [http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-07-11.59.html the last weekly meeting held on December 7th 2010]
** [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

* Architecture meetings
** Architecture meetings will be held on-demand basis.
** [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

* Workshops
** Face-2-face meetings within the team.
** [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarmo Savinen || Team member || Digia || jasavi
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
| Sami Hangaslammi || Team member || Leonidas || sahangas
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
** [http://timoph.fi/qa-tools-stats/ #meego-qa-tools statistics]
** [http://timoph.fi/qa-tools-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools

2011-02-04T08:17:14Z

Samposa: /* Features and Bugs */

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://meego.gitorious.org/meego-quality-assurance/ git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner|Testrunner]] || [http://meego.gitorious.org/meego-quality-assurance/testrunner Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://meego.gitorious.org/meego-quality-assurance/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://meego.gitorious.org/meego-quality-assurance/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| [[Quality/TestSuite/MCTS|MeeGo Core Test Suite]] || [http://meego.gitorious.org/meego-quality-assurance/mcts/ Gitorious] ||Matti Salmi || Jeff Zheng
|-
| [[Quality/Netbook Test Suite and Utilities|MeeGo Netbook Test Suite]] || [http://gitorious.org/qa-tools/mnts/ Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://meego.gitorious.org/meego-quality-assurance/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://meego.gitorious.org/meego-quality-assurance/rich-core Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter Gitorious] || Carol Rus || Raimo Gratseff
|-
| Crash Reporter settings || [http://meego.gitorious.org/meego-quality-assurance/crash-reporter-settings-public Gitorious] || Carol Rus || Raimo Gratseff
|-
| [[Quality/QA-tools/hat-control|Hardware Accessory for Testing (HAT)]] || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| [[Quality/QA-tools/QAReports|QA Reports]] || [http://meego.gitorious.org/meego-quality-assurance/qa-reports Gitorious] || Sami Hangaslammi || Jarno Keskikangas
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] || [http://meego.gitorious.com/meego-quality-assurance/scripts-and-utils Gitorious] || N/A || N/A
|-
| [[Quality/TestSuite/handset-test-suite/handset-ux-test_Releases | handset_ux_tests]] || [http://meego.gitorious.com/meego-quality-assurance/handset-ux-tests Gitorious] || JessicaJi || N/A
|-
| [[Quality/QA-tools/MeeGo_Fast_Feedback_Testing|MeeGo Fast Feedback Testing (MeeGo-FFT)]] || [http://meego.gitorious.org/meego-quality-assurance/meego-testing-hudson-plugin Gitorious] || Alexey Kuznetsov || Timo Härkönen
|-
| Service OS based Flasher || [http://meego.gitorious.org/meego-quality-assurance/meego-ai-serviceos Gitorious] || Jing Wang || N/A
|-
| Qpid C wrapper library - libcqpid || [http://meego.gitorious.org/meego-quality-assurance/libcqpid Gitorious] || Sami Lahtinen || N/A
|}

You can install Testrunner, testrunner-lite, Testplanner, OTS, Meego-ai and eat from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here is the workflow for QA tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file: [[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Roadmap ==
(We need still rough estimates for releases -timakima)

The release dates defined in MeeGo Release plans [[Release_Engineering/Plans/1.1|1.1]] and [[Release_Engineering/Plans/1.2|1.2]]. These dates are the latest estimation. They will be updated as work progresses.

The features in the roadmap are followed with META FEA bugs. The features are split to small tool specific FEA:s that block the feature META bug. The META bug is then the last bug to be closed when the feature is finished. You can also follow the tool bug progression from the dependency trees of the META bugs.

Adding a new feature to the roadmap:
# Add a feature bug describing the main purpose of the feature with META and FEA tags.
# Add separate bugs for each tool specific change
# Add correct dependencies between tool specific bugs (e.g. testrunner bug depends on testrunner-lite bug that depends on test-definition bug...)
# Make all of the bugs block the META bug

{| border="1" cellspacing="0"
|+
|'''Feature''' || '''Release'''
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12980&hide_resolved=0 Measurement support] || 1.2
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12937&hide_resolved=0 MCTS coverage support] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12833&hide_resolved=0 Test environment validation] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=13092&hide_resolved=0 Test equipment control] || TBD
|-
| [http://bugs.meego.com/showdependencytree.cgi?id=12454&hide_resolved=0 Parallel testing] || TBD
|-
| [https://bugs.meego.com/show_bug.cgi?id=12810 Easy install] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12971&hide_resolved=0 Qt Creator integration] || TBD
|-
| [https://bugs.meego.com/showdependencytree.cgi?id=12981&hide_resolved=0 Events feature in automatic testing] || TBD
|}

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=MeeGo%20Quality%20Assurance Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=Fast%20Feedback%20Testing&component=hat&component=libcqpid&component=meego-qa-reports&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=Testrunner&component=testrunner-lite&component=Others&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

We'd like to have the best guess of the moment about the delivery of features. Remember to set the target build of the bug you're working with according to: [[Release Engineering/Plans/1.1]] and [[Release Engineering/Plans/1.2]]

When you add a new bug, add correct dependencies to the corresponding [[Quality/QA-tools#Roadmap|roadmap]] meta bug.

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS test automation design|MCTS Test Automation]] (QA Tools support for MCTS)

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

* Team meetings
** QA tools team meetings will be held on need basis for specific topics. This was agreed in [http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-07-11.59.html the last weekly meeting held on December 7th 2010]
** [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

* Architecture meetings
** Architecture meetings will be held on-demand basis.
** [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

* Workshops
** Face-2-face meetings within the team.
** [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarmo Savinen || Team member || Digia || jasavi
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
| Sami Hangaslammi || Team member || Leonidas || sahangas
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
** [http://timoph.fi/qa-tools-stats/ #meego-qa-tools statistics]
** [http://timoph.fi/qa-tools-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools/Testrunner-lite

2011-01-18T16:41:23Z

Samposa: /* Testrunner-lite and test case verdicts */

= Testrunner-lite =

Testrunner-lite is a lightweight tool for test execution, which reads [[Quality/QA-tools/Test plan|Test Plan XML]] files as input, and produces [http://gitorious.org/qa-tools/test-definition/blobs/master/src/data/testdefinition-results.xsd a common-format] test result file which can be published at [http://qa-reports.meego.com/ qa-reports.meego.com]. With testrunner-lite, you can
* Execute automatic, semi-automatic and manual test cases
* Execute test cases locally or in host-based mode
* Use [[Quality/QA-tools/Testrunner-lite#Filtering options|filters]] to select the test cases to be executed
* Validate the used test plan file automatically

See our demo videos at Youtube:
* [http://www.youtube.com/watch?v=TZhHDPUeHIw Demo of version 1.3.17]
* [http://www.youtube.com/watch?v=A0V94xZ_VwI Demo of version 1.3.11]
* [http://www.youtube.com/watch?v=OSF8tyUxI8U Demo of version 1.3.10]
* [http://www.youtube.com/user/meegoqatools MeeGo QA tools Youtube channel]

There is also a graphical user interface version of the tool called [[Quality/QA-tools/Testrunner|Testrunner]] available.

== Installation ==

Testrunner-lite can be installed on MeeGo Netbook UX, Ubuntu 10.04 (and newer) and Fedora 13 by doing the following:
# [[Quality/QA-tools/How to set up repositories|Set up the repository]]
# Install testrunner-lite

MeeGo Netbook 1.1:

sudo zypper install testrunner-lite

Ubuntu:

sudo apt-get install testrunner-lite

Fedora 13:

# yum install testrunner-lite

== Usage ==

The program is executed from command line

/usr/bin/testrunner-lite [options]
Options

-h, --help
Display short help and exit

-V, --version
Display version information and exit

-f FILE, --file FILE
Input file with test definitions in XML (required)

-o FILE, --output FILE
Output file format; FORMAT can be xml or text (Default xml)

-v, -vv, --verbose[={INFO|DEBUG}]
Enable verbosity mode; -v and --verbose=INFO are equivalent outputting INFO, ERROR and WARNING messages. Similarly -vv and --verbose=DEBUG are equivalent, outputting also debug messages. Default behaviour is silent mode.

-L URL, --logger=URL
Remote HTTP logger. Log messages are sent to given URL in a HTTP POST message. URL format is [http://]host[:port][/path/], where host may be a hostname or an IPv4 address.

-a, --automatic
Enable only automatic tests to be executed

-S, --syslog
Enable logging to syslog

-P, --print-step-output
Output standard streams from test steps.

-m, --manual
Enable only manual tests to be executed

-c, --ci
Disable validation of test definition against schema

-s, --semantic
Enable validation of test definition against stricter (semantics) schema

-A, --validate-only
Do only input xml validation, do not execute tests

-H, --no-hwinfo
Do not try to obtain hardware information

-t [USER@]ADDRESS, --target=[USER@]ADDRESS
Enable host-based testing. If given, commands are executed from test control PC (host) side. ADDRESS is the ipv4 adress of the system under test

-M, --disable-measurement-verdict
Do not fail cases based on measurement data

-C PATH, --chroot=PATH
Run tests inside a chroot environment. Not that this doesn't change the root of the testrunner itself, only the tests will have the new root folder set

=== Filtering options ===

Filtering options allow selecting which tests are to be executed. Filtering options are given as a string. String may contain one or more filter entries. Format of each entry is 'filtername=values' where 'filtername' typically corresponds to an attribute in Test Definition XML and 'values' is a string to which the filter should match. Multiple values can be specified if separated by comma (without extra space). A value containing space must be enclosed in double quotes (""). Current list of forbidden characters in values is as follows: Single quote ('), double quote ("), is-equal-to sign (=), comma (,).

How filtering is carried out: Before processing the filter options, all tests are active by default. Filters can only deactivate tests: If filter value does not match with the value of the corresponding attribute, the test is disabled. Each filter entry is applied for all the active tests in the test package, one after another, in the given order. Filtering is always carried out at the lowest level of the "suite-set-case-step" hierarchy where the corresponding attribute can be defined. Note that some attributes may inherit the value from the upper level. Please refer to Test Definition XML for details.

If 'filtername' is prefixed with dash (-), the filter behaviour is reversed: those tests for which the filter value does match, are disabled.

'filtername' can be any of the following:

* feature
* requirement
* testset
* type
* testcase

The following example does the following: First it disables all test cases except those with attribute type='Integration'. Next, a test case named as Bad Test is disabled. The tests left active will be executed.

testrunner-lite -f tests.xml -o ~/results -l 'type=Integration -testcase="Bad Test"'

The following example executes test cases that specify the requirement attribute with a value containing at least one of the following strings: '50001', '50002', '50003'.

testrunner-lite -f tests.xml -o ~/results -l 'requirement=50001,50002,50003'

Note that each key=values is handled as a separate filter, when checking whether a test case should be filtered. The following example will filter all the test cases,

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1 testset=set2'

whereas the following will accept tests from test sets "set1" and "set2".

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1,set2'

== Manual Test Cases ==

It is also possible to execute manual test cases using testrunner-lite, as defined in Test Definition XML.

In case manual test case is encountered during execution, testrunner-lite will go through the defined test steps and ask user whether the step is passed or failed. The test case will terminate at the first failure, otherwise every step defined will be executed. After the test case is done, user has the option to enter additional comments.

Example output when running manual case:

[INFO] 15:15:53 Starting test case: ExampleTestCase
--- Execute test step ---
Description:
Open calculator. Expected result: calculator opens up.

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Execute test step ---
Description:
Stop calculator

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Test steps executed, case is PASSED ---
Please enter additional comments (ENTER to finish):
Execution was slow.

[INFO] 15:16:41 Finished test case. Result: PASS

== Testrunner-lite and test case verdicts ==

For manual cases the result is naturally set by the user. For '''automatic''' cases a result of a test case is set as follows:
* PASS
** all test steps finished with expected results
* FAIL
** test step has unexpected return value
** test step timed out
** pre-steps on the set the case belongs to have failed
** fail based on [[Quality/QA-tools/Test_plan#Measurement_data|measurement data]]
* N/A
** test case has state=Design in the tests.xml
** test case has no steps

If a test case is filtered it doesn't get any result; meaning it is not written to results file at all.

The goal is to align testrunner-lite functionality with what is agreed [[Quality/Glossary#Test_case_verdict|here]].

== About process control ==

Each test step is executed in a separate shell. Testrunner-lite spawns new process for the execution, and waits for the step to finish (or timeout). In case test step contains command that is started to background, the step returns immediately. After test case has finnished a cleanup routine is executed, where testrunner-lite tries to kill all processes that may have been left running by the test steps. Cleanup for pre-steps and post-steps is done after the post steps are executed (i.e. when test set has been executed).

== About host based execution ==

Testrunner-lite supports host based execution, where testrunner-lite is executed on a PC, and the test steps over ssh on hardware. This requires that key based ssh authentication is enabled between device and host.

Remember that each single step in pre_steps, post_steps or inside testcases is executed in a separate SSH session, so you'll have to make sure that if your steps leave some processes running in the background which inherits its pipes straight from the parent shell, the SSH connection will hang until those processes are terminated (or until they close their pipes). Therefore, you should always redirect the stdout, stderr and stdin streams of your background processes, if you don't want your test steps / pre- / post steps to time out in host-based execution.

See http://www.snailbook.com/faq/background-jobs.auto.html for more information.

== Testrunner-lite state machine ==

Below is a simplified picture describing the operation of testrunner-lite. The program flow can be considered to consist of three phases: initializing, executing and clean up. The execution is done set by set, so that the tool needs to maintain only one set at a time in its memory.

[[File:testrunnerlitefsm.png]]

xfig file:
[[File:testrunnerlitesm.fig]]

== Further development ==
We're actively making testrunner-lite a better tool. See discussion about architecture changes and suggestions in following page:
[[Quality/QA-tools/Testrunner-lite/Testrunner-lite_development|Testrunner-lite development]]

Quality/QA-tools/Testrunner-lite

2011-01-18T16:39:22Z

Samposa: /* Usage */

= Testrunner-lite =

Testrunner-lite is a lightweight tool for test execution, which reads [[Quality/QA-tools/Test plan|Test Plan XML]] files as input, and produces [http://gitorious.org/qa-tools/test-definition/blobs/master/src/data/testdefinition-results.xsd a common-format] test result file which can be published at [http://qa-reports.meego.com/ qa-reports.meego.com]. With testrunner-lite, you can
* Execute automatic, semi-automatic and manual test cases
* Execute test cases locally or in host-based mode
* Use [[Quality/QA-tools/Testrunner-lite#Filtering options|filters]] to select the test cases to be executed
* Validate the used test plan file automatically

See our demo videos at Youtube:
* [http://www.youtube.com/watch?v=TZhHDPUeHIw Demo of version 1.3.17]
* [http://www.youtube.com/watch?v=A0V94xZ_VwI Demo of version 1.3.11]
* [http://www.youtube.com/watch?v=OSF8tyUxI8U Demo of version 1.3.10]
* [http://www.youtube.com/user/meegoqatools MeeGo QA tools Youtube channel]

There is also a graphical user interface version of the tool called [[Quality/QA-tools/Testrunner|Testrunner]] available.

== Installation ==

Testrunner-lite can be installed on MeeGo Netbook UX, Ubuntu 10.04 (and newer) and Fedora 13 by doing the following:
# [[Quality/QA-tools/How to set up repositories|Set up the repository]]
# Install testrunner-lite

MeeGo Netbook 1.1:

sudo zypper install testrunner-lite

Ubuntu:

sudo apt-get install testrunner-lite

Fedora 13:

# yum install testrunner-lite

== Usage ==

The program is executed from command line

/usr/bin/testrunner-lite [options]
Options

-h, --help
Display short help and exit

-V, --version
Display version information and exit

-f FILE, --file FILE
Input file with test definitions in XML (required)

-o FILE, --output FILE
Output file format; FORMAT can be xml or text (Default xml)

-v, -vv, --verbose[={INFO|DEBUG}]
Enable verbosity mode; -v and --verbose=INFO are equivalent outputting INFO, ERROR and WARNING messages. Similarly -vv and --verbose=DEBUG are equivalent, outputting also debug messages. Default behaviour is silent mode.

-L URL, --logger=URL
Remote HTTP logger. Log messages are sent to given URL in a HTTP POST message. URL format is [http://]host[:port][/path/], where host may be a hostname or an IPv4 address.

-a, --automatic
Enable only automatic tests to be executed

-S, --syslog
Enable logging to syslog

-P, --print-step-output
Output standard streams from test steps.

-m, --manual
Enable only manual tests to be executed

-c, --ci
Disable validation of test definition against schema

-s, --semantic
Enable validation of test definition against stricter (semantics) schema

-A, --validate-only
Do only input xml validation, do not execute tests

-H, --no-hwinfo
Do not try to obtain hardware information

-t [USER@]ADDRESS, --target=[USER@]ADDRESS
Enable host-based testing. If given, commands are executed from test control PC (host) side. ADDRESS is the ipv4 adress of the system under test

-M, --disable-measurement-verdict
Do not fail cases based on measurement data

-C PATH, --chroot=PATH
Run tests inside a chroot environment. Not that this doesn't change the root of the testrunner itself, only the tests will have the new root folder set

=== Filtering options ===

Filtering options allow selecting which tests are to be executed. Filtering options are given as a string. String may contain one or more filter entries. Format of each entry is 'filtername=values' where 'filtername' typically corresponds to an attribute in Test Definition XML and 'values' is a string to which the filter should match. Multiple values can be specified if separated by comma (without extra space). A value containing space must be enclosed in double quotes (""). Current list of forbidden characters in values is as follows: Single quote ('), double quote ("), is-equal-to sign (=), comma (,).

How filtering is carried out: Before processing the filter options, all tests are active by default. Filters can only deactivate tests: If filter value does not match with the value of the corresponding attribute, the test is disabled. Each filter entry is applied for all the active tests in the test package, one after another, in the given order. Filtering is always carried out at the lowest level of the "suite-set-case-step" hierarchy where the corresponding attribute can be defined. Note that some attributes may inherit the value from the upper level. Please refer to Test Definition XML for details.

If 'filtername' is prefixed with dash (-), the filter behaviour is reversed: those tests for which the filter value does match, are disabled.

'filtername' can be any of the following:

* feature
* requirement
* testset
* type
* testcase

The following example does the following: First it disables all test cases except those with attribute type='Integration'. Next, a test case named as Bad Test is disabled. The tests left active will be executed.

testrunner-lite -f tests.xml -o ~/results -l 'type=Integration -testcase="Bad Test"'

The following example executes test cases that specify the requirement attribute with a value containing at least one of the following strings: '50001', '50002', '50003'.

testrunner-lite -f tests.xml -o ~/results -l 'requirement=50001,50002,50003'

Note that each key=values is handled as a separate filter, when checking whether a test case should be filtered. The following example will filter all the test cases,

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1 testset=set2'

whereas the following will accept tests from test sets "set1" and "set2".

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1,set2'

== Manual Test Cases ==

It is also possible to execute manual test cases using testrunner-lite, as defined in Test Definition XML.

In case manual test case is encountered during execution, testrunner-lite will go through the defined test steps and ask user whether the step is passed or failed. The test case will terminate at the first failure, otherwise every step defined will be executed. After the test case is done, user has the option to enter additional comments.

Example output when running manual case:

[INFO] 15:15:53 Starting test case: ExampleTestCase
--- Execute test step ---
Description:
Open calculator. Expected result: calculator opens up.

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Execute test step ---
Description:
Stop calculator

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Test steps executed, case is PASSED ---
Please enter additional comments (ENTER to finish):
Execution was slow.

[INFO] 15:16:41 Finished test case. Result: PASS

== Testrunner-lite and test case verdicts ==

For manual cases the result is naturally set by the user. For '''automatic''' cases a result of a test case is set as follows:
* PASS
** all test steps finished with expected results
* FAIL
** test step has unexpected return value
** test step timed out
** pre-steps on the set the case belongs to have failed
** fail based on measurement data (up-coming feature)
* N/A
** test case has state=Design in the tests.xml
** test case has no steps

If a test case is filtered it doesn't get any result; meaning it is not written to results file at all.

The goal is to align testrunner-lite functionality with what is agreed [[Quality/Glossary#Test_case_verdict|here]].

== About process control ==

Each test step is executed in a separate shell. Testrunner-lite spawns new process for the execution, and waits for the step to finish (or timeout). In case test step contains command that is started to background, the step returns immediately. After test case has finnished a cleanup routine is executed, where testrunner-lite tries to kill all processes that may have been left running by the test steps. Cleanup for pre-steps and post-steps is done after the post steps are executed (i.e. when test set has been executed).

== About host based execution ==

Testrunner-lite supports host based execution, where testrunner-lite is executed on a PC, and the test steps over ssh on hardware. This requires that key based ssh authentication is enabled between device and host.

Remember that each single step in pre_steps, post_steps or inside testcases is executed in a separate SSH session, so you'll have to make sure that if your steps leave some processes running in the background which inherits its pipes straight from the parent shell, the SSH connection will hang until those processes are terminated (or until they close their pipes). Therefore, you should always redirect the stdout, stderr and stdin streams of your background processes, if you don't want your test steps / pre- / post steps to time out in host-based execution.

See http://www.snailbook.com/faq/background-jobs.auto.html for more information.

== Testrunner-lite state machine ==

Below is a simplified picture describing the operation of testrunner-lite. The program flow can be considered to consist of three phases: initializing, executing and clean up. The execution is done set by set, so that the tool needs to maintain only one set at a time in its memory.

[[File:testrunnerlitefsm.png]]

xfig file:
[[File:testrunnerlitesm.fig]]

== Further development ==
We're actively making testrunner-lite a better tool. See discussion about architecture changes and suggestions in following page:
[[Quality/QA-tools/Testrunner-lite/Testrunner-lite_development|Testrunner-lite development]]

Quality/QA-tools/Testrunner-lite

2011-01-18T16:37:49Z

Samposa: /* Usage */

= Testrunner-lite =

Testrunner-lite is a lightweight tool for test execution, which reads [[Quality/QA-tools/Test plan|Test Plan XML]] files as input, and produces [http://gitorious.org/qa-tools/test-definition/blobs/master/src/data/testdefinition-results.xsd a common-format] test result file which can be published at [http://qa-reports.meego.com/ qa-reports.meego.com]. With testrunner-lite, you can
* Execute automatic, semi-automatic and manual test cases
* Execute test cases locally or in host-based mode
* Use [[Quality/QA-tools/Testrunner-lite#Filtering options|filters]] to select the test cases to be executed
* Validate the used test plan file automatically

See our demo videos at Youtube:
* [http://www.youtube.com/watch?v=TZhHDPUeHIw Demo of version 1.3.17]
* [http://www.youtube.com/watch?v=A0V94xZ_VwI Demo of version 1.3.11]
* [http://www.youtube.com/watch?v=OSF8tyUxI8U Demo of version 1.3.10]
* [http://www.youtube.com/user/meegoqatools MeeGo QA tools Youtube channel]

There is also a graphical user interface version of the tool called [[Quality/QA-tools/Testrunner|Testrunner]] available.

== Installation ==

Testrunner-lite can be installed on MeeGo Netbook UX, Ubuntu 10.04 (and newer) and Fedora 13 by doing the following:
# [[Quality/QA-tools/How to set up repositories|Set up the repository]]
# Install testrunner-lite

MeeGo Netbook 1.1:

sudo zypper install testrunner-lite

Ubuntu:

sudo apt-get install testrunner-lite

Fedora 13:

# yum install testrunner-lite

== Usage ==

The program is executed from command line

/usr/bin/testrunner-lite [options]
Options

-h, --help
Display short help and exit

-V, --version
Display version information and exit

-f FILE, --file FILE
Input file with test definitions in XML (required)

-o FILE, --output FILE
Output file format; FORMAT can be xml or text (Default xml)

-v, -vv, --verbose[={INFO|DEBUG}]
Enable verbosity mode; -v and --verbose=INFO are equivalent outputting INFO, ERROR and WARNING messages. Similarly -vv and --verbose=DEBUG are equivalent, outputting also debug messages. Default behaviour is silent mode.

-L URL, --logger=URL
Remote HTTP logger. Log messages are sent to given URL in a HTTP POST message. URL format is [http://]host[:port][/path/], where host may be a hostname or an IPv4 address.

-a, --automatic
Enable only automatic tests to be executed

-S, --syslog
Enable logging to syslog

-P, --print-step-output
Output standard streams from test steps.

-m, --manual
Enable only manual tests to be executed

-c, --ci
Disable validation of test definition against schema

-s, --semantic
Enable validation of test definition against stricter (semantics) schema

-A, --validate-only
Do only input xml validation, do not execute tests

-H, --no-hwinfo
Do not try to obtain hardware information

-t [USER@]ADDRESS, --target=[USER@]ADDRESS
Enable host-based testing. If given, commands are executed from test control PC (host) side. ADDRESS is the ipv4 adress of the system under test

-M, --disable-measurement-verdict
Do not fail cases based on measurement data.

-C PATH, --chroot=PATH
Run tests inside a chroot environment. Not that this doesn't change the root of the testrunner itself, only the tests will have the new root folder set

=== Filtering options ===

Filtering options allow selecting which tests are to be executed. Filtering options are given as a string. String may contain one or more filter entries. Format of each entry is 'filtername=values' where 'filtername' typically corresponds to an attribute in Test Definition XML and 'values' is a string to which the filter should match. Multiple values can be specified if separated by comma (without extra space). A value containing space must be enclosed in double quotes (""). Current list of forbidden characters in values is as follows: Single quote ('), double quote ("), is-equal-to sign (=), comma (,).

How filtering is carried out: Before processing the filter options, all tests are active by default. Filters can only deactivate tests: If filter value does not match with the value of the corresponding attribute, the test is disabled. Each filter entry is applied for all the active tests in the test package, one after another, in the given order. Filtering is always carried out at the lowest level of the "suite-set-case-step" hierarchy where the corresponding attribute can be defined. Note that some attributes may inherit the value from the upper level. Please refer to Test Definition XML for details.

If 'filtername' is prefixed with dash (-), the filter behaviour is reversed: those tests for which the filter value does match, are disabled.

'filtername' can be any of the following:

* feature
* requirement
* testset
* type
* testcase

The following example does the following: First it disables all test cases except those with attribute type='Integration'. Next, a test case named as Bad Test is disabled. The tests left active will be executed.

testrunner-lite -f tests.xml -o ~/results -l 'type=Integration -testcase="Bad Test"'

The following example executes test cases that specify the requirement attribute with a value containing at least one of the following strings: '50001', '50002', '50003'.

testrunner-lite -f tests.xml -o ~/results -l 'requirement=50001,50002,50003'

Note that each key=values is handled as a separate filter, when checking whether a test case should be filtered. The following example will filter all the test cases,

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1 testset=set2'

whereas the following will accept tests from test sets "set1" and "set2".

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1,set2'

== Manual Test Cases ==

It is also possible to execute manual test cases using testrunner-lite, as defined in Test Definition XML.

In case manual test case is encountered during execution, testrunner-lite will go through the defined test steps and ask user whether the step is passed or failed. The test case will terminate at the first failure, otherwise every step defined will be executed. After the test case is done, user has the option to enter additional comments.

Example output when running manual case:

[INFO] 15:15:53 Starting test case: ExampleTestCase
--- Execute test step ---
Description:
Open calculator. Expected result: calculator opens up.

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Execute test step ---
Description:
Stop calculator

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Test steps executed, case is PASSED ---
Please enter additional comments (ENTER to finish):
Execution was slow.

[INFO] 15:16:41 Finished test case. Result: PASS

== Testrunner-lite and test case verdicts ==

For manual cases the result is naturally set by the user. For '''automatic''' cases a result of a test case is set as follows:
* PASS
** all test steps finished with expected results
* FAIL
** test step has unexpected return value
** test step timed out
** pre-steps on the set the case belongs to have failed
** fail based on measurement data (up-coming feature)
* N/A
** test case has state=Design in the tests.xml
** test case has no steps

If a test case is filtered it doesn't get any result; meaning it is not written to results file at all.

The goal is to align testrunner-lite functionality with what is agreed [[Quality/Glossary#Test_case_verdict|here]].

== About process control ==

Each test step is executed in a separate shell. Testrunner-lite spawns new process for the execution, and waits for the step to finish (or timeout). In case test step contains command that is started to background, the step returns immediately. After test case has finnished a cleanup routine is executed, where testrunner-lite tries to kill all processes that may have been left running by the test steps. Cleanup for pre-steps and post-steps is done after the post steps are executed (i.e. when test set has been executed).

== About host based execution ==

Testrunner-lite supports host based execution, where testrunner-lite is executed on a PC, and the test steps over ssh on hardware. This requires that key based ssh authentication is enabled between device and host.

Remember that each single step in pre_steps, post_steps or inside testcases is executed in a separate SSH session, so you'll have to make sure that if your steps leave some processes running in the background which inherits its pipes straight from the parent shell, the SSH connection will hang until those processes are terminated (or until they close their pipes). Therefore, you should always redirect the stdout, stderr and stdin streams of your background processes, if you don't want your test steps / pre- / post steps to time out in host-based execution.

See http://www.snailbook.com/faq/background-jobs.auto.html for more information.

== Testrunner-lite state machine ==

Below is a simplified picture describing the operation of testrunner-lite. The program flow can be considered to consist of three phases: initializing, executing and clean up. The execution is done set by set, so that the tool needs to maintain only one set at a time in its memory.

[[File:testrunnerlitefsm.png]]

xfig file:
[[File:testrunnerlitesm.fig]]

== Further development ==
We're actively making testrunner-lite a better tool. See discussion about architecture changes and suggestions in following page:
[[Quality/QA-tools/Testrunner-lite/Testrunner-lite_development|Testrunner-lite development]]

Quality/QA-tools/Test plan

2011-01-18T16:30:06Z

Samposa: /* Fetching Additional Files */

== Test plan overview ==

The basic principle of test plan XML and tool support is that you can use 'any' executable for testing.
Test results are checked from exit codes (automated testing) or prompt (manual testing). Executed test plan XMLs produce XML results - wrapping variety of test methods in consistent format required by test automation and data processing.

The test plan information stored in Test Definition XML files consists of:

1. suite, set, case: Hierarchical structure of the tests.
2. feature, subfeature, requirement: Information about why the tested software has been implemented in the first place (which is why it’s being tested as well).
3. type: Information about the viewpoint of the tests (which quality aspect of the software they are testing, see DevelopmentTestArea in Agile Testing Wiki).
4. level: Information about which test level the tests belong - for test level.
5. domain: Information about which architectural domains the tests are focused on.
6. description: descriptions of the tests (what each test is, and what it is supposed to do).
7. step: Execution instructions (for automated tests), which determine the actual commands to execute to run each test.

''Note:'' that all of the above are not mandatory. Mandatory fields for executing tests are defined with test plan XML validation schema.

== Creating Test plan ==

There are certain mandatory things you’ll need to provide in a test definition XML.

The structure of the XML and the possible attributes/values are defined in the [[Quality/QA-tools/Test-definition|Test Definition XML schema]]. It’s important to validate your XML against the up-to-date schema.

Test plans can be created either by using [[Quality/QA-tools/Testplanner|Testplanner]] tool or manually.

=== Test Cases ===

The main thing in test plan are the test cases, which define what to execute (test steps), what to expect from the execution (expected results) and also some additional information for reporting purposes. An example test case could be:

<case name="my-first-case">
<description>Creating my very first test case</description>
<step>ls</step>
<step>uname -r</step>
</case>

What did we just do? We created a simple test case [[#About_name_attribute|named]] <code>my-first-test-case</code>, which executes two steps and expects that they return the common “success return value” 0. Want to check for some other return value? Use <code>expected_result</code>:

<step expected_result="-1">ls</step>

In case you feel that some test cases are insignificant, i.e. that they shouldn’t be taken into consideration when choosing whether the test run is passed or failed, you can use insignificant-attribute (defaults to <code>false</code>, i.e. every case is considered to be significant):

<case name="not-so-important-test" insignificant="true">

In addition to test execution, the “story” of the test case should also be told, and for telling this story, we have fields for test type (what quality characteristics are tested) and test level (what level of system is being exercised. The possible values for both of these are listed in the appendices. Let’s add level and type to our test case:

<case name="my-first-case" level="Product" type="Functional">
<description>Creating my very first test case</description>
<step expected_result="-1">ls</step>
<step>uname -r</step>
</case>

And that’s our first test case! Not that difficult...

=== Grouping Cases - Sets and Suites ===

Grouping cases to sets and suites makes your life easier and more organized. A good idea is to group test cases that test the same features into same sets, and sets that test the same architectural domain under same suites. E.g.:

<suite name="my-multimedia-tests" domain="Multimedia">
<set name="video-playback-tests" feature="Video Playback">
<description>Video</description>
<case ...
<case ...
</set>
</suite>

So, sets contain test cases, and may also contain description (the same way as the cases do). More about sets in Test Definition Execution part

'''Note:''' There are several attributes that need to be defined on the test case, test set, or test suite level in the test definition XML.<br /> You can find more information about these attributes in the Appendices section of this page.

=== Time outs ===
Test steps will time out by default after 90 seconds. You can change the default time out by adding timeout attribute to your test case. for example

<case name="my-test-case" timeout="120">

Note that this the timeout is for single test step not for the whole case.

For [pre|post]-steps the the default timeout is 180 seconds, it can also be changed:

<pre_steps timeout="600">

=== Putting It All Together ===

Now that you have some knowledge on grouping and test cases in general, it is time to put it all together. Before showing an example, it is important to note that all the mentioned case attributes (e.g. level, type) are inheritable. So, if you have e.g. a set which contains only certain type of cases, you can define the type at the set-level, instead of writing it separately into each case.

Now, the example. Let’s first add the mandatory tags, which each test definition should have:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
</testdefinition>

That’s the mandatory part; we defined that we have a XML-document, and that this particular one is test definition version 1.0. Let’s add more beef to the bones by adding one suite with couple of sets:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
<suite name="my-multimedia-tests" domain="Multimedia">
<description>Testing AF stuff</description>
<set name="video-playback-tests" feature="Video Playback">
<description>Video playback tests</description>
</set>
<set name="video-recording-tests" feature="Video Recording">
<description>Video recording tests</description>
</set>
</suite>
</testdefinition>

Okay, we have one suite named <code>my-multimedia-tests with two sets</code>, testing video playback and recording features. Cases are still missing:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
<suite name="my-multimedia-tests" domain="Multimedia">
<description>Video playback tests</description>
<set name="video-playback-tests" feature="Video Playback">
<description>Video playback tests</description>
<case name="playback1" type="Functional" level="Component">
<step>execute_playback_test</step>
</case>
</set>
<set name="video-recording-tests" feature="Video Recording">
<description>Video playback tests</description>
<case ...
</set>
</suite>
</testdefinition>

And that’s the basic story. Next chapters cover more details about the definition, but you should now have the general understanding of the subject!

== Controlling Environment for Execution ==

=== Setup and Teardown ===

In case you want to do some setup and cleaning before and after the cases are executed, sets may have pre- and post-steps in them:

<set ...>
<pre_steps>
<step>do_some_setup</step>
</pre_steps>
<case ...
<post_steps>
<step>clean_up</step>
</post_steps>
</set>

If you want to ensure that pre-step is executed properly before starting the real testing, you can use expected result also in those:

<pre_steps>
<step expected_result="1">do_some_setup_that_may_fail</step>
...

:'''Warning''' When using expected result in this context, the process return value will be waited for, thus you cannot use this with daemon or background processes since they basically never return and cause execution to jam. Another thing to note is that the steps are executed in separate shells, so it is not possible to e.g. set environment variables or change directories in pre-steps as those will be lost.

=== Filtering Based on Hardware Identifier ===

If different tests sets for different hardware are required then hwiddetect feature can be utilised. User can define a command used to get a hardware identifier within hwiddetect tag. The hardware identifier returned by the command is matched with optional hwid attribute of a test set. If not equal, test cases in the set are skipped and are not written to the result file. A test set will never be skipped if hwid attribute has not been defined for it. You can also define multiple hwid values separated by comma for a set.

Command defined by hwiddetect can be a shell command or a separate executable. The executable should be included in the test package. Testrunner-lite removes extra whitespace and linefeeds from the the output of the hwiddetect command so that test developer does not need to care about it.

Example usage of hwiddetect:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
<hwiddetect>/usr/bin/getmyhwid</hwiddetect>
<suite name="suite1">
<set name="test_feature_X_on_hw_bar" hwid="bar">
<case name="test_X_1">
<step>echo "hwid is bar"</step>
</case>
</set>
<set name="test_feature_X_on_hw_foo" hwid="foo">
<case name="test_X_1">
<step>echo "hwid is foo"</step>
</case>
</set>
<set name="test_feature_X_on_hw_foo_or_bar" hwid="foo,bar">
<case name="test_X_1">
<step>echo "hwid is foo or bar"</step>
</case>
</set>
</suite>
</testdefinition>

=== Fetching Additional Files ===

In addition to normal result file, you can also fetch what ever files you need with get-tag:

<set ...>
...
<get>
<file>/tmp/myadditionalresult.1</file>
<file delete_after="true">/tmp/myadditionalresult.2</file>
</get>
</set>

In the example above my ''additionalresult.2'' file is deleted after fetching it (think of command ''mv'').

=== Measurement data ===

In case file is tagged as measurement data as in the below example, the data will be evaluated and transferred to results.

<case ...>
...
<get>
<file measurement="true">/path/to/measurement/measurement.txt</file>
</get>
</case>

The measurement data has the following CSV format:
name;value;unit;
name;value;unit;target;failure;
name;value;unit;target;failure;

Where ''name'' and ''unit'' are strings; ''value'' ''target'' and ''failure'' floating point numbers.
Example:
bt.upload;1.4123432;MB/s;
cpu.load;23.41;%;5;90;
mem.load;80.16;%;80;99;

If ''target'' and ''failure'' are specified the measurement can affect (fail) test case result. When ''target'' is greater than ''failure'', the ''value'' should be smaller than ''failure'' and vise versa.

== Manual, Automatic and Semi-automatic Test Cases ==

If not specified by the "manual" attribute all cases are automatic. The value for the attribute is inherited from higher entity (set->case->step). By semi-automatic test case we mean a manual case that has some automatic parts; the idea being that only those steps that have to be manual, are. Note that this does not work the other way around, we cannot have automatic test case with manual steps (would be semantically weird thus our tools do not support it).

The example below tries to clarify the above. The "example_set" shows a manual, automatic and semi-automatic case. The example works with testrunner-lite/testrunner-ui, give it a try: [[File:example_definition.xml]].

<?xml version="1.0" encoding="UTF-8"?>

<testdefinition version="1.0">
<suite name="example_suite">
<set name="example_set">
<description>Example test set with manual, automatic and semi-automatic case.</description>
<case manual="true" name="manual_case">
<description>Manual test case with three steps inside one step tag.</description>
<step>Step 1: execute command ttt on shell.
Step 2: write something into edit box.
Step 3: press ok button.
Expected: Text should be updated into label.
</step>
</case>
<case timeout="96" name="automatic_case">
<description>Automatic test case that executes some shell commands.</description>
<step>ls /tmp</step>
<step expected_result="2">ls /nosuchfile</step>
<step>pwd</step>
</case>
<case manual="true" name="semi_automatic_case">
<description>A case with two automatic and two manual steps.</description>
<step manual="false">xcalc &amp;</step>
<step>Step: Type in 2 + 2 =. Expected: 4 is displayed </step>
<step>Press x² button. Expected: 16 is displayed.</step>
<step manual="false">killall xcalc</step>
</case>
</set>
</suite>
</testdefinition>
(A real test developer / tester might come up with nicer examples, any input welcome).

== Test Plan Execution ==

Test plans are run with test execution tools such as [[Quality/QA-tools/Testrunner|Testrunner]] and [[Quality/QA-tools/Testrunner-lite|testrunner-lite]], which read the plan and produce a result XML file.

== About name attribute ==

Note that the name attribute is of type [http://www.ietf.org/rfc/rfc2396.txt anyURI]. Do not use the following characters in suite,set or case names.
*'''''; / ? : @ & = + $ ,''''' (reserved)
or
* '''''{ } | \ ^ [ ] `''''' (unwise)

Also the use of '''''<space>''''' is not recommended, since some validators accept if for anyURI, some don't. Use '''''_''''' or '''''-''''' instead.

Quality/QA-tools/Testrunner-lite

2011-01-18T06:55:29Z

Samposa: /* testrunner-lite and test case verdicts */

= Testrunner-lite =

Testrunner-lite is a lightweight tool for test execution, which reads [[Quality/QA-tools/Test plan|Test Plan XML]] files as input, and produces [http://gitorious.org/qa-tools/test-definition/blobs/master/src/data/testdefinition-results.xsd a common-format] test result file which can be published at [http://qa-reports.meego.com/ qa-reports.meego.com]. With testrunner-lite, you can
* Execute automatic, semi-automatic and manual test cases
* Execute test cases locally or in host-based mode
* Use [[Quality/QA-tools/Testrunner-lite#Filtering options|filters]] to select the test cases to be executed
* Validate the used test plan file automatically

See our demo videos at Youtube:
* [http://www.youtube.com/watch?v=TZhHDPUeHIw Demo of version 1.3.17]
* [http://www.youtube.com/watch?v=A0V94xZ_VwI Demo of version 1.3.11]
* [http://www.youtube.com/watch?v=OSF8tyUxI8U Demo of version 1.3.10]
* [http://www.youtube.com/user/meegoqatools MeeGo QA tools Youtube channel]

There is also a graphical user interface version of the tool called [[Quality/QA-tools/Testrunner|Testrunner]] available.

== Installation ==

Testrunner-lite can be installed on MeeGo Netbook UX, Ubuntu 10.04 (and newer) and Fedora 13 by doing the following:
# [[Quality/QA-tools/How to set up repositories|Set up the repository]]
# Install testrunner-lite

MeeGo Netbook 1.1:

sudo zypper install testrunner-lite

Ubuntu:

sudo apt-get install testrunner-lite

Fedora 13:

# yum install testrunner-lite

== Usage ==

The program is executed from command line

/usr/bin/testrunner-lite [options]
Options

-h, --help
Display short help and exit

-V, --version
Display version information and exit

-f FILE, --file FILE
Input file with test definitions in XML (required)

-o FILE, --output FILE
Output file format; FORMAT can be xml or text (Default xml)

-v, -vv, --verbose[={INFO|DEBUG}]
Enable verbosity mode; -v and --verbose=INFO are equivalent outputting INFO, ERROR and WARNING messages. Similarly -vv and --verbose=DEBUG are equivalent, outputting also debug messages. Default behaviour is silent mode.

-L URL, --logger=URL
Remote HTTP logger. Log messages are sent to given URL in a HTTP POST message. URL format is [http://]host[:port][/path/], where host may be a hostname or an IPv4 address.

-a, --automatic
Enable only automatic tests to be executed

-S, --syslog
Enable logging to syslog

-P, --print-step-output
Output standard streams from test steps.

-m, --manual
Enable only manual tests to be executed

-c, --ci
Disable validation of test definition against schema

-s, --semantic
Enable validation of test definition against stricter (semantics) schema

-A, --validate-only
Do only input xml validation, do not execute tests

-H, --no-hwinfo
Do not try to obtain hardware information

-t [USER@]ADDRESS, --target=[USER@]ADDRESS
Enable host-based testing. If given, commands are executed from test control PC (host) side. ADDRESS is the ipv4 adress of the system under test.

=== Filtering options ===

Filtering options allow selecting which tests are to be executed. Filtering options are given as a string. String may contain one or more filter entries. Format of each entry is 'filtername=values' where 'filtername' typically corresponds to an attribute in Test Definition XML and 'values' is a string to which the filter should match. Multiple values can be specified if separated by comma (without extra space). A value containing space must be enclosed in double quotes (""). Current list of forbidden characters in values is as follows: Single quote ('), double quote ("), is-equal-to sign (=), comma (,).

How filtering is carried out: Before processing the filter options, all tests are active by default. Filters can only deactivate tests: If filter value does not match with the value of the corresponding attribute, the test is disabled. Each filter entry is applied for all the active tests in the test package, one after another, in the given order. Filtering is always carried out at the lowest level of the "suite-set-case-step" hierarchy where the corresponding attribute can be defined. Note that some attributes may inherit the value from the upper level. Please refer to Test Definition XML for details.

If 'filtername' is prefixed with dash (-), the filter behaviour is reversed: those tests for which the filter value does match, are disabled.

'filtername' can be any of the following:

* feature
* requirement
* testset
* type
* testcase

The following example does the following: First it disables all test cases except those with attribute type='Integration'. Next, a test case named as Bad Test is disabled. The tests left active will be executed.

testrunner-lite -f tests.xml -o ~/results -l 'type=Integration -testcase="Bad Test"'

The following example executes test cases that specify the requirement attribute with a value containing at least one of the following strings: '50001', '50002', '50003'.

testrunner-lite -f tests.xml -o ~/results -l 'requirement=50001,50002,50003'

Note that each key=values is handled as a separate filter, when checking whether a test case should be filtered. The following example will filter all the test cases,

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1 testset=set2'

whereas the following will accept tests from test sets "set1" and "set2".

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1,set2'

== Manual Test Cases ==

It is also possible to execute manual test cases using testrunner-lite, as defined in Test Definition XML.

In case manual test case is encountered during execution, testrunner-lite will go through the defined test steps and ask user whether the step is passed or failed. The test case will terminate at the first failure, otherwise every step defined will be executed. After the test case is done, user has the option to enter additional comments.

Example output when running manual case:

[INFO] 15:15:53 Starting test case: ExampleTestCase
--- Execute test step ---
Description:
Open calculator. Expected result: calculator opens up.

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Execute test step ---
Description:
Stop calculator

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Test steps executed, case is PASSED ---
Please enter additional comments (ENTER to finish):
Execution was slow.

[INFO] 15:16:41 Finished test case. Result: PASS

== Testrunner-lite and test case verdicts ==

For manual cases the result is naturally set by the user. For '''automatic''' cases a result of a test case is set as follows:
* PASS
** all test steps finished with expected results
* FAIL
** test step has unexpected return value
** test step timed out
** pre-steps on the set the case belongs to have failed
** fail based on measurement data (up-coming feature)
* N/A
** test case has state=Design in the tests.xml
** test case has no steps

If a test case is filtered it doesn't get any result; meaning it is not written to results file at all.

The goal is to align testrunner-lite functionality with what is agreed [[Quality/Glossary#Test_case_verdict|here]].

== About process control ==

Each test step is executed in a separate shell. Testrunner-lite spawns new process for the execution, and waits for the step to finish (or timeout). In case test step contains command that is started to background, the step returns immediately. After test case has finnished a cleanup routine is executed, where testrunner-lite tries to kill all processes that may have been left running by the test steps. Cleanup for pre-steps and post-steps is done after the post steps are executed (i.e. when test set has been executed).

== About host based execution ==

Testrunner-lite supports host based execution, where testrunner-lite is executed on a PC, and the test steps over ssh on hardware. This requires that key based ssh authentication is enabled between device and host.

Remember that each single step in pre_steps, post_steps or inside testcases is executed in a separate SSH session, so you'll have to make sure that if your steps leave some processes running in the background which inherits its pipes straight from the parent shell, the SSH connection will hang until those processes are terminated (or until they close their pipes). Therefore, you should always redirect the stdout, stderr and stdin streams of your background processes, if you don't want your test steps / pre- / post steps to time out in host-based execution.

See http://www.snailbook.com/faq/background-jobs.auto.html for more information.

== Testrunner-lite state machine ==

Below is a simplified picture describing the operation of testrunner-lite. The program flow can be considered to consist of three phases: initializing, executing and clean up. The execution is done set by set, so that the tool needs to maintain only one set at a time in its memory.

[[File:testrunnerlitefsm.png]]

xfig file:
[[File:testrunnerlitesm.fig]]

== Further development ==
We're actively making testrunner-lite a better tool. See discussion about architecture changes and suggestions in following page:
[[Quality/QA-tools/Testrunner-lite/Testrunner-lite_development|Testrunner-lite development]]

Quality/QA-tools/Testrunner-lite

2011-01-18T06:45:42Z

Samposa: /* About test case verdicts */

= Testrunner-lite =

Testrunner-lite is a lightweight tool for test execution, which reads [[Quality/QA-tools/Test plan|Test Plan XML]] files as input, and produces [http://gitorious.org/qa-tools/test-definition/blobs/master/src/data/testdefinition-results.xsd a common-format] test result file which can be published at [http://qa-reports.meego.com/ qa-reports.meego.com]. With testrunner-lite, you can
* Execute automatic, semi-automatic and manual test cases
* Execute test cases locally or in host-based mode
* Use [[Quality/QA-tools/Testrunner-lite#Filtering options|filters]] to select the test cases to be executed
* Validate the used test plan file automatically

See our demo videos at Youtube:
* [http://www.youtube.com/watch?v=TZhHDPUeHIw Demo of version 1.3.17]
* [http://www.youtube.com/watch?v=A0V94xZ_VwI Demo of version 1.3.11]
* [http://www.youtube.com/watch?v=OSF8tyUxI8U Demo of version 1.3.10]
* [http://www.youtube.com/user/meegoqatools MeeGo QA tools Youtube channel]

There is also a graphical user interface version of the tool called [[Quality/QA-tools/Testrunner|Testrunner]] available.

== Installation ==

Testrunner-lite can be installed on MeeGo Netbook UX, Ubuntu 10.04 (and newer) and Fedora 13 by doing the following:
# [[Quality/QA-tools/How to set up repositories|Set up the repository]]
# Install testrunner-lite

MeeGo Netbook 1.1:

sudo zypper install testrunner-lite

Ubuntu:

sudo apt-get install testrunner-lite

Fedora 13:

# yum install testrunner-lite

== Usage ==

The program is executed from command line

/usr/bin/testrunner-lite [options]
Options

-h, --help
Display short help and exit

-V, --version
Display version information and exit

-f FILE, --file FILE
Input file with test definitions in XML (required)

-o FILE, --output FILE
Output file format; FORMAT can be xml or text (Default xml)

-v, -vv, --verbose[={INFO|DEBUG}]
Enable verbosity mode; -v and --verbose=INFO are equivalent outputting INFO, ERROR and WARNING messages. Similarly -vv and --verbose=DEBUG are equivalent, outputting also debug messages. Default behaviour is silent mode.

-L URL, --logger=URL
Remote HTTP logger. Log messages are sent to given URL in a HTTP POST message. URL format is [http://]host[:port][/path/], where host may be a hostname or an IPv4 address.

-a, --automatic
Enable only automatic tests to be executed

-S, --syslog
Enable logging to syslog

-P, --print-step-output
Output standard streams from test steps.

-m, --manual
Enable only manual tests to be executed

-c, --ci
Disable validation of test definition against schema

-s, --semantic
Enable validation of test definition against stricter (semantics) schema

-A, --validate-only
Do only input xml validation, do not execute tests

-H, --no-hwinfo
Do not try to obtain hardware information

-t [USER@]ADDRESS, --target=[USER@]ADDRESS
Enable host-based testing. If given, commands are executed from test control PC (host) side. ADDRESS is the ipv4 adress of the system under test.

=== Filtering options ===

Filtering options allow selecting which tests are to be executed. Filtering options are given as a string. String may contain one or more filter entries. Format of each entry is 'filtername=values' where 'filtername' typically corresponds to an attribute in Test Definition XML and 'values' is a string to which the filter should match. Multiple values can be specified if separated by comma (without extra space). A value containing space must be enclosed in double quotes (""). Current list of forbidden characters in values is as follows: Single quote ('), double quote ("), is-equal-to sign (=), comma (,).

How filtering is carried out: Before processing the filter options, all tests are active by default. Filters can only deactivate tests: If filter value does not match with the value of the corresponding attribute, the test is disabled. Each filter entry is applied for all the active tests in the test package, one after another, in the given order. Filtering is always carried out at the lowest level of the "suite-set-case-step" hierarchy where the corresponding attribute can be defined. Note that some attributes may inherit the value from the upper level. Please refer to Test Definition XML for details.

If 'filtername' is prefixed with dash (-), the filter behaviour is reversed: those tests for which the filter value does match, are disabled.

'filtername' can be any of the following:

* feature
* requirement
* testset
* type
* testcase

The following example does the following: First it disables all test cases except those with attribute type='Integration'. Next, a test case named as Bad Test is disabled. The tests left active will be executed.

testrunner-lite -f tests.xml -o ~/results -l 'type=Integration -testcase="Bad Test"'

The following example executes test cases that specify the requirement attribute with a value containing at least one of the following strings: '50001', '50002', '50003'.

testrunner-lite -f tests.xml -o ~/results -l 'requirement=50001,50002,50003'

Note that each key=values is handled as a separate filter, when checking whether a test case should be filtered. The following example will filter all the test cases,

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1 testset=set2'

whereas the following will accept tests from test sets "set1" and "set2".

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1,set2'

== Manual Test Cases ==

It is also possible to execute manual test cases using testrunner-lite, as defined in Test Definition XML.

In case manual test case is encountered during execution, testrunner-lite will go through the defined test steps and ask user whether the step is passed or failed. The test case will terminate at the first failure, otherwise every step defined will be executed. After the test case is done, user has the option to enter additional comments.

Example output when running manual case:

[INFO] 15:15:53 Starting test case: ExampleTestCase
--- Execute test step ---
Description:
Open calculator. Expected result: calculator opens up.

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Execute test step ---
Description:
Stop calculator

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Test steps executed, case is PASSED ---
Please enter additional comments (ENTER to finish):
Execution was slow.

[INFO] 15:16:41 Finished test case. Result: PASS

== testrunner-lite and test case verdicts ==

For manual cases the result is naturally set by the user. For '''automatic''' cases a result of a test case is set as follows:
* PASS
** all test steps finished with expected results
* FAIL
** test step has unexpected return value
** test step timed out
** pre-steps on the set the case belongs to have failed
** fail based on measurement data (up-coming feature)
* N/A
** test case has state=Design in the tests.xml
** test case has no steps

If a test case is filtered it doesn't get any result; meaning it is not written to results file at all.

The goal is to align testrunner-lite functionality with what is agreed [[Quality/Glossary#Test_case_verdict|here]].

== About process control ==

Each test step is executed in a separate shell. Testrunner-lite spawns new process for the execution, and waits for the step to finish (or timeout). In case test step contains command that is started to background, the step returns immediately. After test case has finnished a cleanup routine is executed, where testrunner-lite tries to kill all processes that may have been left running by the test steps. Cleanup for pre-steps and post-steps is done after the post steps are executed (i.e. when test set has been executed).

== About host based execution ==

Testrunner-lite supports host based execution, where testrunner-lite is executed on a PC, and the test steps over ssh on hardware. This requires that key based ssh authentication is enabled between device and host.

Remember that each single step in pre_steps, post_steps or inside testcases is executed in a separate SSH session, so you'll have to make sure that if your steps leave some processes running in the background which inherits its pipes straight from the parent shell, the SSH connection will hang until those processes are terminated (or until they close their pipes). Therefore, you should always redirect the stdout, stderr and stdin streams of your background processes, if you don't want your test steps / pre- / post steps to time out in host-based execution.

See http://www.snailbook.com/faq/background-jobs.auto.html for more information.

== Testrunner-lite state machine ==

Below is a simplified picture describing the operation of testrunner-lite. The program flow can be considered to consist of three phases: initializing, executing and clean up. The execution is done set by set, so that the tool needs to maintain only one set at a time in its memory.

[[File:testrunnerlitefsm.png]]

xfig file:
[[File:testrunnerlitesm.fig]]

== Further development ==
We're actively making testrunner-lite a better tool. See discussion about architecture changes and suggestions in following page:
[[Quality/QA-tools/Testrunner-lite/Testrunner-lite_development|Testrunner-lite development]]

Quality/QA-tools/Test plan

2011-01-12T12:44:47Z

Samposa: /* Test Cases */

== Test plan overview ==

The basic principle of test plan XML and tool support is that you can use 'any' executable for testing.
Test results are checked from exit codes (automated testing) or prompt (manual testing). Executed test plan XMLs produce XML results - wrapping variety of test methods in consistent format required by test automation and data processing.

The test plan information stored in Test Definition XML files consists of:

1. suite, set, case: Hierarchical structure of the tests.
2. feature, subfeature, requirement: Information about why the tested software has been implemented in the first place (which is why it’s being tested as well).
3. type: Information about the viewpoint of the tests (which quality aspect of the software they are testing, see DevelopmentTestArea in Agile Testing Wiki).
4. level: Information about which test level the tests belong - for test level.
5. domain: Information about which architectural domains the tests are focused on.
6. description: descriptions of the tests (what each test is, and what it is supposed to do).
7. step: Execution instructions (for automated tests), which determine the actual commands to execute to run each test.

''Note:'' that all of the above are not mandatory. Mandatory fields for executing tests are defined with test plan XML validation schema.

== Creating Test plan ==

There are certain mandatory things you’ll need to provide in a test definition XML.

The structure of the XML and the possible attributes/values are defined in the [[Quality/QA-tools/Test-definition|Test Definition XML schema]]. It’s important to validate your XML against the up-to-date schema.

=== Test Cases ===

The main thing in test plan are the test cases, which define what to execute (test steps), what to expect from the execution (expected results) and also some additional information for reporting purposes. An example test case could be:

<case name="my-first-case">
<description>Creating my very first test case</description>
<step>ls</step>
<step>uname -r</step>
</case>

What did we just do? We created a simple test case [[#About_name_attribute|named]] <code>my-first-test-case</code>, which executes two steps and expects that they return the common “success return value” 0. Want to check for some other return value? Use <code>expected_result</code>:

<step expected_result="-1">ls</step>

In case you feel that some test cases are insignificant, i.e. that they shouldn’t be taken into consideration when choosing whether the test run is passed or failed, you can use insignificant-attribute (defaults to <code>false</code>, i.e. every case is considered to be significant):

<case name="not-so-important-test" insignificant="true">

In addition to test execution, the “story” of the test case should also be told, and for telling this story, we have fields for test type (what quality characteristics are tested) and test level (what level of system is being exercised. The possible values for both of these are listed in the appendices. Let’s add level and type to our test case:

<case name="my-first-case" level="Product" type="Functional">
<description>Creating my very first test case</description>
<step expected_result="-1">ls</step>
<step>uname -r</step>
</case>

And that’s our first test case! Not that difficult...

=== Grouping Cases - Sets and Suites ===

Grouping cases to sets and suites makes your life easier and more organized. A good idea is to group test cases that test the same features into same sets, and sets that test the same architectural domain under same suites. E.g.:

<suite name="my-multimedia-tests" domain="Multimedia">
<set name="video-playback-tests" feature="Video Playback">
<description>Video</description>
<case ...
<case ...
</set>
</suite>

So, sets contain test cases, and may also contain description (the same way as the cases do). More about sets in Test Definition Execution part

'''Note:''' There are several attributes that need to be defined on the test case, test set, or test suite level in the test definition XML.<br /> You can find more information about these attributes in the Appendices section of this page.

=== Time outs ===
Test steps will time out by default after 90 seconds. You can change the default time out by adding timeout attribute to your test case. for example

<case name="my-test-case" timeout="120">

Note that this the timeout is for single test step not for the whole case.

For [pre|post]-steps the the default timeout is 180 seconds, it can also be changed:

<pre_steps timeout="600">

=== Putting It All Together ===

Now that you have some knowledge on grouping and test cases in general, it is time to put it all together. Before showing an example, it is important to note that all the mentioned case attributes (e.g. level, type) are inheritable. So, if you have e.g. a set which contains only certain type of cases, you can define the type at the set-level, instead of writing it separately into each case.

Now, the example. Let’s first add the mandatory tags, which each test definition should have:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
</testdefinition>

That’s the mandatory part; we defined that we have a XML-document, and that this particular one is test definition version 1.0. Let’s add more beef to the bones by adding one suite with couple of sets:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
<suite name="my-multimedia-tests" domain="Multimedia">
<description>Testing AF stuff</description>
<set name="video-playback-tests" feature="Video Playback">
<description>Video playback tests</description>
</set>
<set name="video-recording-tests" feature="Video Recording">
<description>Video recording tests</description>
</set>
</suite>
</testdefinition>

Okay, we have one suite named <code>my-multimedia-tests with two sets</code>, testing video playback and recording features. Cases are still missing:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
<suite name="my-multimedia-tests" domain="Multimedia">
<description>Video playback tests</description>
<set name="video-playback-tests" feature="Video Playback">
<description>Video playback tests</description>
<case name="playback1" type="Functional" level="Component">
<step>execute_playback_test</step>
</case>
</set>
<set name="video-recording-tests" feature="Video Recording">
<description>Video playback tests</description>
<case ...
</set>
</suite>
</testdefinition>

And that’s the basic story. Next chapters cover more details about the definition, but you should now have the general understanding of the subject!

== Controlling Environment for Execution ==

=== Setup and Teardown ===

In case you want to do some setup and cleaning before and after the cases are executed, sets may have pre- and post-steps in them:

<set ...>
<pre_steps>
<step>do_some_setup</step>
</pre_steps>
<case ...
<post_steps>
<step>clean_up</step>
</post_steps>
</set>

If you want to ensure that pre-step is executed properly before starting the real testing, you can use expected result also in those:

<pre_steps>
<step expected_result="1">do_some_setup_that_may_fail</step>
...

:'''Warning''' When using expected result in this context, the process return value will be waited for, thus you cannot use this with daemon or background processes since they basically never return and cause execution to jam. Another thing to note is that the steps are executed in separate shells, so it is not possible to e.g. set environment variables or change directories in pre-steps as those will be lost.

=== Filtering Based on Hardware Identifier ===

If different tests sets for different hardware are required then hwiddetect feature can be utilised. User can define a command used to get a hardware identifier within hwiddetect tag. The hardware identifier returned by the command is matched with optional hwid attribute of a test set. If not equal, test cases in the set are skipped and are not written to the result file. A test set will never be skipped if hwid attribute has not been defined for it. You can also define multiple hwid values separated by comma for a set.

Command defined by hwiddetect can be a shell command or a separate executable. The executable should be included in the test package. Testrunner-lite removes extra whitespace and linefeeds from the the output of the hwiddetect command so that test developer does not need to care about it.

Example usage of hwiddetect:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
<hwiddetect>/usr/bin/getmyhwid</hwiddetect>
<suite name="suite1">
<set name="test_feature_X_on_hw_bar" hwid="bar">
<case name="test_X_1">
<step>echo "hwid is bar"</step>
</case>
</set>
<set name="test_feature_X_on_hw_foo" hwid="foo">
<case name="test_X_1">
<step>echo "hwid is foo"</step>
</case>
</set>
<set name="test_feature_X_on_hw_foo_or_bar" hwid="foo,bar">
<case name="test_X_1">
<step>echo "hwid is foo or bar"</step>
</case>
</set>
</suite>
</testdefinition>

=== Fetching Additional Files ===

In addition to normal result file, you can also fetch what ever files you need with get-tag:

<set ...>
...
<get>
<file>/tmp/myadditionalresult.1</file>
<file>/tmp/myadditionalresult.2</file>
</get>
</set>

== Manual, Automatic and Semi-automatic Test Cases ==

If not specified by the "manual" attribute all cases are automatic. The value for the attribute is inherited from higher entity (set->case->step). By semi-automatic test case we mean a manual case that has some automatic parts; the idea being that only those steps that have to be manual, are. Note that this does not work the other way around, we cannot have automatic test case with manual steps (would be semantically weird thus our tools do not support it).

The example below tries to clarify the above. The "example_set" shows a manual, automatic and semi-automatic case. The example works with testrunner-lite/testrunner-ui, give it a try: [[File:example_definition.xml]].

<?xml version="1.0" encoding="UTF-8"?>

<testdefinition version="1.0">
<suite name="example_suite">
<set name="example_set">
<description>Example test set with manual, automatic and semi-automatic case.</description>
<case manual="true" name="manual_case">
<description>Manual test case with three steps inside one step tag.</description>
<step>Step 1: execute command ttt on shell.
Step 2: write something into edit box.
Step 3: press ok button.
Expected: Text should be updated into label.
</step>
</case>
<case timeout="96" name="automatic_case">
<description>Automatic test case that executes some shell commands.</description>
<step>ls /tmp</step>
<step expected_result="2">ls /nosuchfile</step>
<step>pwd</step>
</case>
<case manual="true" name="semi_automatic_case">
<description>A case with two automatic and two manual steps.</description>
<step manual="false">xcalc &amp;</step>
<step>Step: Type in 2 + 2 =. Expected: 4 is displayed </step>
<step>Press x² button. Expected: 16 is displayed.</step>
<step manual="false">killall xcalc</step>
</case>
</set>
</suite>
</testdefinition>
(A real test developer / tester might come up with nicer examples, any input welcome).

== Test Plan Execution ==

Test plans are run with test execution tools such as [[Quality/QA-tools/Testrunner|Testrunner]] and [[Quality/QA-tools/Testrunner-lite|testrunner-lite]], which read the plan and produce a result XML file.

== About name attribute ==

Note that the name attribute is of type [http://www.ietf.org/rfc/rfc2396.txt anyURI]. Do not use the following characters in suite,set or case names.
*'''''; / ? : @ & = + $ ,''''' (reserved)
or
* '''''{ } | \ ^ [ ] `''''' (unwise)

Also the use of '''''<space>''''' is not recommended, since some validators accept if for anyURI, some don't. Use '''''_''''' or '''''-''''' instead.

Quality/QA-tools/Test plan

2011-01-12T12:31:36Z

Samposa: /* Test Plan Execution */

== Test plan overview ==

The basic principle of test plan XML and tool support is that you can use 'any' executable for testing.
Test results are checked from exit codes (automated testing) or prompt (manual testing). Executed test plan XMLs produce XML results - wrapping variety of test methods in consistent format required by test automation and data processing.

The test plan information stored in Test Definition XML files consists of:

1. suite, set, case: Hierarchical structure of the tests.
2. feature, subfeature, requirement: Information about why the tested software has been implemented in the first place (which is why it’s being tested as well).
3. type: Information about the viewpoint of the tests (which quality aspect of the software they are testing, see DevelopmentTestArea in Agile Testing Wiki).
4. level: Information about which test level the tests belong - for test level.
5. domain: Information about which architectural domains the tests are focused on.
6. description: descriptions of the tests (what each test is, and what it is supposed to do).
7. step: Execution instructions (for automated tests), which determine the actual commands to execute to run each test.

''Note:'' that all of the above are not mandatory. Mandatory fields for executing tests are defined with test plan XML validation schema.

== Creating Test plan ==

There are certain mandatory things you’ll need to provide in a test definition XML.

The structure of the XML and the possible attributes/values are defined in the [[Quality/QA-tools/Test-definition|Test Definition XML schema]]. It’s important to validate your XML against the up-to-date schema.

=== Test Cases ===

The main thing in test plan are the test cases, which define what to execute (test steps), what to expect from the execution (expected results) and also some additional information for reporting purposes. An example test case could be:

<case name="my-first-case">
<description>Creating my very first test case</description>
<step>ls</step>
<step>uname -r</step>
</case>

What did we just do? We created a simple test case named <code>my-first-test-case</code>, which executes two steps and expects that they return the common “success return value” 0. Want to check for some other return value? Use <code>expected_result</code>:

<step expected_result="-1">ls</step>

In case you feel that some test cases are insignificant, i.e. that they shouldn’t be taken into consideration when choosing whether the test run is passed or failed, you can use insignificant-attribute (defaults to <code>false</code>, i.e. every case is considered to be significant):

<case name="not-so-important-test" insignificant="true">

In addition to test execution, the “story” of the test case should also be told, and for telling this story, we have fields for test type (what quality characteristics are tested) and test level (what level of system is being exercised. The possible values for both of these are listed in the appendices. Let’s add level and type to our test case:

<case name="my-first-case" level="Product" type="Functional">
<description>Creating my very first test case</description>
<step expected_result="-1">ls</step>
<step>uname -r</step>
</case>

And that’s our first test case! Not that difficult...

=== Grouping Cases - Sets and Suites ===

Grouping cases to sets and suites makes your life easier and more organized. A good idea is to group test cases that test the same features into same sets, and sets that test the same architectural domain under same suites. E.g.:

<suite name="my-multimedia-tests" domain="Multimedia">
<set name="video-playback-tests" feature="Video Playback">
<description>Video</description>
<case ...
<case ...
</set>
</suite>

So, sets contain test cases, and may also contain description (the same way as the cases do). More about sets in Test Definition Execution part

'''Note:''' There are several attributes that need to be defined on the test case, test set, or test suite level in the test definition XML.<br /> You can find more information about these attributes in the Appendices section of this page.

=== Time outs ===
Test steps will time out by default after 90 seconds. You can change the default time out by adding timeout attribute to your test case. for example

<case name="my-test-case" timeout="120">

Note that this the timeout is for single test step not for the whole case.

For [pre|post]-steps the the default timeout is 180 seconds, it can also be changed:

<pre_steps timeout="600">

=== Putting It All Together ===

Now that you have some knowledge on grouping and test cases in general, it is time to put it all together. Before showing an example, it is important to note that all the mentioned case attributes (e.g. level, type) are inheritable. So, if you have e.g. a set which contains only certain type of cases, you can define the type at the set-level, instead of writing it separately into each case.

Now, the example. Let’s first add the mandatory tags, which each test definition should have:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
</testdefinition>

That’s the mandatory part; we defined that we have a XML-document, and that this particular one is test definition version 1.0. Let’s add more beef to the bones by adding one suite with couple of sets:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
<suite name="my-multimedia-tests" domain="Multimedia">
<description>Testing AF stuff</description>
<set name="video-playback-tests" feature="Video Playback">
<description>Video playback tests</description>
</set>
<set name="video-recording-tests" feature="Video Recording">
<description>Video recording tests</description>
</set>
</suite>
</testdefinition>

Okay, we have one suite named <code>my-multimedia-tests with two sets</code>, testing video playback and recording features. Cases are still missing:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
<suite name="my-multimedia-tests" domain="Multimedia">
<description>Video playback tests</description>
<set name="video-playback-tests" feature="Video Playback">
<description>Video playback tests</description>
<case name="playback1" type="Functional" level="Component">
<step>execute_playback_test</step>
</case>
</set>
<set name="video-recording-tests" feature="Video Recording">
<description>Video playback tests</description>
<case ...
</set>
</suite>
</testdefinition>

And that’s the basic story. Next chapters cover more details about the definition, but you should now have the general understanding of the subject!

== Controlling Environment for Execution ==

=== Setup and Teardown ===

In case you want to do some setup and cleaning before and after the cases are executed, sets may have pre- and post-steps in them:

<set ...>
<pre_steps>
<step>do_some_setup</step>
</pre_steps>
<case ...
<post_steps>
<step>clean_up</step>
</post_steps>
</set>

If you want to ensure that pre-step is executed properly before starting the real testing, you can use expected result also in those:

<pre_steps>
<step expected_result="1">do_some_setup_that_may_fail</step>
...

:'''Warning''' When using expected result in this context, the process return value will be waited for, thus you cannot use this with daemon or background processes since they basically never return and cause execution to jam. Another thing to note is that the steps are executed in separate shells, so it is not possible to e.g. set environment variables or change directories in pre-steps as those will be lost.

=== Filtering Based on Hardware Identifier ===

If different tests sets for different hardware are required then hwiddetect feature can be utilised. User can define a command used to get a hardware identifier within hwiddetect tag. The hardware identifier returned by the command is matched with optional hwid attribute of a test set. If not equal, test cases in the set are skipped and are not written to the result file. A test set will never be skipped if hwid attribute has not been defined for it. You can also define multiple hwid values separated by comma for a set.

Command defined by hwiddetect can be a shell command or a separate executable. The executable should be included in the test package. Testrunner-lite removes extra whitespace and linefeeds from the the output of the hwiddetect command so that test developer does not need to care about it.

Example usage of hwiddetect:

<?xml version="1.0" encoding="UTF-8"?>
<testdefinition version="1.0">
<hwiddetect>/usr/bin/getmyhwid</hwiddetect>
<suite name="suite1">
<set name="test_feature_X_on_hw_bar" hwid="bar">
<case name="test_X_1">
<step>echo "hwid is bar"</step>
</case>
</set>
<set name="test_feature_X_on_hw_foo" hwid="foo">
<case name="test_X_1">
<step>echo "hwid is foo"</step>
</case>
</set>
<set name="test_feature_X_on_hw_foo_or_bar" hwid="foo,bar">
<case name="test_X_1">
<step>echo "hwid is foo or bar"</step>
</case>
</set>
</suite>
</testdefinition>

=== Fetching Additional Files ===

In addition to normal result file, you can also fetch what ever files you need with get-tag:

<set ...>
...
<get>
<file>/tmp/myadditionalresult.1</file>
<file>/tmp/myadditionalresult.2</file>
</get>
</set>

== Manual, Automatic and Semi-automatic Test Cases ==

If not specified by the "manual" attribute all cases are automatic. The value for the attribute is inherited from higher entity (set->case->step). By semi-automatic test case we mean a manual case that has some automatic parts; the idea being that only those steps that have to be manual, are. Note that this does not work the other way around, we cannot have automatic test case with manual steps (would be semantically weird thus our tools do not support it).

The example below tries to clarify the above. The "example_set" shows a manual, automatic and semi-automatic case. The example works with testrunner-lite/testrunner-ui, give it a try: [[File:example_definition.xml]].

<?xml version="1.0" encoding="UTF-8"?>

<testdefinition version="1.0">
<suite name="example_suite">
<set name="example_set">
<description>Example test set with manual, automatic and semi-automatic case.</description>
<case manual="true" name="manual_case">
<description>Manual test case with three steps inside one step tag.</description>
<step>Step 1: execute command ttt on shell.
Step 2: write something into edit box.
Step 3: press ok button.
Expected: Text should be updated into label.
</step>
</case>
<case timeout="96" name="automatic_case">
<description>Automatic test case that executes some shell commands.</description>
<step>ls /tmp</step>
<step expected_result="2">ls /nosuchfile</step>
<step>pwd</step>
</case>
<case manual="true" name="semi_automatic_case">
<description>A case with two automatic and two manual steps.</description>
<step manual="false">xcalc &amp;</step>
<step>Step: Type in 2 + 2 =. Expected: 4 is displayed </step>
<step>Press x² button. Expected: 16 is displayed.</step>
<step manual="false">killall xcalc</step>
</case>
</set>
</suite>
</testdefinition>
(A real test developer / tester might come up with nicer examples, any input welcome).

== Test Plan Execution ==

Test plans are run with test execution tools such as [[Quality/QA-tools/Testrunner|Testrunner]] and [[Quality/QA-tools/Testrunner-lite|testrunner-lite]], which read the plan and produce a result XML file.

== About name attribute ==

Note that the name attribute is of type [http://www.ietf.org/rfc/rfc2396.txt anyURI]. Do not use the following characters in suite,set or case names.
*'''''; / ? : @ & = + $ ,''''' (reserved)
or
* '''''{ } | \ ^ [ ] `''''' (unwise)

Also the use of '''''<space>''''' is not recommended, since some validators accept if for anyURI, some don't. Use '''''_''''' or '''''-''''' instead.

Quality/QA-tools/Testrunner-lite

2011-01-07T19:07:43Z

Samposa: /* About test case verdicts */

= Testrunner-lite =

Testrunner-lite is a lightweight tool for test execution, which reads [[Quality/QA-tools/Test plan|Test Plan XML]] files as input, and produces [http://gitorious.org/qa-tools/test-definition/blobs/master/src/data/testdefinition-results.xsd a common-format] test result file which can be published at [http://qa-reports.meego.com/ qa-reports.meego.com]. With testrunner-lite, you can
* Execute automatic, semi-automatic and manual test cases
* Execute test cases locally or in host-based mode
* Use [[Quality/QA-tools/Testrunner-lite#Filtering options|filters]] to select the test cases to be executed
* Validate the used test plan file automatically

See our demo videos at Youtube:
* [http://www.youtube.com/watch?v=TZhHDPUeHIw Demo of version 1.3.17]
* [http://www.youtube.com/watch?v=A0V94xZ_VwI Demo of version 1.3.11]
* [http://www.youtube.com/watch?v=OSF8tyUxI8U Demo of version 1.3.10]
* [http://www.youtube.com/user/meegoqatools MeeGo QA tools Youtube channel]

There is also a graphical user interface version of the tool called [[Quality/QA-tools/Testrunner|Testrunner]] available.

== Installation ==

Testrunner-lite can be installed on MeeGo Netbook UX, Ubuntu 10.04 (and newer) and Fedora 13. First, [[Quality/QA-tools/How to set up repositories|set up the repository]]. Then, install testrunner-lite:

MeeGo Netbook 1.1:

sudo zypper install testrunner-lite

Ubuntu:

sudo apt-get install testrunner-lite

Fedora 13:

# yum install testrunner-lite

== Usage ==

The program is executed from command line

/usr/bin/testrunner-lite [options]
Options

-h, --help
Display short help and exit

-V, --version
Display version information and exit

-f FILE, --file FILE
Input file with test definitions in XML (required)

-o FILE, --output FILE
Output file format; FORMAT can be xml or text (Default xml)

-v, -vv, --verbose[={INFO|DEBUG}]
Enable verbosity mode; -v and --verbose=INFO are equivalent outputting INFO, ERROR and WARNING messages. Similarly -vv and --verbose=DEBUG are equivalent, outputting also debug messages. Default behaviour is silent mode.

-L URL, --logger=URL
Remote HTTP logger. Log messages are sent to given URL in a HTTP POST message. URL format is [http://]host[:port][/path/], where host may be a hostname or an IPv4 address.

-a, --automatic
Enable only automatic tests to be executed

-S, --syslog
Enable logging to syslog

-P, --print-step-output
Output standard streams from test steps.

-m, --manual
Enable only manual tests to be executed

-c, --ci
Disable validation of test definition against schema

-s, --semantic
Enable validation of test definition against stricter (semantics) schema

-A, --validate-only
Do only input xml validation, do not execute tests

-H, --no-hwinfo
Do not try to obtain hardware information

-t [USER@]ADDRESS, --target=[USER@]ADDRESS
Enable host-based testing. If given, commands are executed from test control PC (host) side. ADDRESS is the ipv4 adress of the system under test.

=== Filtering options ===

Filtering options allow selecting which tests are to be executed. Filtering options are given as a string. String may contain one or more filter entries. Format of each entry is 'filtername=values' where 'filtername' typically corresponds to an attribute in Test Definition XML and 'values' is a string to which the filter should match. Multiple values can be specified if separated by comma (without extra space). A value containing space must be enclosed in double quotes (""). Current list of forbidden characters in values is as follows: Single quote ('), double quote ("), is-equal-to sign (=), comma (,).

How filtering is carried out: Before processing the filter options, all tests are active by default. Filters can only deactivate tests: If filter value does not match with the value of the corresponding attribute, the test is disabled. Each filter entry is applied for all the active tests in the test package, one after another, in the given order. Filtering is always carried out at the lowest level of the "suite-set-case-step" hierarchy where the corresponding attribute can be defined. Note that some attributes may inherit the value from the upper level. Please refer to Test Definition XML for details.

If 'filtername' is prefixed with dash (-), the filter behaviour is reversed: those tests for which the filter value does match, are disabled.

'filtername' can be any of the following:

* feature
* requirement
* testset
* type
* testcase

The following example does the following: First it disables all test cases except those with attribute type='Integration'. Next, a test case named as Bad Test is disabled. The tests left active will be executed.

testrunner-lite -f tests.xml -o ~/results -l 'type=Integration -testcase="Bad Test"'

The following example executes test cases that specify the requirement attribute with a value containing at least one of the following strings: '50001', '50002', '50003'.

testrunner-lite -f tests.xml -o ~/results -l 'requirement=50001,50002,50003'

Note that each key=values is handled as a separate filter, when checking whether a test case should be filtered. The following example will filter all the test cases,

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1 testset=set2'

whereas the following will accept tests from test sets "set1" and "set2".

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1,set2'

== Manual Test Cases ==

It is also possible to execute manual test cases using testrunner-lite, as defined in Test Definition XML.

In case manual test case is encountered during execution, testrunner-lite will go through the defined test steps and ask user whether the step is passed or failed. The test case will terminate at the first failure, otherwise every step defined will be executed. After the test case is done, user has the option to enter additional comments.

Example output when running manual case:

[INFO] 15:15:53 Starting test case: ExampleTestCase
--- Execute test step ---
Description:
Open calculator. Expected result: calculator opens up.

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Execute test step ---
Description:
Stop calculator

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Test steps executed, case is PASSED ---
Please enter additional comments (ENTER to finish):
Execution was slow.

[INFO] 15:16:41 Finished test case. Result: PASS

== About test case verdicts ==

For manual cases the result is naturally set by the user. For automatic cases a result of a test case is set as follows:
* PASS
** all test steps finished with expected results
* FAIL
** test step has unexpected return value
** test step timed out
** pre-steps on the set the case belongs to have failed
** fail based on measurement data (up-coming feature)
* N/A
** test case has state=Design in the tests.xml
** test case has no steps

If a test case is filtered it doesn't get any result; meaning it is not written to results file at all.

== About process control ==

Each test step is executed in a separate shell. Testrunner-lite spawns new process for the execution, and waits for the step to finish (or timeout). In case test step contains command that is started to background, the step returns immediately. After test case has finnished a cleanup routine is executed, where testrunner-lite tries to kill all processes that may have been left running by the test steps. Cleanup for pre-steps and post-steps is done after the post steps are executed (i.e. when test set has been executed).

== About host based execution ==

Testrunner-lite supports host based execution, where testrunner-lite is executed on a PC, and the test steps over ssh on hardware. This requires that key based ssh authentication is enabled between device and host.

Remember that each single step in pre_steps, post_steps or inside testcases is executed in a separate SSH session, so you'll have to make sure that if your steps leave some processes running in the background which inherits its pipes straight from the parent shell, the SSH connection will hang until those processes are terminated (or until they close their pipes). Therefore, you should always redirect the stdout, stderr and stdin streams of your background processes, if you don't want your test steps / pre- / post steps to time out in host-based execution.

See http://www.snailbook.com/faq/background-jobs.auto.html for more information.

== Testrunner-lite state machine ==

Below is a simplified picture describing the operation of testrunner-lite. The program flow can be considered to consist of three phases: initializing, executing and clean up. The execution is done set by set, so that the tool needs to maintain only one set at a time in its memory.

[[File:testrunnerlitefsm.png]]

xfig file:
[[File:testrunnerlitesm.fig]]

== Further development ==
We're actively making testrunner-lite a better tool. See discussion about architecture changes and suggestions in following page:
[[Quality/QA-tools/Testrunner-lite/Testrunner-lite_development|Testrunner-lite development]]

Quality/QA-tools/Testrunner-lite

2011-01-07T15:17:45Z

Samposa: /* About test case verdicts */

= Testrunner-lite =

Testrunner-lite is a lightweight tool for test execution, which reads [[Quality/QA-tools/Test plan|Test Plan XML]] files as input, and produces [http://gitorious.org/qa-tools/test-definition/blobs/master/src/data/testdefinition-results.xsd a common-format] test result file which can be published at [http://qa-reports.meego.com/ qa-reports.meego.com]. With testrunner-lite, you can
* Execute automatic, semi-automatic and manual test cases
* Execute test cases locally or in host-based mode
* Use [[Quality/QA-tools/Testrunner-lite#Filtering options|filters]] to select the test cases to be executed
* Validate the used test plan file automatically

See our demo videos at Youtube:
* [http://www.youtube.com/watch?v=TZhHDPUeHIw Demo of version 1.3.17]
* [http://www.youtube.com/watch?v=A0V94xZ_VwI Demo of version 1.3.11]
* [http://www.youtube.com/watch?v=OSF8tyUxI8U Demo of version 1.3.10]
* [http://www.youtube.com/user/meegoqatools MeeGo QA tools Youtube channel]

There is also a graphical user interface version of the tool called [[Quality/QA-tools/Testrunner|Testrunner]] available.

== Installation ==

Testrunner-lite can be installed on MeeGo Netbook UX, Ubuntu 10.04 (and newer) and Fedora 13. First, [[Quality/QA-tools/How to set up repositories|set up the repository]]. Then, install testrunner-lite:

MeeGo Netbook 1.1:

sudo zypper install testrunner-lite

Ubuntu:

sudo apt-get install testrunner-lite

Fedora 13:

# yum install testrunner-lite

== Usage ==

The program is executed from command line

/usr/bin/testrunner-lite [options]
Options

-h, --help
Display short help and exit

-V, --version
Display version information and exit

-f FILE, --file FILE
Input file with test definitions in XML (required)

-o FILE, --output FILE
Output file format; FORMAT can be xml or text (Default xml)

-v, -vv, --verbose[={INFO|DEBUG}]
Enable verbosity mode; -v and --verbose=INFO are equivalent outputting INFO, ERROR and WARNING messages. Similarly -vv and --verbose=DEBUG are equivalent, outputting also debug messages. Default behaviour is silent mode.

-L URL, --logger=URL
Remote HTTP logger. Log messages are sent to given URL in a HTTP POST message. URL format is [http://]host[:port][/path/], where host may be a hostname or an IPv4 address.

-a, --automatic
Enable only automatic tests to be executed

-S, --syslog
Enable logging to syslog

-P, --print-step-output
Output standard streams from test steps.

-m, --manual
Enable only manual tests to be executed

-c, --ci
Disable validation of test definition against schema

-s, --semantic
Enable validation of test definition against stricter (semantics) schema

-A, --validate-only
Do only input xml validation, do not execute tests

-H, --no-hwinfo
Do not try to obtain hardware information

-t [USER@]ADDRESS, --target=[USER@]ADDRESS
Enable host-based testing. If given, commands are executed from test control PC (host) side. ADDRESS is the ipv4 adress of the system under test.

=== Filtering options ===

Filtering options allow selecting which tests are to be executed. Filtering options are given as a string. String may contain one or more filter entries. Format of each entry is 'filtername=values' where 'filtername' typically corresponds to an attribute in Test Definition XML and 'values' is a string to which the filter should match. Multiple values can be specified if separated by comma (without extra space). A value containing space must be enclosed in double quotes (""). Current list of forbidden characters in values is as follows: Single quote ('), double quote ("), is-equal-to sign (=), comma (,).

How filtering is carried out: Before processing the filter options, all tests are active by default. Filters can only deactivate tests: If filter value does not match with the value of the corresponding attribute, the test is disabled. Each filter entry is applied for all the active tests in the test package, one after another, in the given order. Filtering is always carried out at the lowest level of the "suite-set-case-step" hierarchy where the corresponding attribute can be defined. Note that some attributes may inherit the value from the upper level. Please refer to Test Definition XML for details.

If 'filtername' is prefixed with dash (-), the filter behaviour is reversed: those tests for which the filter value does match, are disabled.

'filtername' can be any of the following:

* feature
* requirement
* testset
* type
* testcase

The following example does the following: First it disables all test cases except those with attribute type='Integration'. Next, a test case named as Bad Test is disabled. The tests left active will be executed.

testrunner-lite -f tests.xml -o ~/results -l 'type=Integration -testcase="Bad Test"'

The following example executes test cases that specify the requirement attribute with a value containing at least one of the following strings: '50001', '50002', '50003'.

testrunner-lite -f tests.xml -o ~/results -l 'requirement=50001,50002,50003'

Note that each key=values is handled as a separate filter, when checking whether a test case should be filtered. The following example will filter all the test cases,

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1 testset=set2'

whereas the following will accept tests from test sets "set1" and "set2".

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1,set2'

== Manual Test Cases ==

It is also possible to execute manual test cases using testrunner-lite, as defined in Test Definition XML.

In case manual test case is encountered during execution, testrunner-lite will go through the defined test steps and ask user whether the step is passed or failed. The test case will terminate at the first failure, otherwise every step defined will be executed. After the test case is done, user has the option to enter additional comments.

Example output when running manual case:

[INFO] 15:15:53 Starting test case: ExampleTestCase
--- Execute test step ---
Description:
Open calculator. Expected result: calculator opens up.

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Execute test step ---
Description:
Stop calculator

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Test steps executed, case is PASSED ---
Please enter additional comments (ENTER to finish):
Execution was slow.

[INFO] 15:16:41 Finished test case. Result: PASS

== About test case verdicts ==

For manual cases the result is naturally set by the user. For automatic cases a result of a test case is set as follows:
* PASS
** all test steps finished with expected results
* FAIL
** test step has unexpected return value
** test step timed out
** fail based on measurement data (up-coming feature)
* N/A
** test case has state=Design in the tests.xml
** test case has no steps

If a test case is filtered it doesn't get any result; meaning it is not written to results file at all.

== About process control ==

Each test step is executed in a separate shell. Testrunner-lite spawns new process for the execution, and waits for the step to finish (or timeout). In case test step contains command that is started to background, the step returns immediately. After test case has finnished a cleanup routine is executed, where testrunner-lite tries to kill all processes that may have been left running by the test steps. Cleanup for pre-steps and post-steps is done after the post steps are executed (i.e. when test set has been executed).

== About host based execution ==

Testrunner-lite supports host based execution, where testrunner-lite is executed on a PC, and the test steps over ssh on hardware. This requires that key based ssh authentication is enabled between device and host.

Remember that each single step in pre_steps, post_steps or inside testcases is executed in a separate SSH session, so you'll have to make sure that if your steps leave some processes running in the background which inherits its pipes straight from the parent shell, the SSH connection will hang until those processes are terminated (or until they close their pipes). Therefore, you should always redirect the stdout, stderr and stdin streams of your background processes, if you don't want your test steps / pre- / post steps to time out in host-based execution.

See http://www.snailbook.com/faq/background-jobs.auto.html for more information.

== Testrunner-lite state machine ==

Below is a simplified picture describing the operation of testrunner-lite. The program flow can be considered to consist of three phases: initializing, executing and clean up. The execution is done set by set, so that the tool needs to maintain only one set at a time in its memory.

[[File:testrunnerlitefsm.png]]

xfig file:
[[File:testrunnerlitesm.fig]]

== Further development ==
We're actively making testrunner-lite a better tool. See discussion about architecture changes and suggestions in following page:
[[Quality/QA-tools/Testrunner-lite/Testrunner-lite_development|Testrunner-lite development]]

Quality/QA-tools/Testrunner-lite

2011-01-07T15:11:11Z

Samposa: /* About test case verdicts */

= Testrunner-lite =

Testrunner-lite is a lightweight tool for test execution, which reads [[Quality/QA-tools/Test plan|Test Plan XML]] files as input, and produces [http://gitorious.org/qa-tools/test-definition/blobs/master/src/data/testdefinition-results.xsd a common-format] test result file which can be published at [http://qa-reports.meego.com/ qa-reports.meego.com]. With testrunner-lite, you can
* Execute automatic, semi-automatic and manual test cases
* Execute test cases locally or in host-based mode
* Use [[Quality/QA-tools/Testrunner-lite#Filtering options|filters]] to select the test cases to be executed
* Validate the used test plan file automatically

See our demo videos at Youtube:
* [http://www.youtube.com/watch?v=TZhHDPUeHIw Demo of version 1.3.17]
* [http://www.youtube.com/watch?v=A0V94xZ_VwI Demo of version 1.3.11]
* [http://www.youtube.com/watch?v=OSF8tyUxI8U Demo of version 1.3.10]
* [http://www.youtube.com/user/meegoqatools MeeGo QA tools Youtube channel]

There is also a graphical user interface version of the tool called [[Quality/QA-tools/Testrunner|Testrunner]] available.

== Installation ==

Testrunner-lite can be installed on MeeGo Netbook UX, Ubuntu 10.04 (and newer) and Fedora 13. First, [[Quality/QA-tools/How to set up repositories|set up the repository]]. Then, install testrunner-lite:

MeeGo Netbook 1.1:

sudo zypper install testrunner-lite

Ubuntu:

sudo apt-get install testrunner-lite

Fedora 13:

# yum install testrunner-lite

== Usage ==

The program is executed from command line

/usr/bin/testrunner-lite [options]
Options

-h, --help
Display short help and exit

-V, --version
Display version information and exit

-f FILE, --file FILE
Input file with test definitions in XML (required)

-o FILE, --output FILE
Output file format; FORMAT can be xml or text (Default xml)

-v, -vv, --verbose[={INFO|DEBUG}]
Enable verbosity mode; -v and --verbose=INFO are equivalent outputting INFO, ERROR and WARNING messages. Similarly -vv and --verbose=DEBUG are equivalent, outputting also debug messages. Default behaviour is silent mode.

-L URL, --logger=URL
Remote HTTP logger. Log messages are sent to given URL in a HTTP POST message. URL format is [http://]host[:port][/path/], where host may be a hostname or an IPv4 address.

-a, --automatic
Enable only automatic tests to be executed

-S, --syslog
Enable logging to syslog

-P, --print-step-output
Output standard streams from test steps.

-m, --manual
Enable only manual tests to be executed

-c, --ci
Disable validation of test definition against schema

-s, --semantic
Enable validation of test definition against stricter (semantics) schema

-A, --validate-only
Do only input xml validation, do not execute tests

-H, --no-hwinfo
Do not try to obtain hardware information

-t [USER@]ADDRESS, --target=[USER@]ADDRESS
Enable host-based testing. If given, commands are executed from test control PC (host) side. ADDRESS is the ipv4 adress of the system under test.

=== Filtering options ===

Filtering options allow selecting which tests are to be executed. Filtering options are given as a string. String may contain one or more filter entries. Format of each entry is 'filtername=values' where 'filtername' typically corresponds to an attribute in Test Definition XML and 'values' is a string to which the filter should match. Multiple values can be specified if separated by comma (without extra space). A value containing space must be enclosed in double quotes (""). Current list of forbidden characters in values is as follows: Single quote ('), double quote ("), is-equal-to sign (=), comma (,).

How filtering is carried out: Before processing the filter options, all tests are active by default. Filters can only deactivate tests: If filter value does not match with the value of the corresponding attribute, the test is disabled. Each filter entry is applied for all the active tests in the test package, one after another, in the given order. Filtering is always carried out at the lowest level of the "suite-set-case-step" hierarchy where the corresponding attribute can be defined. Note that some attributes may inherit the value from the upper level. Please refer to Test Definition XML for details.

If 'filtername' is prefixed with dash (-), the filter behaviour is reversed: those tests for which the filter value does match, are disabled.

'filtername' can be any of the following:

* feature
* requirement
* testset
* type
* testcase

The following example does the following: First it disables all test cases except those with attribute type='Integration'. Next, a test case named as Bad Test is disabled. The tests left active will be executed.

testrunner-lite -f tests.xml -o ~/results -l 'type=Integration -testcase="Bad Test"'

The following example executes test cases that specify the requirement attribute with a value containing at least one of the following strings: '50001', '50002', '50003'.

testrunner-lite -f tests.xml -o ~/results -l 'requirement=50001,50002,50003'

Note that each key=values is handled as a separate filter, when checking whether a test case should be filtered. The following example will filter all the test cases,

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1 testset=set2'

whereas the following will accept tests from test sets "set1" and "set2".

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1,set2'

== Manual Test Cases ==

It is also possible to execute manual test cases using testrunner-lite, as defined in Test Definition XML.

In case manual test case is encountered during execution, testrunner-lite will go through the defined test steps and ask user whether the step is passed or failed. The test case will terminate at the first failure, otherwise every step defined will be executed. After the test case is done, user has the option to enter additional comments.

Example output when running manual case:

[INFO] 15:15:53 Starting test case: ExampleTestCase
--- Execute test step ---
Description:
Open calculator. Expected result: calculator opens up.

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Execute test step ---
Description:
Stop calculator

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Test steps executed, case is PASSED ---
Please enter additional comments (ENTER to finish):
Execution was slow.

[INFO] 15:16:41 Finished test case. Result: PASS

== About test case verdicts ==

For manual cases the result is naturally set by the user. For automatic cases a result of a test case is set as follows:
* PASS
** all test steps finished with expected results
* FAIL
** test step has unexpected return value
** test step timed out
** fail based on measurement data (up-coming feature)
* N/A
** test case has state=Design in the tests.xml
** test case has no steps

== About process control ==

Each test step is executed in a separate shell. Testrunner-lite spawns new process for the execution, and waits for the step to finish (or timeout). In case test step contains command that is started to background, the step returns immediately. After test case has finnished a cleanup routine is executed, where testrunner-lite tries to kill all processes that may have been left running by the test steps. Cleanup for pre-steps and post-steps is done after the post steps are executed (i.e. when test set has been executed).

== About host based execution ==

Testrunner-lite supports host based execution, where testrunner-lite is executed on a PC, and the test steps over ssh on hardware. This requires that key based ssh authentication is enabled between device and host.

Remember that each single step in pre_steps, post_steps or inside testcases is executed in a separate SSH session, so you'll have to make sure that if your steps leave some processes running in the background which inherits its pipes straight from the parent shell, the SSH connection will hang until those processes are terminated (or until they close their pipes). Therefore, you should always redirect the stdout, stderr and stdin streams of your background processes, if you don't want your test steps / pre- / post steps to time out in host-based execution.

See http://www.snailbook.com/faq/background-jobs.auto.html for more information.

== Testrunner-lite state machine ==

Below is a simplified picture describing the operation of testrunner-lite. The program flow can be considered to consist of three phases: initializing, executing and clean up. The execution is done set by set, so that the tool needs to maintain only one set at a time in its memory.

[[File:testrunnerlitefsm.png]]

xfig file:
[[File:testrunnerlitesm.fig]]

== Further development ==
We're actively making testrunner-lite a better tool. See discussion about architecture changes and suggestions in following page:
[[Quality/QA-tools/Testrunner-lite/Testrunner-lite_development|Testrunner-lite development]]

Quality/QA-tools/Testrunner-lite

2011-01-07T15:06:33Z

Samposa: /* About test case verdicts */

= Testrunner-lite =

Testrunner-lite is a lightweight tool for test execution, which reads [[Quality/QA-tools/Test plan|Test Plan XML]] files as input, and produces [http://gitorious.org/qa-tools/test-definition/blobs/master/src/data/testdefinition-results.xsd a common-format] test result file which can be published at [http://qa-reports.meego.com/ qa-reports.meego.com]. With testrunner-lite, you can
* Execute automatic, semi-automatic and manual test cases
* Execute test cases locally or in host-based mode
* Use [[Quality/QA-tools/Testrunner-lite#Filtering options|filters]] to select the test cases to be executed
* Validate the used test plan file automatically

See our demo videos at Youtube:
* [http://www.youtube.com/watch?v=TZhHDPUeHIw Demo of version 1.3.17]
* [http://www.youtube.com/watch?v=A0V94xZ_VwI Demo of version 1.3.11]
* [http://www.youtube.com/watch?v=OSF8tyUxI8U Demo of version 1.3.10]
* [http://www.youtube.com/user/meegoqatools MeeGo QA tools Youtube channel]

There is also a graphical user interface version of the tool called [[Quality/QA-tools/Testrunner|Testrunner]] available.

== Installation ==

Testrunner-lite can be installed on MeeGo Netbook UX, Ubuntu 10.04 (and newer) and Fedora 13. First, [[Quality/QA-tools/How to set up repositories|set up the repository]]. Then, install testrunner-lite:

MeeGo Netbook 1.1:

sudo zypper install testrunner-lite

Ubuntu:

sudo apt-get install testrunner-lite

Fedora 13:

# yum install testrunner-lite

== Usage ==

The program is executed from command line

/usr/bin/testrunner-lite [options]
Options

-h, --help
Display short help and exit

-V, --version
Display version information and exit

-f FILE, --file FILE
Input file with test definitions in XML (required)

-o FILE, --output FILE
Output file format; FORMAT can be xml or text (Default xml)

-v, -vv, --verbose[={INFO|DEBUG}]
Enable verbosity mode; -v and --verbose=INFO are equivalent outputting INFO, ERROR and WARNING messages. Similarly -vv and --verbose=DEBUG are equivalent, outputting also debug messages. Default behaviour is silent mode.

-L URL, --logger=URL
Remote HTTP logger. Log messages are sent to given URL in a HTTP POST message. URL format is [http://]host[:port][/path/], where host may be a hostname or an IPv4 address.

-a, --automatic
Enable only automatic tests to be executed

-S, --syslog
Enable logging to syslog

-P, --print-step-output
Output standard streams from test steps.

-m, --manual
Enable only manual tests to be executed

-c, --ci
Disable validation of test definition against schema

-s, --semantic
Enable validation of test definition against stricter (semantics) schema

-A, --validate-only
Do only input xml validation, do not execute tests

-H, --no-hwinfo
Do not try to obtain hardware information

-t [USER@]ADDRESS, --target=[USER@]ADDRESS
Enable host-based testing. If given, commands are executed from test control PC (host) side. ADDRESS is the ipv4 adress of the system under test.

=== Filtering options ===

Filtering options allow selecting which tests are to be executed. Filtering options are given as a string. String may contain one or more filter entries. Format of each entry is 'filtername=values' where 'filtername' typically corresponds to an attribute in Test Definition XML and 'values' is a string to which the filter should match. Multiple values can be specified if separated by comma (without extra space). A value containing space must be enclosed in double quotes (""). Current list of forbidden characters in values is as follows: Single quote ('), double quote ("), is-equal-to sign (=), comma (,).

How filtering is carried out: Before processing the filter options, all tests are active by default. Filters can only deactivate tests: If filter value does not match with the value of the corresponding attribute, the test is disabled. Each filter entry is applied for all the active tests in the test package, one after another, in the given order. Filtering is always carried out at the lowest level of the "suite-set-case-step" hierarchy where the corresponding attribute can be defined. Note that some attributes may inherit the value from the upper level. Please refer to Test Definition XML for details.

If 'filtername' is prefixed with dash (-), the filter behaviour is reversed: those tests for which the filter value does match, are disabled.

'filtername' can be any of the following:

* feature
* requirement
* testset
* type
* testcase

The following example does the following: First it disables all test cases except those with attribute type='Integration'. Next, a test case named as Bad Test is disabled. The tests left active will be executed.

testrunner-lite -f tests.xml -o ~/results -l 'type=Integration -testcase="Bad Test"'

The following example executes test cases that specify the requirement attribute with a value containing at least one of the following strings: '50001', '50002', '50003'.

testrunner-lite -f tests.xml -o ~/results -l 'requirement=50001,50002,50003'

Note that each key=values is handled as a separate filter, when checking whether a test case should be filtered. The following example will filter all the test cases,

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1 testset=set2'

whereas the following will accept tests from test sets "set1" and "set2".

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1,set2'

== Manual Test Cases ==

It is also possible to execute manual test cases using testrunner-lite, as defined in Test Definition XML.

In case manual test case is encountered during execution, testrunner-lite will go through the defined test steps and ask user whether the step is passed or failed. The test case will terminate at the first failure, otherwise every step defined will be executed. After the test case is done, user has the option to enter additional comments.

Example output when running manual case:

[INFO] 15:15:53 Starting test case: ExampleTestCase
--- Execute test step ---
Description:
Open calculator. Expected result: calculator opens up.

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Execute test step ---
Description:
Stop calculator

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Test steps executed, case is PASSED ---
Please enter additional comments (ENTER to finish):
Execution was slow.

[INFO] 15:16:41 Finished test case. Result: PASS

== About test case verdicts ==

For manual cases the result is naturally set by the user. For automatic cases a result of a test case is set as follows:
* PASS
** all test steps finished with expected results
* FAIL
** test step has unexpected return value
** test step timed out
* N/A
** test case has state=Design in the tests.xml
** test case has no steps

== About process control ==

Each test step is executed in a separate shell. Testrunner-lite spawns new process for the execution, and waits for the step to finish (or timeout). In case test step contains command that is started to background, the step returns immediately. After test case has finnished a cleanup routine is executed, where testrunner-lite tries to kill all processes that may have been left running by the test steps. Cleanup for pre-steps and post-steps is done after the post steps are executed (i.e. when test set has been executed).

== About host based execution ==

Testrunner-lite supports host based execution, where testrunner-lite is executed on a PC, and the test steps over ssh on hardware. This requires that key based ssh authentication is enabled between device and host.

Remember that each single step in pre_steps, post_steps or inside testcases is executed in a separate SSH session, so you'll have to make sure that if your steps leave some processes running in the background which inherits its pipes straight from the parent shell, the SSH connection will hang until those processes are terminated (or until they close their pipes). Therefore, you should always redirect the stdout, stderr and stdin streams of your background processes, if you don't want your test steps / pre- / post steps to time out in host-based execution.

See http://www.snailbook.com/faq/background-jobs.auto.html for more information.

== Testrunner-lite state machine ==

Below is a simplified picture describing the operation of testrunner-lite. The program flow can be considered to consist of three phases: initializing, executing and clean up. The execution is done set by set, so that the tool needs to maintain only one set at a time in its memory.

[[File:testrunnerlitefsm.png]]

xfig file:
[[File:testrunnerlitesm.fig]]

== Further development ==
We're actively making testrunner-lite a better tool. See discussion about architecture changes and suggestions in following page:
[[Quality/QA-tools/Testrunner-lite/Testrunner-lite_development|Testrunner-lite development]]

Quality/QA-tools/Testrunner-lite

2011-01-07T15:05:38Z

Samposa: /* About test case verdicts */

= Testrunner-lite =

Testrunner-lite is a lightweight tool for test execution, which reads [[Quality/QA-tools/Test plan|Test Plan XML]] files as input, and produces [http://gitorious.org/qa-tools/test-definition/blobs/master/src/data/testdefinition-results.xsd a common-format] test result file which can be published at [http://qa-reports.meego.com/ qa-reports.meego.com]. With testrunner-lite, you can
* Execute automatic, semi-automatic and manual test cases
* Execute test cases locally or in host-based mode
* Use [[Quality/QA-tools/Testrunner-lite#Filtering options|filters]] to select the test cases to be executed
* Validate the used test plan file automatically

See our demo videos at Youtube:
* [http://www.youtube.com/watch?v=TZhHDPUeHIw Demo of version 1.3.17]
* [http://www.youtube.com/watch?v=A0V94xZ_VwI Demo of version 1.3.11]
* [http://www.youtube.com/watch?v=OSF8tyUxI8U Demo of version 1.3.10]
* [http://www.youtube.com/user/meegoqatools MeeGo QA tools Youtube channel]

There is also a graphical user interface version of the tool called [[Quality/QA-tools/Testrunner|Testrunner]] available.

== Installation ==

Testrunner-lite can be installed on MeeGo Netbook UX, Ubuntu 10.04 (and newer) and Fedora 13. First, [[Quality/QA-tools/How to set up repositories|set up the repository]]. Then, install testrunner-lite:

MeeGo Netbook 1.1:

sudo zypper install testrunner-lite

Ubuntu:

sudo apt-get install testrunner-lite

Fedora 13:

# yum install testrunner-lite

== Usage ==

The program is executed from command line

/usr/bin/testrunner-lite [options]
Options

-h, --help
Display short help and exit

-V, --version
Display version information and exit

-f FILE, --file FILE
Input file with test definitions in XML (required)

-o FILE, --output FILE
Output file format; FORMAT can be xml or text (Default xml)

-v, -vv, --verbose[={INFO|DEBUG}]
Enable verbosity mode; -v and --verbose=INFO are equivalent outputting INFO, ERROR and WARNING messages. Similarly -vv and --verbose=DEBUG are equivalent, outputting also debug messages. Default behaviour is silent mode.

-L URL, --logger=URL
Remote HTTP logger. Log messages are sent to given URL in a HTTP POST message. URL format is [http://]host[:port][/path/], where host may be a hostname or an IPv4 address.

-a, --automatic
Enable only automatic tests to be executed

-S, --syslog
Enable logging to syslog

-P, --print-step-output
Output standard streams from test steps.

-m, --manual
Enable only manual tests to be executed

-c, --ci
Disable validation of test definition against schema

-s, --semantic
Enable validation of test definition against stricter (semantics) schema

-A, --validate-only
Do only input xml validation, do not execute tests

-H, --no-hwinfo
Do not try to obtain hardware information

-t [USER@]ADDRESS, --target=[USER@]ADDRESS
Enable host-based testing. If given, commands are executed from test control PC (host) side. ADDRESS is the ipv4 adress of the system under test.

=== Filtering options ===

Filtering options allow selecting which tests are to be executed. Filtering options are given as a string. String may contain one or more filter entries. Format of each entry is 'filtername=values' where 'filtername' typically corresponds to an attribute in Test Definition XML and 'values' is a string to which the filter should match. Multiple values can be specified if separated by comma (without extra space). A value containing space must be enclosed in double quotes (""). Current list of forbidden characters in values is as follows: Single quote ('), double quote ("), is-equal-to sign (=), comma (,).

How filtering is carried out: Before processing the filter options, all tests are active by default. Filters can only deactivate tests: If filter value does not match with the value of the corresponding attribute, the test is disabled. Each filter entry is applied for all the active tests in the test package, one after another, in the given order. Filtering is always carried out at the lowest level of the "suite-set-case-step" hierarchy where the corresponding attribute can be defined. Note that some attributes may inherit the value from the upper level. Please refer to Test Definition XML for details.

If 'filtername' is prefixed with dash (-), the filter behaviour is reversed: those tests for which the filter value does match, are disabled.

'filtername' can be any of the following:

* feature
* requirement
* testset
* type
* testcase

The following example does the following: First it disables all test cases except those with attribute type='Integration'. Next, a test case named as Bad Test is disabled. The tests left active will be executed.

testrunner-lite -f tests.xml -o ~/results -l 'type=Integration -testcase="Bad Test"'

The following example executes test cases that specify the requirement attribute with a value containing at least one of the following strings: '50001', '50002', '50003'.

testrunner-lite -f tests.xml -o ~/results -l 'requirement=50001,50002,50003'

Note that each key=values is handled as a separate filter, when checking whether a test case should be filtered. The following example will filter all the test cases,

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1 testset=set2'

whereas the following will accept tests from test sets "set1" and "set2".

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1,set2'

== Manual Test Cases ==

It is also possible to execute manual test cases using testrunner-lite, as defined in Test Definition XML.

In case manual test case is encountered during execution, testrunner-lite will go through the defined test steps and ask user whether the step is passed or failed. The test case will terminate at the first failure, otherwise every step defined will be executed. After the test case is done, user has the option to enter additional comments.

Example output when running manual case:

[INFO] 15:15:53 Starting test case: ExampleTestCase
--- Execute test step ---
Description:
Open calculator. Expected result: calculator opens up.

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Execute test step ---
Description:
Stop calculator

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Test steps executed, case is PASSED ---
Please enter additional comments (ENTER to finish):
Execution was slow.

[INFO] 15:16:41 Finished test case. Result: PASS

== About test case verdicts ==

For manual cases the result is naturally set by the user. For automatic cases a result of a test case is as follows:
* PASS
** all test steps finished with expected results
* FAIL
** test step has unexpected return value
** test step timed out
* N/A
** test case has state=Design in the tests.xml
** automatic test case has no steps

== About process control ==

Each test step is executed in a separate shell. Testrunner-lite spawns new process for the execution, and waits for the step to finish (or timeout). In case test step contains command that is started to background, the step returns immediately. After test case has finnished a cleanup routine is executed, where testrunner-lite tries to kill all processes that may have been left running by the test steps. Cleanup for pre-steps and post-steps is done after the post steps are executed (i.e. when test set has been executed).

== About host based execution ==

Testrunner-lite supports host based execution, where testrunner-lite is executed on a PC, and the test steps over ssh on hardware. This requires that key based ssh authentication is enabled between device and host.

Remember that each single step in pre_steps, post_steps or inside testcases is executed in a separate SSH session, so you'll have to make sure that if your steps leave some processes running in the background which inherits its pipes straight from the parent shell, the SSH connection will hang until those processes are terminated (or until they close their pipes). Therefore, you should always redirect the stdout, stderr and stdin streams of your background processes, if you don't want your test steps / pre- / post steps to time out in host-based execution.

See http://www.snailbook.com/faq/background-jobs.auto.html for more information.

== Testrunner-lite state machine ==

Below is a simplified picture describing the operation of testrunner-lite. The program flow can be considered to consist of three phases: initializing, executing and clean up. The execution is done set by set, so that the tool needs to maintain only one set at a time in its memory.

[[File:testrunnerlitefsm.png]]

xfig file:
[[File:testrunnerlitesm.fig]]

== Further development ==
We're actively making testrunner-lite a better tool. See discussion about architecture changes and suggestions in following page:
[[Quality/QA-tools/Testrunner-lite/Testrunner-lite_development|Testrunner-lite development]]

Quality/QA-tools/Testrunner-lite

2011-01-07T15:05:16Z

Samposa: /* Manual Test Cases */

= Testrunner-lite =

Testrunner-lite is a lightweight tool for test execution, which reads [[Quality/QA-tools/Test plan|Test Plan XML]] files as input, and produces [http://gitorious.org/qa-tools/test-definition/blobs/master/src/data/testdefinition-results.xsd a common-format] test result file which can be published at [http://qa-reports.meego.com/ qa-reports.meego.com]. With testrunner-lite, you can
* Execute automatic, semi-automatic and manual test cases
* Execute test cases locally or in host-based mode
* Use [[Quality/QA-tools/Testrunner-lite#Filtering options|filters]] to select the test cases to be executed
* Validate the used test plan file automatically

See our demo videos at Youtube:
* [http://www.youtube.com/watch?v=TZhHDPUeHIw Demo of version 1.3.17]
* [http://www.youtube.com/watch?v=A0V94xZ_VwI Demo of version 1.3.11]
* [http://www.youtube.com/watch?v=OSF8tyUxI8U Demo of version 1.3.10]
* [http://www.youtube.com/user/meegoqatools MeeGo QA tools Youtube channel]

There is also a graphical user interface version of the tool called [[Quality/QA-tools/Testrunner|Testrunner]] available.

== Installation ==

Testrunner-lite can be installed on MeeGo Netbook UX, Ubuntu 10.04 (and newer) and Fedora 13. First, [[Quality/QA-tools/How to set up repositories|set up the repository]]. Then, install testrunner-lite:

MeeGo Netbook 1.1:

sudo zypper install testrunner-lite

Ubuntu:

sudo apt-get install testrunner-lite

Fedora 13:

# yum install testrunner-lite

== Usage ==

The program is executed from command line

/usr/bin/testrunner-lite [options]
Options

-h, --help
Display short help and exit

-V, --version
Display version information and exit

-f FILE, --file FILE
Input file with test definitions in XML (required)

-o FILE, --output FILE
Output file format; FORMAT can be xml or text (Default xml)

-v, -vv, --verbose[={INFO|DEBUG}]
Enable verbosity mode; -v and --verbose=INFO are equivalent outputting INFO, ERROR and WARNING messages. Similarly -vv and --verbose=DEBUG are equivalent, outputting also debug messages. Default behaviour is silent mode.

-L URL, --logger=URL
Remote HTTP logger. Log messages are sent to given URL in a HTTP POST message. URL format is [http://]host[:port][/path/], where host may be a hostname or an IPv4 address.

-a, --automatic
Enable only automatic tests to be executed

-S, --syslog
Enable logging to syslog

-P, --print-step-output
Output standard streams from test steps.

-m, --manual
Enable only manual tests to be executed

-c, --ci
Disable validation of test definition against schema

-s, --semantic
Enable validation of test definition against stricter (semantics) schema

-A, --validate-only
Do only input xml validation, do not execute tests

-H, --no-hwinfo
Do not try to obtain hardware information

-t [USER@]ADDRESS, --target=[USER@]ADDRESS
Enable host-based testing. If given, commands are executed from test control PC (host) side. ADDRESS is the ipv4 adress of the system under test.

=== Filtering options ===

Filtering options allow selecting which tests are to be executed. Filtering options are given as a string. String may contain one or more filter entries. Format of each entry is 'filtername=values' where 'filtername' typically corresponds to an attribute in Test Definition XML and 'values' is a string to which the filter should match. Multiple values can be specified if separated by comma (without extra space). A value containing space must be enclosed in double quotes (""). Current list of forbidden characters in values is as follows: Single quote ('), double quote ("), is-equal-to sign (=), comma (,).

How filtering is carried out: Before processing the filter options, all tests are active by default. Filters can only deactivate tests: If filter value does not match with the value of the corresponding attribute, the test is disabled. Each filter entry is applied for all the active tests in the test package, one after another, in the given order. Filtering is always carried out at the lowest level of the "suite-set-case-step" hierarchy where the corresponding attribute can be defined. Note that some attributes may inherit the value from the upper level. Please refer to Test Definition XML for details.

If 'filtername' is prefixed with dash (-), the filter behaviour is reversed: those tests for which the filter value does match, are disabled.

'filtername' can be any of the following:

* feature
* requirement
* testset
* type
* testcase

The following example does the following: First it disables all test cases except those with attribute type='Integration'. Next, a test case named as Bad Test is disabled. The tests left active will be executed.

testrunner-lite -f tests.xml -o ~/results -l 'type=Integration -testcase="Bad Test"'

The following example executes test cases that specify the requirement attribute with a value containing at least one of the following strings: '50001', '50002', '50003'.

testrunner-lite -f tests.xml -o ~/results -l 'requirement=50001,50002,50003'

Note that each key=values is handled as a separate filter, when checking whether a test case should be filtered. The following example will filter all the test cases,

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1 testset=set2'

whereas the following will accept tests from test sets "set1" and "set2".

testrunner-lite -f tests.xml -o ~/results -l 'testset=set1,set2'

== Manual Test Cases ==

It is also possible to execute manual test cases using testrunner-lite, as defined in Test Definition XML.

In case manual test case is encountered during execution, testrunner-lite will go through the defined test steps and ask user whether the step is passed or failed. The test case will terminate at the first failure, otherwise every step defined will be executed. After the test case is done, user has the option to enter additional comments.

Example output when running manual case:

[INFO] 15:15:53 Starting test case: ExampleTestCase
--- Execute test step ---
Description:
Open calculator. Expected result: calculator opens up.

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Execute test step ---
Description:
Stop calculator

Please enter the result ([P/p]ass or [F/f]ail):
P

--- Test steps executed, case is PASSED ---
Please enter additional comments (ENTER to finish):
Execution was slow.

[INFO] 15:16:41 Finished test case. Result: PASS

== About test case verdicts ==

For manual cases the result is naturally set by the user. For automatic cases a result of a test case is as follows:
* PASS
** all test steps finished with expected results
* FAIL
** test step has unexpected return value
** test step timed out
* N/A
** test case has state=design in the tests.xml
** automatic test case has no steps

== About process control ==

Each test step is executed in a separate shell. Testrunner-lite spawns new process for the execution, and waits for the step to finish (or timeout). In case test step contains command that is started to background, the step returns immediately. After test case has finnished a cleanup routine is executed, where testrunner-lite tries to kill all processes that may have been left running by the test steps. Cleanup for pre-steps and post-steps is done after the post steps are executed (i.e. when test set has been executed).

== About host based execution ==

Testrunner-lite supports host based execution, where testrunner-lite is executed on a PC, and the test steps over ssh on hardware. This requires that key based ssh authentication is enabled between device and host.

Remember that each single step in pre_steps, post_steps or inside testcases is executed in a separate SSH session, so you'll have to make sure that if your steps leave some processes running in the background which inherits its pipes straight from the parent shell, the SSH connection will hang until those processes are terminated (or until they close their pipes). Therefore, you should always redirect the stdout, stderr and stdin streams of your background processes, if you don't want your test steps / pre- / post steps to time out in host-based execution.

See http://www.snailbook.com/faq/background-jobs.auto.html for more information.

== Testrunner-lite state machine ==

Below is a simplified picture describing the operation of testrunner-lite. The program flow can be considered to consist of three phases: initializing, executing and clean up. The execution is done set by set, so that the tool needs to maintain only one set at a time in its memory.

[[File:testrunnerlitefsm.png]]

xfig file:
[[File:testrunnerlitesm.fig]]

== Further development ==
We're actively making testrunner-lite a better tool. See discussion about architecture changes and suggestions in following page:
[[Quality/QA-tools/Testrunner-lite/Testrunner-lite_development|Testrunner-lite development]]

Quality/QA-tools

2010-12-23T10:42:38Z

Samposa: /* Features and Bugs */

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://meego.gitorious.org/meego-quality-assurance/ git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Currently test reports are published manually to wiki
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools weekly meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner-ui|Testrunner UI]] || [http://meego.gitorious.org/meego-quality-assurance/testrunner Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://meego.gitorious.org/meego-quality-assurance/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://meego.gitorious.org/meego-quality-assurance/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| MeeGo Core Test Suite || [http://gitorious.org/qa-tools/mcts Gitorious] ||Matti Salmi || Jeff Zheng
|-
| MeeGo Netbook Test Suite || [http://gitorious.org/qa-tools/mnts Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://meego.gitorious.org/meego-quality-assurance/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://gitorious.org/qa-tools/rich-core Gitorious] || Riku Halonen || Sami Lahtinen
|-
| Crash Reporter || [http://gitorious.org/qa-tools/crash-reporter Gitorious] || Riku Halonen || Raimo Gratseff
|-
| Crash Reporter settings || [http://gitorious.org/qa-tools/crash-reporter-settings-public Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/hat-control|Hardware Accessory for Testing (HAT)]] || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| [[Quality/QA-tools/QAReports|QA Reports]] || [http://meego.gitorious.org/meego-quality-assurance/qa-reports Gitorious] || Janne Hietamäki || Jarno Keskikangas
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] || [http://meego.gitorious.com/meego-quality-assurance/scripts-and-utils Gitorious] || N/A || N/A
|-
| [[Quality/TestSuite/handset-test-suite/handset-ux-test_Releases | handset_ux_tests]] || [http://meego.gitorious.com/meego-quality-assurance/handset-ux-tests Gitorious] || JessicaJi || N/A
|-
| Fast Feedback Testing (FFT) || [http://meego.gitorious.org/meego-quality-assurance/meego-testing-hudson-plugin Gitorious] || Alexey Kuznetsov || Timo Härkönen
|}

You can install Testrunner UI, testrunner-lite, Testplanner, OTS, Meego-ai and eat from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here's the workflow for QA-tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file:
[[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

If you shoot a video to YouTube, promote it on meego-qa mailing list!

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=Development%20Tools Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=testrunner%20UI&component=testrunner-lite&classification=MeeGo%20Projects&product=Meego%20Quality%20Assurance Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=testrunner%20UI&component=testrunner-lite&classification=MeeGo%20Projects&product=MeeGo%20Quality%20Assurance All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS-Test-Automation-Design|MCTS Test Automation]] (QA Tools support for MCTS) '''WORK IN PROGRESS'''

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

=== Team meetings ===

QA tools team meetings will be held on need basis for specific topics. This was agreed in [http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-07-11.59.html the last weekly meeting held on December 7th 2010]

* [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

=== Architecture meetings ===

Architecture meetings will be held on-demand basis i.e. when topics do not fit in QA-Tools weekly meeting time frame.

* [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

=== Workshops ===

Face-2-face meetings within the team.

* [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
* [http://timoph.fi/qa-tools-stats/ #meego-qa-tools statistics]
* [http://timoph.fi/qa-tools-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools

2010-12-10T13:06:04Z

Samposa: /* Tools and Maintainers */

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://meego.gitorious.org/meego-quality-assurance/ git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Currently test reports are published manually to wiki
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools weekly meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner-ui|Testrunner UI]] || [http://gitorious.org/qa-tools/testrunner-ui Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://gitorious.org/qa-tools/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://meego.gitorious.org/meego-quality-assurance/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| MeeGo Core Test Suite || [http://gitorious.org/qa-tools/mcts Gitorious] ||Matti Salmi || Jeff Zheng
|-
| MeeGo Netbook Test Suite || [http://gitorious.org/qa-tools/mnts Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://meego.gitorious.org/meego-quality-assurance/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://gitorious.org/qa-tools/rich-core Gitorious] || Riku Halonen || Sami Lahtinen
|-
| Crash Reporter || [http://gitorious.org/qa-tools/crash-reporter Gitorious] || Riku Halonen || Raimo Gratseff
|-
| Crash Reporter settings || [http://gitorious.org/qa-tools/crash-reporter-settings-public Gitorious] || Riku Halonen || N/A
|-
| Hardware Accessory for Testing (HAT) || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| QA Reports || [http://github.com/leonidas/meego-qa-reports Github] || Janne Hietamäki || Jarno Keskikangas
|-
| [[Quality/QA-tools/Scripts_and_utils|Scripts and utils]] || [http://meego.gitorious.com/meego-quality-assurance/scripts-and-utils Gitorious] || N/A || N/A
|-
| Fast Feedback Testing (FFT) || [http://meego.gitorious.org/meego-quality-assurance/meego-testing-hudson-plugin Gitorious] || Alexey Kuznetsov || Timo Härkönen
|}

You can install Testrunner UI, testrunner-lite, Testplanner, OTS, Meego-ai and eat from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here's the workflow for QA-tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file:
[[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

If you shoot a video to YouTube, promote it on meego-qa mailing list!

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=Development%20Tools Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=testrunner%20UI&component=testrunner-lite&classification=MeeGo%20Platform&product=Development%20Tools Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=testrunner%20UI&component=testrunner-lite&classification=MeeGo%20Platform&product=Development%20Tools All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS-Test-Automation-Design|MCTS Test Automation]] (QA Tools support for MCTS) WORK IN PROGRESS

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

=== Team meetings ===

QA tools team meetings will be held on need basis for specific topics. This was agreed in [http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-07-11.59.html the last weekly meeting held on December 7th 2010]

* [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

=== Architecture meetings ===

Architecture meetings will be held on-demand basis i.e. when topics do not fit in QA-Tools weekly meeting time frame.

* [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

=== Workshops ===

Face-2-face meetings within the team.

* [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
* [http://koti.kapsi.fi/mjolk/qa-tools/ #meego-qa-tools statistics]
* [http://koti.kapsi.fi/mjolk/qa-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* [http://meegoqatools.wordpress.com/ QA-tools team blog]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools

2010-12-02T07:09:09Z

Samposa: /* Tools and Maintainers */

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://gitorious.org/qa-tools git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Currently test reports are published manually to wiki
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools weekly meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner-ui|Testrunner UI]] || [http://gitorious.org/qa-tools/testrunner-ui Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://gitorious.org/qa-tools/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://meego.gitorious.org/meego-quality-assurance/enables-automated-testing Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://gitorious.org/qa-tools/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| MeeGo Core Test Suite || [http://gitorious.org/qa-tools/mcts Gitorious] ||Matti Salmi || Jeff Zheng
|-
| MeeGo Netbook Test Suite || [http://gitorious.org/qa-tools/mnts Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://gitorious.org/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://gitorious.org/qa-tools/rich-core Gitorious] || Riku Halonen || Sami Lahtinen
|-
| Crash Reporter || [http://gitorious.org/qa-tools/crash-reporter Gitorious] || Riku Halonen || Raimo Gratseff
|-
| Crash Reporter settings || [http://gitorious.org/qa-tools/crash-reporter-settings-public Gitorious] || Riku Halonen || N/A
|-
| Hardware Accessory for Testing (HAT) || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| QA Reports || [http://github.com/leonidas/meego-qa-reports Github] || Janne Hietamäki || Jarno Keskikangas
|}

You can install Testrunner UI, testrunner-lite, Testplanner, OTS, Meego-ai and eat from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here's the workflow for QA-tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file:
[[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

If you shoot a video to YouTube, promote it on meego-qa mailing list!

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=Development%20Tools Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=testrunner%20UI&component=testrunner-lite&classification=MeeGo%20Platform&product=Development%20Tools Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=testrunner%20UI&component=testrunner-lite&classification=MeeGo%20Platform&product=Development%20Tools All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS-Test-Automation-Design|MCTS Test Automation]] (QA Tools support for MCTS)

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

=== Weekly meetings ===

Meetings will be held weekly on Tuesdays 12:00 UTC.

* [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

=== Architecture meetings ===

Architecture meetings will be held on-demand basis i.e. when topics do not fit in QA-Tools weekly meeting time frame.

* [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

=== Workshops ===

Face-2-face meetings within the team.

* [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
* [http://koti.kapsi.fi/mjolk/qa-tools/ #meego-qa-tools statistics]
* [http://koti.kapsi.fi/mjolk/qa-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

Quality/QA-tools

2010-12-02T07:07:44Z

Samposa: /* Tools and Maintainers */

= Quality Assurance (QA) Tools =

Quality Assurance tools are developed to ensure MeeGo SW quality. QA tools team develops and maintains tools for Quality Assurance.

Open source tools – available for all, available for development and contributions. Make people accountable for quality.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines]. In addition, you may take personal clone from our [http://gitorious.org/qa-tools git repositories] and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

'''Targets:'''

* Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
* Currently test reports are published manually to wiki
* Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

== Tools and Maintainers ==

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed [[Quality/QA-tools/Meetings|in the QA-tools weekly meeting]] Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow [http://meego.com/about/contribution-guidelines MeeGo contribution guidelines] to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

{| border="1" cellspacing="0"
|+
|'''Tool (link to wiki page)''' ||'''Gitorious'''||'''Maintainer''' ||'''Substitute'''
|-
| [[Quality/QA-tools/Test-definition|test-definition]]||[http://meego.gitorious.org/meego-quality-assurance/test-definition Gitorious] || Sampo Saaristo || Timo Härkönen
|-
| [[Quality/QA-tools/Testrunner-lite|testrunner-lite]] ||[http://meego.gitorious.org/meego-quality-assurance/testrunner-lite Gitorious] || Sampo Saaristo || Kyösti Ranto
|-
| [[Quality/QA-tools/Testrunner-ui|Testrunner UI]] || [http://gitorious.org/qa-tools/testrunner-ui Gitorious] || Kyösti Ranto || Timo Härkönen
|-
| [[Quality/QA-tools/Testplanner|Testplanner]] ||[http://gitorious.org/qa-tools/testplanner Gitorious] || Kyösti Ranto || N/A
|-
| [[Quality/QA-tools/Eat|eat - enables automated testing]] ||[http://gitorious.org/qa-tools/eat Gitorious] || Timo Härkönen || Timo Mäkimattila
|-
| [[Quality/QA-tools/OTS|ots - open testing system]] ||[http://gitorious.org/qa-tools/ots Gitorious] || Teemu Vainio || Tom Galvin
|-
| [[Quality/QA-tools/Autotest-guide#Automatic_image_installations|MeeGo Automated installer]] ||[http://gitorious.org/qa-tools/meego-ai Gitorious] || Timo Härkönen || N/A
|-
| MeeGo Core Test Suite || [http://gitorious.org/qa-tools/mcts Gitorious] ||Matti Salmi || Jeff Zheng
|-
| MeeGo Netbook Test Suite || [http://gitorious.org/qa-tools/mnts Gitorious] ||Jeff Zheng || N/A
|-
| [[Quality/QA-tools/qtuitest-mbt-adapter|Model-Based Testing adapter for qtuitest]] || [http://gitorious.org/qa-tools/qtuitest-mbt-adapter Gitorious] || Riku Halonen || N/A
|-
| [[Quality/QA-tools/Min|MIN test framework]] || [http://gitorious.org/min Gitorious] || Sampo Saaristo || Timo Mäkimattila
|-
| [[Quality/QA-tools/TDriver|Testability Driver]] || [http://gitorious.org/TDriver Gitorious] || Petri Kiiskinen || Tatu Lahtela
|-
| Rich Core dumper || [http://gitorious.org/qa-tools/rich-core Gitorious] || Riku Halonen || Sami Lahtinen
|-
| Crash Reporter || [http://gitorious.org/qa-tools/crash-reporter Gitorious] || Riku Halonen || Raimo Gratseff
|-
| Crash Reporter settings || [http://gitorious.org/qa-tools/crash-reporter-settings-public Gitorious] || Riku Halonen || N/A
|-
| Hardware Accessory for Testing (HAT) || [http://meego.gitorious.org/meego-quality-assurance/hat-control Gitorious] || Marko Junttila || Riku Halonen
|-
| QA Reports || [http://github.com/leonidas/meego-qa-reports Github] || Janne Hietamäki || Jarno Keskikangas
|}

You can install Testrunner UI, testrunner-lite, Testplanner, OTS, Meego-ai and eat from Tools:Testing repository. The instructions for setting up the repositories can be found [[Quality/QA-tools/How_to_set_up_repositories|here]].

See the rest of our team members and our collaboration spaces [[Quality/QA-tools#Team_Members_and_Collaboration_Spaces|below]]. If you are interested in the user experience work regarding these tools, you can find more information [[Quality/QA-tools/User experience|here]].

=== Overview ===

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

[[File:testautomationtools.png]]

xfig file:[[File:qatools.fig]]

== Release Practices ==

Here's the workflow for QA-tools release practices.

{| border="1" cellspacing="0"
|+
|'''Role'''
|'''Description'''
|-
| Developer || Anyone who wants to participate in qa-tools development
|-
| VCS Maintainer || Component owner who has commit rights in version control system (VCS)
|-
| Package Maintainer || Integrator whose responsibility is the OBS packaging
|-
| Release Management || Third party who is responsible of trunk:testing releases(?)
|}

[[File:Release.png]]

Kivio file:
[[File:Release.flw]]

# Developer creates merge request(s) in gitorious.
# VCS Maintainer tests and accepts merge requests.
# VCS Maintainer checks/updates change logs.
# VCS Maintainer tags a version.
# VCS maintainer sends email to [http://lists.meego.com/pipermail/meego-qa/ meego-qa mailing list] based on the following template
Topic: Integration request: package-name version
PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs
# Package maintainer updates the OBS package.
# Package maintainer tests the OBS package.
# If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
# Host side tools are updated to tools:testing after verifying functionality
# Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
# Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

If you shoot a video to YouTube, promote it on meego-qa mailing list!

''' YouTube videos '''

YouTube is a good way to communicate new features. You can find existing demo videos on [http://www.youtube.com/user/meegoqatools meegoqatools channel on Youtube].

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: [[YouTube_Hints]]

=== Release checklist ===
To make sure fixes are released without delay, check that the following conditions are met

# Change logs are updated and contain relevant references to MeeGo bugzilla
# Created obs request include fixes bug numbers from MeeGo bugzilla
# Bugzilla items listed in changes are set as resolved
# Spec file matches [[Packaging/Guidelines|MeeGo packaging guidelines]]
# Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
# Host side tool packages use the same source tar ball to produce debian and rpm packages

== Features and Bugs ==
Want to report an feature idea or bug to us? - [http://bugs.meego.com/enter_bug.cgi?product=Development%20Tools Please do it here]

* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=ASSIGNED&component=eat&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=testrunner%20UI&component=testrunner-lite&classification=MeeGo%20Platform&product=Development%20Tools Assigned bugs and features - Working on it]
* [http://bugs.meego.com/buglist.cgi?query_format=advanced&order=Importance&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&bug_status=RESOLVED&component=eat&component=min&component=ots&component=TDriver&component=testdefinition&component=Testplanner&component=testrunner%20UI&component=testrunner-lite&classification=MeeGo%20Platform&product=Development%20Tools All open features and bugs in priority order]

Bugzilla workflow: [[Bugzilla/how-report-bugs]]

== Documentation ==

This section will contain links to various guides and user documentation. See [[Quality/QA-tools#Tools_and_Maintainers|the wiki pages of the tools]] for tool-specific documentation.

* [[Quality/QA-tools/Autotest-guide|Autotest-Guide]]
* [[Quality/QA-tools/How_to_set_up_repositories|How to set up the repositories that are needed to install QA tools]]

== Design/ Planning ==

* [[Quality/QA-tools/MCTS-Test-Automation-Design|MCTS Test Automation]] (QA Tools support for MCTS)

== Meetings ==

All meetings will be held in <code>#meego-meeting</code> on <code>irc.freenode.net</code>.

=== Weekly meetings ===

Meetings will be held weekly on Tuesdays 12:00 UTC.

* [[Quality/QA-tools/Meetings|Meeting agendas and minutes]]

=== Architecture meetings ===

Architecture meetings will be held on-demand basis i.e. when topics do not fit in QA-Tools weekly meeting time frame.

* [[Quality/QA-tools/Arch-Meetings|Meeting agendas and minutes]]

=== Workshops ===

Face-2-face meetings within the team.

* [[Quality/QA-tools/Workshops|Meeting agendas and minutes]]

== Team Members and Collaboration Spaces==

The current team members are (in no particular order):

{| border="1" cellspacing="0"
|+
| '''Name'''
| '''Role'''
| '''Affiliation'''
| '''IRC nickname'''
|-
| Ville Ilvonen || Team lead (act.) || Nokia || vilvo
|-
| Riku Halonen || Team member || Nokia || rikhalon
|-
| Kari Sievi || Team member || Digia || sievi
|-
| Timo Härkönen || Team member || Digia || timoph
|-
| Carol Rus || Team member || Digia || carrus
|-
| Sami Lahtinen || Team member || Digia || slahtinen
|-
| Raimo Gratseff || Team member || Digia || rrraimo
|-
| Kyösti Ranto || Team member || Digia || kyranto
|-
| Arto Sinnelä || Team member || Digia || asinnela
|-
| Joonas Kylänpää || Team member || Digia || Kaadlajk
|-
| Timo Mäkimattila || Team member || Digia || timakima
|-
| Elias Luttinen || Team member || Digia || eluttine
|-
| Ville Niutanen || Team member || Digia || Villen
|-
| Esa-Pekka Miettinen || Team member || Digia || E-P
|-
| Vesa Poikajärvi || Team member || Digia || vesse
|-
| Sampo Saaristo || Team member || Sofica || sampos
|-
| Ling Yu || Team member || Intel || -
|-
| Jing Wang || Team member || Intel || -
|-
| Teemu Vainio || Team member || Ixonos || tvainio
|-
| Tuomo Mäkinen || Team member || Ixonos || -
|-
| Jouni Leppäkases || Team member || Ixonos || jouni
|-
| Tom Galvin || Team member || Ixonos || -
|-
| Alexey Kuznetsov || Team member || Digia || alkuznet
|-
| Sergey Timofeev || Team member || Digia || setimofe
|-
| Daniil Chuiko || Team member || Digia || dachuiko
|-
| Jarno Keskikangas || Team member || Leonidas || jakeskik
|-
| Janne Hietamäki || Team member || Leonidas || _janne
|-
|}

Team communication is in English. Our collaboration spaces are:
* [http://lists.meego.com/listinfo/meego-dev meego-dev@meego.com mailing list], please prefix with 'QA-tools' for team related topics.
** Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
* [http://lists.meego.com/listinfo/meego-qa meego-qa@lists.meego.com mailing list]
* [http://webchat.freenode.net/?channels=meego-qa-tools #meego-qa-tools IRC channel on irc.freenode.net]
* [http://koti.kapsi.fi/mjolk/qa-tools/ #meego-qa-tools statistics]
* [http://koti.kapsi.fi/mjolk/qa-logs/ #meego-qa-tools irc logs]
* Gitorious team, http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* [http://www.youtube.com/user/meegoqatools Youtube channel for demo videos]
* MeeGo OBS - devel:quality
* This wiki area
* [[Quality/QA-tools/ServiceOS|ServiceOS]]
* [[Quality/QA-tools/PXEInstall|PXEInstallation]]

[[Category:QA]]

YouTube Hints

2010-11-24T11:17:46Z

Samposa: /* Do and donts */

There have been a lot of challenges in shooting, editing, and uploading a video successfully to YouTube especially on Linux. This page contains some hints how to shoot a video and how to upload it.

Feel free to add/combine your experiences here.

=== Formats ===
YouTube accepts a lot of video formats. However, in many cases the contents of an accepted video is scrambled when viewing the uploaded video. Uploading has been most reliable when using AVI format with a well-known resolution.

=== Screencasts ===
Use XVidCap to capture a screencast from a selected area of your desktop in MPEG format.

If you prefer shooting mute screencasts and record a soundtrack separately, you can use any recording software for recording. A common option is gnome-sound-recorder.

Gstreamer can be used to add a soundtrack to a mute screencast. The following spell combines an MPEG video with an ogg vorbis soundtrack and produces an avi video.

gst-launch-0.10 filesrc location=my_video.mpeg ! \
decodebin ! videoscale ! video/x-raw-yuv,width=1280,height=720 ! ffmpegcolorspace ! queue ! xvidenc ! \
queue max-size-bytes=1000000 max-size-time=0 ! avimux name=mux \
filesrc location=my_audio.ogg ! oggdemux ! vorbisdec ! audioconvert ! audioresample ! lame ! \
queue max-size-bytes=1000000 max-size-time=0 ! mux. \
mux. ! queue max-size-bytes=1000000 ! filesink location=final_video.avi

=== Do and donts ===
Pitivi video editor looks promising for editing videos. However, it does not cope well with non-standard resolutions, formats, or framerates. And it's not known how to export a proper HD video from it.

Record-my-desktop. Any experiences?

* Do not tweak the settings, go with defaults :) (sampos)
* I had problems converting the ogv from record-my-desktop to avi with mencoder or ffmpeg, finally managed with winff. (sampos)