MeeGo wiki - User contributions [en]

MeeGo Conference 2010/Unconference

2010-11-12T07:08:45Z

Ttoropainen: /* I plan to present ... */

Day 3 of the [http://conference2010.meego.com MeeGo Conference] will be primarily an unconference run in the BarCamp style. We will also have one room with scheduled activities running in parallel to the unconference for Birds of a Feather sessions, lightning talks and similar activities.

== What is an Unconference and Why should I attend? ==

We've been getting a lot of questions about the unconference day from people who aren't convinced that it is worth their time to attend. [[User:Dawnfoster|I (Dawn)]] have attended many unconferences and organized several of them, and I can honestly say that I usually get as much, if not more, out of them than I do from traditional conferences.

=== Brief Description ===

An unconference is an ad-hoc gathering organized by the attendees for the attendees. The concept originated out of a desire for people to share and learn in an open environment. It is an intense event with discussions, demos and interaction from participants. You never quite know what to expect at an unconference, since each one is shaped by the attendees. When you arrive on Wednesday morning, there will be an agenda framework (times / rooms), but the content for the sessions will be decided by the participants.

=== Why should I attend ===

'''To Present and Lead'''
* Anyone can present or lead a discussion, so you can talk about any topic of interest to you.
* If you submitted a session, and it wasn't approved for the main conference, but you still think it would be interesting, you can present it here.
* We will have plenty of smaller rooms for niche topics of interest to a few people - those rarely fit into a conference format, but work nicely in one of our smaller unconference rooms.
* It's a great opportunity to follow up on a topic presented in the first days and have a more detailed discussion.

'''To Collaborate and Network'''
* Use the time to collaborate with people on an interesting MeeGo project.
* Great opportunity to talk to experts about subjects not covered in the first two days.
* Network and meet other people interested in our work
* Learn more about what others need and get feedback on new ways to use MeeGo
* Collaborate on actual work related to MeeGo in smaller groups
* Have group discussions or meetings

'''To Learn and Innovate'''
* It is a great way to spot new talents, new movements, tendencies and upcoming technologies that weren't mature enough for the main conference.
* An unconference is basically like the conference, but with subjects chosen by the audience (by interest), which means there's a high chance of finding interesting topics.
* We have a community of interesting smart people, so we should expect plenty of interesting sessions.
* The flexible format allows for interesting out of the box ideas and ways to accommodate different formats.
* You are encouraged to use the [http://en.wikipedia.org/wiki/Law_of_Two_Feet#Law_of_Two_Feet Law of Two Feet] - if you go to a session and it isn't what you had hoped, you are encouraged to get up and move to another session.
* Come home with a fresh set of cutting-edge ideas and inspiration at the front of innovation within MeeGo.

=== More information ===
* [http://www.youtube.com/watch?v=7XlqEDIJzfw What is BarCamp? video] - short and worth your time to watch.
* [http://en.wikipedia.org/wiki/Barcamp BarCamp Wikipedia entry] with some history and other information.

== Session Ideas ==

'''IMPORTANT: Adding a session to the list below does not reserve a spot on the schedule board on Wednesday. The only way to guarantee a schedule slot is by showing up early and putting your session on the grid in person. See the [[#Schedule]] section below for more details.'''

=== Tips for Presenters ===

* Discussion formats work slightly better than standard conference presentations. Leading a group of people in a discussion is much more interactive and gives people a nice break from the past 2 days of presentations.
* Prepare in advance - if you plan to lead a session, prepare as you would for any other conference.
* Have a topic that isn't 45 minutes of material? Team up with one or more people with related topics and share the session.
* Respect people's time and do not allow your session to run longer than scheduled. Other people will be waiting to get into the room to set up for the next session.

=== I plan to present ... ===

Leave a note with the topic you plan to present along with your name and a link to more information about yourself.

* The title of an example topic description --[[User:Dawnfoster|Dawnfoster]] 22:09, 21 September 2010 (UTC)
* [http://conference2010.meego.com/session/accounts-sso Introduction to the new Accounts & Single Sign On framework] --[[User:Mardy|Mardy]] 24 Sep 2010 11:52:11 (UTC)
* [http://conference2010.meego.com/session/content-fw-my-god-its-full-data Introduction to Tracker/Content FW: one of really innovating bits in Meego] --[[User:Ifrade|Frade]] 06 Oct 2010 13:21:00 (EETC)
* [[MeeGo_Conference2010/QtM_BoF_Unconference|QtMobility & MeeGo BoF/Unconference]] -- [[User:mikeleib|mikeleib]] 06 Oct 2010 04:03 UTC
* [[MeeGo_Conference_2010/Unconference/ARM|MeeGo ARM Discussion Forum]] -- [[User:stskeeps|stskeeps]] 07 Oct 2010 15:57 UTC
* [[MeeGo_Conference_2010/Unconference/Plasma|Plasma and QtComponents BoF]] -- [[User:mart|mart]] 07 Oct 2010 23:10 UTC
* [[MeeGo_Conference_2010/Unconference/TrackerHandsOn|Hands on BOF on making efficient SPARQL queries and mastering the different APIs of MeeGo's metadata service, Tracker]] -- [[User:pvanhoof|pvanhoof]], [[User:abustany|abustany]] 11 Oct 2010 10:34 UTC
* [http://conference2010.meego.com/session/working-together-world-ready-meego-developers-and-translators MeeGo Internationalization and Localization Discussion] [[User:margie|margie]]--A continuation of the discussion of the new process we are proposing to use for the next MeeGo release
* [http://wiki.meego.com/MeeGo_Conference_2010/Unconference/Hardware_adaptations_in_MeeGo Hardware adaptations in MeeGo] [[User:stskeeps|stskeeps]] -- An introduction to HW adaptations in MeeGo and the architecture and requirements process around it
* [http://lists.meego.com/pipermail/meego-qa/2010-November/000091.html Open test management proposal with tool support] [[User:vilvo|vilvo]] -- A proposal to make MeeGo core OS and UX profiles test management more open and how that can be supported with Quality Assurance tools.
* [http://lists.meego.com/pipermail/meego-community/2010-November/002504.html Get your hands dirty with test tools] [[User:timoph|timoph]] -- Information and live demos on test tools
* [[MeeGo_Conference2010/MCTS_CSA_Future|MeeGo Core Test Suite Current State and future workshop]] --

=== I would like to see someone talk about ... ===

Have something that you want to see someone else talk about? Feel free to use this space to brainstorm topics.

* Tips for contributing code to MeeGo and getting it accepted --[[User:Dawnfoster|Dawnfoster]] 22:09, 21 September 2010 (UTC)

* How to become a MeeGo platform hacker/developer- How to become part of the specification, design and development of the core platform, in one's favorite vertical.

* Q&A and technical discussion of core compliance package list --[[User:Joel|Joel Clark]] 09:04, 5 November 2010 (UTC)

== Schedule ==

=== Schedule Grid and Process for Scheduling Sessions ===

We will have a big grid on the wall where people can post their session ideas on the morning of the event (no sessions reserved in advance). Make sure you arrive on time if you plan to lead a session!

'''To Schedule a Session''':
* The schedule board will open up after the welcome is completed.
* Write your topic in big letters on your post it note.
* Include your contact information in smaller letters (name and cell phone or email address)
* Put your post it note on the board picking a room and time that looks good to you.
* Make sure that big meaty topics are in large rooms with niche topics in small rooms.

'''Things to keep in mind''':
* The schedule board is dynamic and may change at any time during the day.
* The schedule board on the wall is always the master.
* We may need to move sessions, so make sure you double check the board to make sure that you are going to the right room / time for the session you want to attend.
* Sessions that seem very similar may be asked to combine.
* We reserve the right to move things around to make sure that popular topics have large rooms.

=== Day Schedule and Grid Format ===

This is an example of how the grid will look and is for planning purposes only - '''please do not enter sessions or add other information to the grid'''

{| border="1"
!Time || Presidents 1 (sits 300) || Presidents 2 (sits 280) || Vavasour (sits 80) || 1872 area (sits 120) || Havelock (sits 130)||Box 1 (sits 20)||Box 2 (sits 20)|| Box 3 (sits 20)|| Box 4 (sits 20)|| Box 5 (sits 20)|| Box 6 (sits 20)|| Other
|-
!9:00 - 9:15 ||colspan=13|Atrium: Welcome - What is an unconference and how do I participate?
|-
!9:15 - 9:45 ||colspan=13|Atrium: Schedule board open - people post sessions on the grid.
|-
!9:45 - 10:00||colspan=13|Atrium: Schedule board adjustments (combine duplicates, adjust rooms, etc.)
|-
|10:00 - 10:45|| BOFs / Scheduled || Unconf Session || Unconf Session || Unconf Session || Unconf Session||Unconf Session||Unconf Session || Unconf Session ||Unconf Session || Unconf Session || Unconf Session
|-
|11:00 - 11:45|| BOFs / Scheduled || Unconf Session || Unconf Session || Unconf Session || Unconf Session||Unconf Session||Unconf Session || Unconf Session ||Unconf Session || Unconf Session || Unconf Session
|-
|12:00 - 12:45|| Lightning Talks || Unconf Session || Unconf Session || Unconf Session || Unconf Session||Unconf Session||Unconf Session || Unconf Session ||Unconf Session || Unconf Session || Unconf Session
|-
!12:45 - 13:45||colspan=13|Lunch and Networking
|-
|13:45 - 14:30|| BOFs / Scheduled || Unconf Session || Unconf Session || Unconf Session || Unconf Session||Unconf Session||Unconf Session || Unconf Session ||Unconf Session || Unconf Session || Unconf Session
|-
|14:45 - 15:30|| BOFs / Scheduled || Unconf Session || Unconf Session || Unconf Session || Unconf Session||Unconf Session||Unconf Session || Unconf Session ||Unconf Session || Unconf Session || Unconf Session
|-
!15:30 - 16:00||colspan=13|We need to be out of Aviva no later than 16:00.
|-
!17:45||colspan=13|Doors open for the Ireland / Norway Football game.
|-
|}

Note: The BOF / Scheduled session room is out of scope of the unconference team and will be decided and planned by the content committee.

== Requirements ==

* Rooms set up - confirm that we do plan to use all of the rooms on the schedule board & we need to know which rooms have projectors.
* A bunch of 4x6" post it notes
* A schedule board pre-printed with times and rooms with spaces large enough to accommodate 4x6" post it notes (alternative is painter's tape & a wall)
* Sharpie pens

== Organizing Team ==

* Lead: [[User:Dawnfoster|Dawn Foster]]
* Notes / wiki organizer - put a process in place to allow people to take notes and maintain a copy (physical board is the master) of the schedule in the wiki.
* Schedule volunteers - 2 people (2 shifts - each working half the day) to help manage the schedule board, help people get things scheduled, update schedule changes in the wiki copy, etc.
* Note taking volunteers - each session needs one or more people to take notes and put them in the wiki

MeeGo Conference 2010/Unconference

2010-11-12T06:58:56Z

Ttoropainen: /* I plan to present ... */

Day 3 of the [http://conference2010.meego.com MeeGo Conference] will be primarily an unconference run in the BarCamp style. We will also have one room with scheduled activities running in parallel to the unconference for Birds of a Feather sessions, lightning talks and similar activities.

== What is an Unconference and Why should I attend? ==

We've been getting a lot of questions about the unconference day from people who aren't convinced that it is worth their time to attend. [[User:Dawnfoster|I (Dawn)]] have attended many unconferences and organized several of them, and I can honestly say that I usually get as much, if not more, out of them than I do from traditional conferences.

=== Brief Description ===

An unconference is an ad-hoc gathering organized by the attendees for the attendees. The concept originated out of a desire for people to share and learn in an open environment. It is an intense event with discussions, demos and interaction from participants. You never quite know what to expect at an unconference, since each one is shaped by the attendees. When you arrive on Wednesday morning, there will be an agenda framework (times / rooms), but the content for the sessions will be decided by the participants.

=== Why should I attend ===

'''To Present and Lead'''
* Anyone can present or lead a discussion, so you can talk about any topic of interest to you.
* If you submitted a session, and it wasn't approved for the main conference, but you still think it would be interesting, you can present it here.
* We will have plenty of smaller rooms for niche topics of interest to a few people - those rarely fit into a conference format, but work nicely in one of our smaller unconference rooms.
* It's a great opportunity to follow up on a topic presented in the first days and have a more detailed discussion.

'''To Collaborate and Network'''
* Use the time to collaborate with people on an interesting MeeGo project.
* Great opportunity to talk to experts about subjects not covered in the first two days.
* Network and meet other people interested in our work
* Learn more about what others need and get feedback on new ways to use MeeGo
* Collaborate on actual work related to MeeGo in smaller groups
* Have group discussions or meetings

'''To Learn and Innovate'''
* It is a great way to spot new talents, new movements, tendencies and upcoming technologies that weren't mature enough for the main conference.
* An unconference is basically like the conference, but with subjects chosen by the audience (by interest), which means there's a high chance of finding interesting topics.
* We have a community of interesting smart people, so we should expect plenty of interesting sessions.
* The flexible format allows for interesting out of the box ideas and ways to accommodate different formats.
* You are encouraged to use the [http://en.wikipedia.org/wiki/Law_of_Two_Feet#Law_of_Two_Feet Law of Two Feet] - if you go to a session and it isn't what you had hoped, you are encouraged to get up and move to another session.
* Come home with a fresh set of cutting-edge ideas and inspiration at the front of innovation within MeeGo.

=== More information ===
* [http://www.youtube.com/watch?v=7XlqEDIJzfw What is BarCamp? video] - short and worth your time to watch.
* [http://en.wikipedia.org/wiki/Barcamp BarCamp Wikipedia entry] with some history and other information.

== Session Ideas ==

'''IMPORTANT: Adding a session to the list below does not reserve a spot on the schedule board on Wednesday. The only way to guarantee a schedule slot is by showing up early and putting your session on the grid in person. See the [[#Schedule]] section below for more details.'''

=== Tips for Presenters ===

* Discussion formats work slightly better than standard conference presentations. Leading a group of people in a discussion is much more interactive and gives people a nice break from the past 2 days of presentations.
* Prepare in advance - if you plan to lead a session, prepare as you would for any other conference.
* Have a topic that isn't 45 minutes of material? Team up with one or more people with related topics and share the session.
* Respect people's time and do not allow your session to run longer than scheduled. Other people will be waiting to get into the room to set up for the next session.

=== I plan to present ... ===

Leave a note with the topic you plan to present along with your name and a link to more information about yourself.

* The title of an example topic description --[[User:Dawnfoster|Dawnfoster]] 22:09, 21 September 2010 (UTC)
* [http://conference2010.meego.com/session/accounts-sso Introduction to the new Accounts & Single Sign On framework] --[[User:Mardy|Mardy]] 24 Sep 2010 11:52:11 (UTC)
* [http://conference2010.meego.com/session/content-fw-my-god-its-full-data Introduction to Tracker/Content FW: one of really innovating bits in Meego] --[[User:Ifrade|Frade]] 06 Oct 2010 13:21:00 (EETC)
* [[MeeGo_Conference2010/QtM_BoF_Unconference|QtMobility & MeeGo BoF/Unconference]] -- [[User:mikeleib|mikeleib]] 06 Oct 2010 04:03 UTC
* [[MeeGo_Conference_2010/Unconference/ARM|MeeGo ARM Discussion Forum]] -- [[User:stskeeps|stskeeps]] 07 Oct 2010 15:57 UTC
* [[MeeGo_Conference_2010/Unconference/Plasma|Plasma and QtComponents BoF]] -- [[User:mart|mart]] 07 Oct 2010 23:10 UTC
* [[MeeGo_Conference_2010/Unconference/TrackerHandsOn|Hands on BOF on making efficient SPARQL queries and mastering the different APIs of MeeGo's metadata service, Tracker]] -- [[User:pvanhoof|pvanhoof]], [[User:abustany|abustany]] 11 Oct 2010 10:34 UTC
* [http://conference2010.meego.com/session/working-together-world-ready-meego-developers-and-translators MeeGo Internationalization and Localization Discussion] [[User:margie|margie]]--A continuation of the discussion of the new process we are proposing to use for the next MeeGo release
* [http://wiki.meego.com/MeeGo_Conference_2010/Unconference/Hardware_adaptations_in_MeeGo Hardware adaptations in MeeGo] [[User:stskeeps|stskeeps]] -- An introduction to HW adaptations in MeeGo and the architecture and requirements process around it
* [http://lists.meego.com/pipermail/meego-qa/2010-November/000091.html Open test management proposal with tool support] [[User:vilvo|vilvo]] -- A proposal to make MeeGo core OS and UX profiles test management more open and how that can be supported with Quality Assurance tools.
* [http://lists.meego.com/pipermail/meego-community/2010-November/002504.html Get your hands dirty with test tools] [[User:timoph|timoph]] -- Information and live demos on test tools
* [[MeeGo_Conference2010/MCTS_CSA_Future|MeeGo Core Test Suite Current State and future workshop]] -- [[User:ttoropainen|ttoropainen]]

=== I would like to see someone talk about ... ===

Have something that you want to see someone else talk about? Feel free to use this space to brainstorm topics.

* Tips for contributing code to MeeGo and getting it accepted --[[User:Dawnfoster|Dawnfoster]] 22:09, 21 September 2010 (UTC)

* How to become a MeeGo platform hacker/developer- How to become part of the specification, design and development of the core platform, in one's favorite vertical.

* Q&A and technical discussion of core compliance package list --[[User:Joel|Joel Clark]] 09:04, 5 November 2010 (UTC)

== Schedule ==

=== Schedule Grid and Process for Scheduling Sessions ===

We will have a big grid on the wall where people can post their session ideas on the morning of the event (no sessions reserved in advance). Make sure you arrive on time if you plan to lead a session!

'''To Schedule a Session''':
* The schedule board will open up after the welcome is completed.
* Write your topic in big letters on your post it note.
* Include your contact information in smaller letters (name and cell phone or email address)
* Put your post it note on the board picking a room and time that looks good to you.
* Make sure that big meaty topics are in large rooms with niche topics in small rooms.

'''Things to keep in mind''':
* The schedule board is dynamic and may change at any time during the day.
* The schedule board on the wall is always the master.
* We may need to move sessions, so make sure you double check the board to make sure that you are going to the right room / time for the session you want to attend.
* Sessions that seem very similar may be asked to combine.
* We reserve the right to move things around to make sure that popular topics have large rooms.

=== Day Schedule and Grid Format ===

This is an example of how the grid will look and is for planning purposes only - '''please do not enter sessions or add other information to the grid'''

{| border="1"
!Time || Presidents 1 (sits 300) || Presidents 2 (sits 280) || Vavasour (sits 80) || 1872 area (sits 120) || Havelock (sits 130)||Box 1 (sits 20)||Box 2 (sits 20)|| Box 3 (sits 20)|| Box 4 (sits 20)|| Box 5 (sits 20)|| Box 6 (sits 20)|| Other
|-
!9:00 - 9:15 ||colspan=13|Atrium: Welcome - What is an unconference and how do I participate?
|-
!9:15 - 9:45 ||colspan=13|Atrium: Schedule board open - people post sessions on the grid.
|-
!9:45 - 10:00||colspan=13|Atrium: Schedule board adjustments (combine duplicates, adjust rooms, etc.)
|-
|10:00 - 10:45|| BOFs / Scheduled || Unconf Session || Unconf Session || Unconf Session || Unconf Session||Unconf Session||Unconf Session || Unconf Session ||Unconf Session || Unconf Session || Unconf Session
|-
|11:00 - 11:45|| BOFs / Scheduled || Unconf Session || Unconf Session || Unconf Session || Unconf Session||Unconf Session||Unconf Session || Unconf Session ||Unconf Session || Unconf Session || Unconf Session
|-
|12:00 - 12:45|| Lightning Talks || Unconf Session || Unconf Session || Unconf Session || Unconf Session||Unconf Session||Unconf Session || Unconf Session ||Unconf Session || Unconf Session || Unconf Session
|-
!12:45 - 13:45||colspan=13|Lunch and Networking
|-
|13:45 - 14:30|| BOFs / Scheduled || Unconf Session || Unconf Session || Unconf Session || Unconf Session||Unconf Session||Unconf Session || Unconf Session ||Unconf Session || Unconf Session || Unconf Session
|-
|14:45 - 15:30|| BOFs / Scheduled || Unconf Session || Unconf Session || Unconf Session || Unconf Session||Unconf Session||Unconf Session || Unconf Session ||Unconf Session || Unconf Session || Unconf Session
|-
!15:30 - 16:00||colspan=13|We need to be out of Aviva no later than 16:00.
|-
!17:45||colspan=13|Doors open for the Ireland / Norway Football game.
|-
|}

Note: The BOF / Scheduled session room is out of scope of the unconference team and will be decided and planned by the content committee.

== Requirements ==

* Rooms set up - confirm that we do plan to use all of the rooms on the schedule board & we need to know which rooms have projectors.
* A bunch of 4x6" post it notes
* A schedule board pre-printed with times and rooms with spaces large enough to accommodate 4x6" post it notes (alternative is painter's tape & a wall)
* Sharpie pens

== Organizing Team ==

* Lead: [[User:Dawnfoster|Dawn Foster]]
* Notes / wiki organizer - put a process in place to allow people to take notes and maintain a copy (physical board is the master) of the schedule in the wiki.
* Schedule volunteers - 2 people (2 shifts - each working half the day) to help manage the schedule board, help people get things scheduled, update schedule changes in the wiki copy, etc.
* Note taking volunteers - each session needs one or more people to take notes and put them in the wiki

MeeGo Conference2010/MCTS CSA Future

2010-11-12T06:56:57Z

Ttoropainen: Created page with "'''MeeGo Core Test Suite Current State and future workshop''' MeeGo Core Test Suite (MCTS) [1] is set of test components intended for verification of MeeGo Core OS according to …"

'''MeeGo Core Test Suite Current State and future workshop'''

MeeGo Core Test Suite (MCTS) [1] is set of test components intended for verification of MeeGo Core OS according to test plan [2] covering functional and non-functional (efficiency and reliability) aspects [3]. MCTS is actively used by MeeGo Core OS QA. Currently MCTS offers more than 2000 test cases for various components including component level test cases and system level, E2E type test cases.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test cases are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

Currently we have had contributions to MCTS from different sources and there are lots of variations in methods how test cases are created, test case quality and how test cases are documented. Also there are overlapping test cases e.g. currently BlueZ test cases can be found from three different test components.

For the future of MCTS I see following as key points.
* At the end of the day we shall have on unified MCTS applicable for any reference HW.
* Every test case has been described according to test definition [4]
* MCTS will have only one test component for each library
* MCTS has only one Common library providing services for all MCTS components
* All test cases in MCTS have been developed according to guidelines [5] at:
* API coverage has been defined according to defined methods [6] and published with test assets [7].
* Each component has good descriptions in Test Specification [8]
* Each Component has a test plan [9]

Target of this activity is that MeeGo Developers can trust and rely on MCTS results with any hardware

As addition to these I see one more quite comprehensive element and it is freedom of test developers to choose language they want Python, Perl, C++, and so forth). In order to make usage of MCTS easy for anyone it is important to unify the ways test cases are defined, documented and so forth.

[1] MeeGo Core Test Suite - MCTS - http://gitorious.org/qa-tools/mcts

[2] MeeGo 1.2 Core OS Test Plan - http://wiki.meego.com/Quality/TestPlan/MeeGo_Core_Test_Plan

[3] Test Areas describing quality characteristics - http://wiki.meego.com/Quality/TestAreas

[4] Test definition - http://wiki.meego.com/Test_Packaging/Test_Plan

[5] MCTS Development Guideline - http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline

[6] API analysis for functional test case design - http://wiki.meego.com/Quality/TestSuite
MCTS/MCTS_API_analysis)

[7] Coverage analysis spreadsheets - http://gitorious.org/qa-tools/mcts-coverage

[8] Example of test specification of MCTS test suite - http://wiki.meego.com/QualityTestSuiteVideo_Playback_Driver_Test_Specification

[9] Template for test plan - http://wiki.meego.com/CompTestPlanTemplate)

Kindest Regards,

Toropainen

MeeGo Conference 2010/Unconference

2010-11-12T06:53:22Z

Ttoropainen: /* I plan to present ... */

Quality/Plans/Testability-checklist

2010-10-12T05:53:14Z

Ttoropainen: /* Testability Checklist */

For a previous version of the Testability Checklist, see: http://wiki.meego.com/Quality/Testability_checklist_MeeGo_1.1

== Introduction ==
Describing user interaction is a simple way to describe features, thus, they should be simple to write and to understand. By describing user interaction with the features in Bugzilla, features can contain sufficient information to make test development possible without increasing the workload of the feature owner (as much as the [http://wiki.meego.com/Quality/Testability_checklist_MeeGo_1.1 "MeeGo 1.1 Testability Checklist"]). The testability checklist used for MeeGo 1.1 contained items that were related to the process of feature creation which should be self evident and not be included in testability checklist.

== Describing the Feature ==
The most effective way to create testable features from QA point of view is to describe how the user interacts with the feature. The objective of writing such descriptions is to explain the feature to the extent that a test to verify the feature can be written. To that end, after writing a description, ask yourself this question:

'''Can someone write a test using the information in this description?'''

If the answer is 'yes' then the description is complete, if the answer is 'no', then it may be worth it to consider rewriting the it or (if this seems like it will not help) add extra information (see the extra information section).

Some of the use case may not be fully applicable for Core OS QA but in most of the use cases can be divided to smaller particles and can be considered as enablers for the main use case. E.g. Browsing use case can be verified using these smaller particles (enablers) as follows:
* Scan and connect to WLAN Access Point and conduct data transfer
* Download specific file over WLAN with performance measurement (Throughput)

== Structuring the Description ==
There is a good wikipedia article on the concept of the describing a feature from a user's point of view [http://en.wikipedia.org/wiki/User_story here.] While there are some differences from the article because MeeGo is not a small project that is being implemented by one team.

Three basic forms for a description are:
* I as a <user type/role> want to <perform action> and <result/goal>
* "As a <role>, I want <goal/desire> so that <benefit>" (from wikipedia)
* "As a <role>, I want <goal/desire>" (shortened version from wikipedia)

It is important to define the goal clearly. This allows the success criteria of the test to be clearly defined.

It is also important to define the set of actions leading up to the goal clearly. For example, if the user needs to tilt the device from horizontal to vertical for an action to occur, this needs to be described.

=== Dissected Example ===
The following feature can be found [http://bugs.meego.com/show_bug.cgi?id=6724 here] and can be considered a good example of a feature description:

''User is browsing multimedia content in browser and a system event triggers a sound, e.g. battery low or alarm. System sound is mixed in with the browser audio.''

We can see in this example:
* The user action (browsing multimedia content) is defined clearly.
* There is system action (system event triggering a sound).
* The goal is clearly defined ("System sound is mixed in with the browser audio").
From the above description, a test can be written.

== Extra Information ==
The basic description from a user's point of view does not usually include the provision for extra information (things that the user would not see). This is usually not a problem for a single, coherent, project. In a project with as many components (from as many sources) as MeeGo, extra information may be required.

NOTE: Adding extra information is not a requirement, if your feature is adequately described by the description itself, then, stop there.

Some suggestions on what extra information to add:
* Requirements: e.g. binutils needs to be installed for this feature to work.
* Time Limitations: The operation needs to be performed in less than 5 seconds.
* Specific API that should be used (if this is not obvious).

== Testability Checklist ==
Even though Testability Checklist is tool for QA teams, persons creating features in Bugzilla should have a thorough look to this in order to make feature flow fluently. Testability Checklist contains mandatory parts (to fulfill the criteria to have Testability marked as YES) and optional part which is recommended for one creating features to consider seriously if there is such information available that will help QA teams to conduct their work at appropriate level, leading well tested MeeGo features and Releases.

'''Mandatory:'''
* '''Can I write a test based on the information in this description?''' 
** Is the goal defined clearly? 
** Is the procedure required to reach the goal clearly defined?
** Is the description short enough?
** Is there any further information that is easy to put down and may help a test developer?
* '''Are relevant technology parameters available (Sample rates, Bit Rates, supported picture sizes, etc.) described?'''
* '''For Core OS: Is API Under test defined clearly?''' 
'''Optional (and highly recommended):'''
* Is there be reference to API documentation of API(s) implementing features? Location where more information is available? 
* Is there '''Performance''' aspects or requirements (response time, latency, raw throughput) that should be verified concerning this feature? 
* Is there '''Reliability''' aspects or requirements (expected use rate per day, how long feature should survive in stress situation) that should be verified?
* Is there '''Power Management''' aspects or requirements (E.g. local music playback use case should not consume more than XX,XX mW) that should be verified?
* Are there clear targets in terms of performance or reliability concerning this feature?
* Is there significant interactions (e.g. local music playback while browsing) with other features which should be taken into account when designing test cases concerning this feature?

== Conclusion ==
By creating descriptions at user level for each feature, the feature owner can avoid having to write a [http://wiki.meego.com/Quality/Testability_checklist_MeeGo_1.1 "Testability Checklist"] as was required for MeeGo 1.1. Adequately written descriptions should make life easier for both the product owner and the test developer.

Quality/Plans/MeeGo Core Test Plan

2010-09-27T06:11:56Z

Ttoropainen: /* References */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Each component will have detailed test plan for the specific component. Test Plans are available in [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details (each component name is link to test plan).

Test plan shall follow structure described below: :

* 1 Introduction
* 2 Test Coverage
** 2.1 Software Coverage
** 2.2 Hardware Coverage
** 2.3 Items not Covered
* 3 Running the tests
** 3.1 Executing tests
** 3.2 Test case arguments
** 3.3 Test cases
** 3.4 Configuration file
* 4 Module Design and Architecture
** 4.1 Architecture diagram
* 5 References
* 6 Change History

See [http://wiki.meego.com/Quality/TestSuite/Video_Playback_Driver_Test_Specification Video playback driver test plan] for reference.

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

[[File:Coreosqawkcycle.PNG]]

==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionality which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for release content developed under Bugzilla components
* Set of fully automated system level test cases verifying functionality from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, QA will start the full validation testing for it, following test will be involved:

* Component Functional Test – which will follow the Bugzilla component test plan to run all tests.
* System functional Test – which are defined in the individual Bugzilla component test plan.
* System Non-functional Test – which are defined in the individual Bugzilla component test plan.

Purpose of the Full Pass is to measure release maturity in detailed level. In Full Pass testing entire test asset is executed for all the features released and previously marked as verified. Thus visibility and detailed information about release maturity is gained. Target is to have two Full Pass testing cycles during release life cycle. First Pull Pass test round starts at feature complete and last round ends few days prior to release date.

Between these two rounds failing cases and related bugs are followed closely in weekly testing. If there are very good grounds Full Pass testing can be executed more than twice during release cycle life cycle.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== Risks ==

One main expectation from program management is that QA is able to pinpoint risks concerning Quality Risks. For MeeGo 1.2 QA sees following risks:

{|cellspacing="0" border="1"
!|Risk #
!|Description
!|impact
!|Severity
!|Mitigation actions

|-
|| 1 || Insufficient documentation and/or low quality feature descriptions not meeting that testability requirements || Bad quality test asset not verifying feature in detailed level. || '''High''' || Significant effort put to feature testability in early phase of release life cycle. Review test assets with feature owner and assignees.
|-
|| 2 || Insufficient amount of system level test cases to show functionality of the release || Unclear release quality statement. Community confused and developers hunting errors which do not exist|| '''High''' || Significant effort to system level test case development. Each functionality (main success scenario and it's variants tested at least with one set of parameters.
|-
|| 3 || Insufficient amount of system level test cases to show functionality of the release || Unclear release quality statement. Community confused and developers hunting errors which do not exist|| '''High''' || Significant effort to system level test case development. Each functionality (main success scenario and it's variants tested at least with one set of parameters.
|-
|| 4 || Changed components are verified in weekly testing but if dependencies are not understood impact to indirectly affected component not verified. || Surprises in release quality || Medium || Developers to address dependencies in detailed level showing significant dependencies
|}

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release
* MeeGo test area terminology: http://wiki.meego.com/Quality/TestAreas
* Release Engineering Process: http://wiki.meego.com/Release_Engineering/Process
* Release Engineering/Release Timeline: http://wiki.meego.com/Release_Engineering/Release_Timeline
* MCTS Development Guideline: http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline
* QA tools team: http://wiki.meego.com/index.php?title=Quality/TestPlan/MeeGo_Core_Test_Plan&action=edit&section=28
* MeeGo Handset Testing: http://wiki.meego.com/Quality#MeeGo_Handset_Testing
* Testability Checklist: http://wiki.meego.com/Quality/TestabilityChecklist
* Test Design Process And Guideline: http://wiki.meego.com/Quality/TestDesignProcessAndGuideline
* Test Packaging: http://wiki.meego.com/Test_Packaging/Test_Plan
* Common Test Plan Template: http://wiki.meego.com/TestCaseTemplate
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MCTS API analysis: http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis
* Video Playback Driver Test Specification: http://wiki.meego.com/Quality/TestSuite/Video_Playback_Driver_Test_Specification

Quality/Plans/MeeGo Core Test Plan

2010-09-27T06:08:58Z

Ttoropainen: /* References */

Quality/Plans/MeeGo Core Test Plan

2010-09-27T05:52:54Z

Ttoropainen: /* Sanity (Daily) Testing */

Quality/Plans/MeeGo Core Test Plan

2010-09-27T05:31:37Z

Ttoropainen: /* Risks */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Each component will have detailed test plan for the specific component. Test Plans are available in [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details (each component name is link to test plan).

Test plan shall follow structure described below: :

* 1 Introduction
* 2 Test Coverage
** 2.1 Software Coverage
** 2.2 Hardware Coverage
** 2.3 Items not Covered
* 3 Running the tests
** 3.1 Executing tests
** 3.2 Test case arguments
** 3.3 Test cases
** 3.4 Configuration file
* 4 Module Design and Architecture
** 4.1 Architecture diagram
* 5 References
* 6 Change History

See [http://wiki.meego.com/Quality/TestSuite/Video_Playback_Driver_Test_Specification Video playback driver test plan] for reference.

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

[[File:Coreosqawkcycle.PNG]]

==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionality which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for Bugzilla components
* Set of fully automated system level test cases verifying functionality from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, QA will start the full validation testing for it, following test will be involved:

* Component Functional Test – which will follow the Bugzilla component test plan to run all tests.
* System functional Test – which are defined in the individual Bugzilla component test plan.
* System Non-functional Test – which are defined in the individual Bugzilla component test plan.

Purpose of the Full Pass is to measure release maturity in detailed level. In Full Pass testing entire test asset is executed for all the features released and previously marked as verified. Thus visibility and detailed information about release maturity is gained. Target is to have two Full Pass testing cycles during release life cycle. First Pull Pass test round starts at feature complete and last round ends few days prior to release date.

Between these two rounds failing cases and related bugs are followed closely in weekly testing. If there are very good grounds Full Pass testing can be executed more than twice during release cycle life cycle.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== Risks ==

One main expectation from program management is that QA is able to pinpoint risks concerning Quality Risks. For MeeGo 1.2 QA sees following risks:

{|cellspacing="0" border="1"
!|Risk #
!|Description
!|impact
!|Severity
!|Mitigation actions

|-
|| 1 || Insufficient documentation and/or low quality feature descriptions not meeting that testability requirements || Bad quality test asset not verifying feature in detailed level. || '''High''' || Significant effort put to feature testability in early phase of release life cycle. Review test assets with feature owner and assignees.
|-
|| 2 || Insufficient amount of system level test cases to show functionality of the release || Unclear release quality statement. Community confused and developers hunting errors which do not exist|| '''High''' || Significant effort to system level test case development. Each functionality (main success scenario and it's variants tested at least with one set of parameters.
|-
|| 3 || Insufficient amount of system level test cases to show functionality of the release || Unclear release quality statement. Community confused and developers hunting errors which do not exist|| '''High''' || Significant effort to system level test case development. Each functionality (main success scenario and it's variants tested at least with one set of parameters.
|-
|| 4 || Changed components are verified in weekly testing but if dependencies are not understood impact to indirectly affected component not verified. || Surprises in release quality || Medium || Developers to address dependencies in detailed level showing significant dependencies
|}

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T05:31:11Z

Ttoropainen: /* Risks */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Each component will have detailed test plan for the specific component. Test Plans are available in [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details (each component name is link to test plan).

Test plan shall follow structure described below: :

* 1 Introduction
* 2 Test Coverage
** 2.1 Software Coverage
** 2.2 Hardware Coverage
** 2.3 Items not Covered
* 3 Running the tests
** 3.1 Executing tests
** 3.2 Test case arguments
** 3.3 Test cases
** 3.4 Configuration file
* 4 Module Design and Architecture
** 4.1 Architecture diagram
* 5 References
* 6 Change History

See [http://wiki.meego.com/Quality/TestSuite/Video_Playback_Driver_Test_Specification Video playback driver test plan] for reference.

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

[[File:Coreosqawkcycle.PNG]]

==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionality which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for Bugzilla components
* Set of fully automated system level test cases verifying functionality from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, QA will start the full validation testing for it, following test will be involved:

* Component Functional Test – which will follow the Bugzilla component test plan to run all tests.
* System functional Test – which are defined in the individual Bugzilla component test plan.
* System Non-functional Test – which are defined in the individual Bugzilla component test plan.

Purpose of the Full Pass is to measure release maturity in detailed level. In Full Pass testing entire test asset is executed for all the features released and previously marked as verified. Thus visibility and detailed information about release maturity is gained. Target is to have two Full Pass testing cycles during release life cycle. First Pull Pass test round starts at feature complete and last round ends few days prior to release date.

Between these two rounds failing cases and related bugs are followed closely in weekly testing. If there are very good grounds Full Pass testing can be executed more than twice during release cycle life cycle.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== Risks ==

One main expectation from program management is that QA is able to pinpoint risks concerning Quality Risks. For MeeGo 1.2 QA sees following risks:

{|cellspacing="0" border="1"
!|Risk #
!|Description
!|impact
!|Severity
!|Mitigation actions

|-
|| 1 || Insufficient documentation and/or low quality feature descriptions not meeting that testability requirements || Bad quality test asset not verifying feature in detailed level. || '''High''' || Significant effort put to feature testability in early phase of release life cycle. Review test assets with feature owner and assignees.

|-
|| 2 || Insufficient amount of system level test cases to show functionality of the release || Unclear release quality statement. Community confused and developers hunting errors which do not exist|| '''High''' || Significant effort to system level test case development. Each functionality (main success scenario and it's variants tested at least with one set of parameters.

|-
|| 3 || Insufficient amount of system level test cases to show functionality of the release || Unclear release quality statement. Community confused and developers hunting errors which do not exist|| '''High''' || Significant effort to system level test case development. Each functionality (main success scenario and it's variants tested at least with one set of parameters.

|-
|| 4 || Changed components are verified in weekly testing but if dependencies are not understood impact to indirectly affected component not verified. || Surprises in release quality || Medium || Developers to address dependencies in detailed level showing significant dependencies

|}

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T05:29:11Z

Ttoropainen: /* Risks */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Each component will have detailed test plan for the specific component. Test Plans are available in [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details (each component name is link to test plan).

Test plan shall follow structure described below: :

* 1 Introduction
* 2 Test Coverage
** 2.1 Software Coverage
** 2.2 Hardware Coverage
** 2.3 Items not Covered
* 3 Running the tests
** 3.1 Executing tests
** 3.2 Test case arguments
** 3.3 Test cases
** 3.4 Configuration file
* 4 Module Design and Architecture
** 4.1 Architecture diagram
* 5 References
* 6 Change History

See [http://wiki.meego.com/Quality/TestSuite/Video_Playback_Driver_Test_Specification Video playback driver test plan] for reference.

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

[[File:Coreosqawkcycle.PNG]]

==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionality which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for Bugzilla components
* Set of fully automated system level test cases verifying functionality from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, QA will start the full validation testing for it, following test will be involved:

* Component Functional Test – which will follow the Bugzilla component test plan to run all tests.
* System functional Test – which are defined in the individual Bugzilla component test plan.
* System Non-functional Test – which are defined in the individual Bugzilla component test plan.

Purpose of the Full Pass is to measure release maturity in detailed level. In Full Pass testing entire test asset is executed for all the features released and previously marked as verified. Thus visibility and detailed information about release maturity is gained. Target is to have two Full Pass testing cycles during release life cycle. First Pull Pass test round starts at feature complete and last round ends few days prior to release date.

Between these two rounds failing cases and related bugs are followed closely in weekly testing. If there are very good grounds Full Pass testing can be executed more than twice during release cycle life cycle.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== Risks ==

One main expectation from program management is that QA is able to pinpoint risks concerning Quality Risks. For MeeGo 1.2 QA sees following risks:

{|cellspacing="0" border="1"
!|Risk #
!|Description
!|impact
!|Severity
!|Mitigation actions

|-
|| 1 || Insufficient documentation and/or low quality feature descriptions not meeting that testability requirements || Bad quality test asset not verifying feature in detailed level. || '''High''' || Significant effort put to feature testability in early phase of release lifecycle. Review test assets with feature owner and assignees.

|-
|| 2 || Insufficient amount of system level test cases to show functionality of the release || Unclear release quality statement. Community confused and developers hunting errors which do not exist|| '''High''' || Significant effort to system level test case development. Each functionality (main success scenario and it's variants tested at least with one set of parameters.

|-
|| 3 || Insufficient amount of system level test cases to show functionality of the release || Unclear release quality statement. Community confused and developers hunting errors which do not exist|| '''High''' || Significant effort to system level test case development. Each functionality (main success scenario and it's variants tested at least with one set of parameters.

|-
|| 4 || Changed components are verified in weekly testing but if dependencies are not understood impact to indirectly affected component not verified. || Surprises in release quality || Medium || Developers to address dependencies in detailed level showing significant dependencies

|}

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T05:09:23Z

Ttoropainen: /* References */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Each component will have detailed test plan for the specific component. Test Plans are available in [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details (each component name is link to test plan).

Test plan shall follow structure described below: :

* 1 Introduction
* 2 Test Coverage
** 2.1 Software Coverage
** 2.2 Hardware Coverage
** 2.3 Items not Covered
* 3 Running the tests
** 3.1 Executing tests
** 3.2 Test case arguments
** 3.3 Test cases
** 3.4 Configuration file
* 4 Module Design and Architecture
** 4.1 Architecture diagram
* 5 References
* 6 Change History

See [http://wiki.meego.com/Quality/TestSuite/Video_Playback_Driver_Test_Specification Video playback driver test plan] for reference.

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

[[File:Coreosqawkcycle.PNG]]

==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionality which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for Bugzilla components
* Set of fully automated system level test cases verifying functionality from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, QA will start the full validation testing for it, following test will be involved:

* Component Functional Test – which will follow the Bugzilla component test plan to run all tests.
* System functional Test – which are defined in the individual Bugzilla component test plan.
* System Non-functional Test – which are defined in the individual Bugzilla component test plan.

Purpose of the Full Pass is to measure release maturity in detailed level. In Full Pass testing entire test asset is executed for all the features released and previously marked as verified. Thus visibility and detailed information about release maturity is gained. Target is to have two Full Pass testing cycles during release life cycle. First Pull Pass test round starts at feature complete and last round ends few days prior to release date.

Between these two rounds failing cases and related bugs are followed closely in weekly testing. If there are very good grounds Full Pass testing can be executed more than twice during release cycle life cycle.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== Risks ==

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T05:08:45Z

Ttoropainen: /* Component Test Plans and Coverage */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Each component will have detailed test plan for the specific component. Test Plans are available in [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details (each component name is link to test plan).

Test plan shall follow structure described below: :

* 1 Introduction
* 2 Test Coverage
** 2.1 Software Coverage
** 2.2 Hardware Coverage
** 2.3 Items not Covered
* 3 Running the tests
** 3.1 Executing tests
** 3.2 Test case arguments
** 3.3 Test cases
** 3.4 Configuration file
* 4 Module Design and Architecture
** 4.1 Architecture diagram
* 5 References
* 6 Change History

See [http://wiki.meego.com/Quality/TestSuite/Video_Playback_Driver_Test_Specification Video playback driver test plan] for reference.

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

[[File:Coreosqawkcycle.PNG]]

==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionality which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for Bugzilla components
* Set of fully automated system level test cases verifying functionality from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, QA will start the full validation testing for it, following test will be involved:

* Component Functional Test – which will follow the Bugzilla component test plan to run all tests.
* System functional Test – which are defined in the individual Bugzilla component test plan.
* System Non-functional Test – which are defined in the individual Bugzilla component test plan.

Purpose of the Full Pass is to measure release maturity in detailed level. In Full Pass testing entire test asset is executed for all the features released and previously marked as verified. Thus visibility and detailed information about release maturity is gained. Target is to have two Full Pass testing cycles during release life cycle. First Pull Pass test round starts at feature complete and last round ends few days prior to release date.

Between these two rounds failing cases and related bugs are followed closely in weekly testing. If there are very good grounds Full Pass testing can be executed more than twice during release cycle life cycle.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T05:08:01Z

Ttoropainen: /* Component Test Plans and Coverage */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Each component will have detailed test plan for the specific component. Test Plans are available in [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details (each component name is link to test plan).

Test plan shall follow structure described below: :

* 1 Introduction
* 2 Test Coverage
** 2.1 Software Coverage
** 2.2 Hardware Coverage
* 3 Running the tests
** 3.1 Executing tests
** 3.2 Test case arguments
** 3.3 Test cases
** 3.4 Configuration file
* 4 Module Design and Architecture
** 4.1 Architecture diagram
* 5 References
* 6 Change History

See [http://wiki.meego.com/Quality/TestSuite/Video_Playback_Driver_Test_Specification Video playback driver test plan] for reference.

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

[[File:Coreosqawkcycle.PNG]]

==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionality which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for Bugzilla components
* Set of fully automated system level test cases verifying functionality from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, QA will start the full validation testing for it, following test will be involved:

* Component Functional Test – which will follow the Bugzilla component test plan to run all tests.
* System functional Test – which are defined in the individual Bugzilla component test plan.
* System Non-functional Test – which are defined in the individual Bugzilla component test plan.

Purpose of the Full Pass is to measure release maturity in detailed level. In Full Pass testing entire test asset is executed for all the features released and previously marked as verified. Thus visibility and detailed information about release maturity is gained. Target is to have two Full Pass testing cycles during release life cycle. First Pull Pass test round starts at feature complete and last round ends few days prior to release date.

Between these two rounds failing cases and related bugs are followed closely in weekly testing. If there are very good grounds Full Pass testing can be executed more than twice during release cycle life cycle.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T05:03:34Z

Ttoropainen: /* Component Test Plans and Coverage */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Each component will have detailed test plan for the specific component. Test Plans are available in [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details (each )

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

[[File:Coreosqawkcycle.PNG]]

==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionality which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for Bugzilla components
* Set of fully automated system level test cases verifying functionality from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, QA will start the full validation testing for it, following test will be involved:

* Component Functional Test – which will follow the Bugzilla component test plan to run all tests.
* System functional Test – which are defined in the individual Bugzilla component test plan.
* System Non-functional Test – which are defined in the individual Bugzilla component test plan.

Purpose of the Full Pass is to measure release maturity in detailed level. In Full Pass testing entire test asset is executed for all the features released and previously marked as verified. Thus visibility and detailed information about release maturity is gained. Target is to have two Full Pass testing cycles during release life cycle. First Pull Pass test round starts at feature complete and last round ends few days prior to release date.

Between these two rounds failing cases and related bugs are followed closely in weekly testing. If there are very good grounds Full Pass testing can be executed more than twice during release cycle life cycle.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T05:02:20Z

Ttoropainen: /* Full Pass (Milestone) Testing */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

[[File:Coreosqawkcycle.PNG]]

==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionality which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for Bugzilla components
* Set of fully automated system level test cases verifying functionality from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, QA will start the full validation testing for it, following test will be involved:

* Component Functional Test – which will follow the Bugzilla component test plan to run all tests.
* System functional Test – which are defined in the individual Bugzilla component test plan.
* System Non-functional Test – which are defined in the individual Bugzilla component test plan.

Purpose of the Full Pass is to measure release maturity in detailed level. In Full Pass testing entire test asset is executed for all the features released and previously marked as verified. Thus visibility and detailed information about release maturity is gained. Target is to have two Full Pass testing cycles during release life cycle. First Pull Pass test round starts at feature complete and last round ends few days prior to release date.

Between these two rounds failing cases and related bugs are followed closely in weekly testing. If there are very good grounds Full Pass testing can be executed more than twice during release cycle life cycle.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T04:42:55Z

Ttoropainen: /* Sanity (Daily) Testing */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

[[File:Coreosqawkcycle.PNG]]

==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionality which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for Bugzilla components
* Set of fully automated system level test cases verifying functionality from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T04:41:25Z

Ttoropainen: /* Test Cycle */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

[[File:Coreosqawkcycle.PNG]]

==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionality which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionality from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T04:38:24Z

Ttoropainen: /* Test Cycle */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

[[File:Coreosqawkcycle.png]]

==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionality which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionality from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

File:Coreosqawkcycle.PNG

2010-09-27T04:37:17Z

Ttoropainen:

Quality/Plans/MeeGo Core Test Plan

2010-09-27T04:26:50Z

Ttoropainen: /* Test Cycle */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

[[File:coreosqawkcycle.png]]

==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionality which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionality from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T04:24:34Z

Ttoropainen: /* Development teams and MeeGo Core OS QA */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T04:23:57Z

Ttoropainen: /* Development teams and MeeGo Core OS QA */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results)'
* Provide test asset with the component including detailed documentation
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T04:22:17Z

Ttoropainen: /* What will not be tested */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Provide test asset with the component including detailed documentation
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T04:21:36Z

Ttoropainen: /* Testability */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Provide test asset with the component including detailed documentation
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T04:18:59Z

Ttoropainen: /* Development teams and MeeGo Core OS QA */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Provide test asset with the component including detailed documentation
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T04:17:08Z

Ttoropainen: /* Development teams and MeeGo Core OS QA */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Provide test asset with the component
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T04:10:57Z

Ttoropainen:

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas]

=== Development teams and MeeGo Core OS QA ===

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T04:04:43Z

Ttoropainen: /* Objectives */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines].

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas article]

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T04:04:07Z

Ttoropainen: /* Objectives */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in MCTS Quality Guidlines.

For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas article]

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T04:00:14Z

Ttoropainen: /* Full Pass (Milestone) Testing */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

*New features introduced in specific Core OS weekly release are working as specified as a part of system.
*Stakeholders and community get visibility to release quality.
*Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

*Delivered features for MeeGo Core OS are working as specified as a part of system.
*MeeGo Core OS is performing well and is reliable.
*Program maturity statement can be given.
For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas article]

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-27T03:59:25Z

Ttoropainen: /* Objectives */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

*New features introduced in specific Core OS weekly release are working as specified as a part of system.
*Stakeholders and community get visibility to release quality.
*Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

*Delivered features for MeeGo Core OS are working as specified as a part of system.
*MeeGo Core OS is performing well and is reliable.
*Program maturity statement can be given.
For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas article]

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the BAT test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-24T05:25:49Z

Ttoropainen: /* References */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

*New features introduced in specific Core OS weekly release are working as specified as a part of system.
*Stakeholders and community get visibility to release quality.
*Validate that relevant bugs are fixed in software release.

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

*Delivered features for MeeGo Core OS are working as specified as a part of system.
*MeeGo Core OS is performing well and is reliable.
*Program maturity statement can be given.
For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas article]

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the BAT test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-24T05:25:04Z

Ttoropainen: /* Test enviromental requirements */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

*New features introduced in specific Core OS weekly release are working as specified as a part of system.
*Stakeholders and community get visibility to release quality.
*Validate that relevant bugs are fixed in software release.

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

*Delivered features for MeeGo Core OS are working as specified as a part of system.
*MeeGo Core OS is performing well and is reliable.
*Program maturity statement can be given.
For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas article]

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the BAT test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
' MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-24T04:03:36Z

Ttoropainen: /* References */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

*New features introduced in specific Core OS weekly release are working as specified as a part of system.
*Stakeholders and community get visibility to release quality.
*Validate that relevant bugs are fixed in software release.

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

*Delivered features for MeeGo Core OS are working as specified as a part of system.
*MeeGo Core OS is performing well and is reliable.
*Program maturity statement can be given.
For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas article]

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the BAT test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
' MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release

Quality/Plans/MeeGo Core Test Plan

2010-09-23T11:30:06Z

Ttoropainen: /* Hardware Platforms */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

*New features introduced in specific Core OS weekly release are working as specified as a part of system.
*Stakeholders and community get visibility to release quality.
*Validate that relevant bugs are fixed in software release.

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

*Delivered features for MeeGo Core OS are working as specified as a part of system.
*MeeGo Core OS is performing well and is reliable.
*Program maturity statement can be given.
For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas article]

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the BAT test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com

Quality/Plans/MeeGo Core Test Plan

2010-09-23T11:29:29Z

Ttoropainen: /* Test Coverage Measurement */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

*New features introduced in specific Core OS weekly release are working as specified as a part of system.
*Stakeholders and community get visibility to release quality.
*Validate that relevant bugs are fixed in software release.

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

*Delivered features for MeeGo Core OS are working as specified as a part of system.
*MeeGo Core OS is performing well and is reliable.
*Program maturity statement can be given.
For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas article]

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis MCTS API analysis] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the BAT test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
'''MeeGo v1.1'''
* Aava Koski
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com

Quality/Plans/MeeGo Core Test Plan

2010-09-23T11:29:00Z

Ttoropainen: /* Test cases */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

*New features introduced in specific Core OS weekly release are working as specified as a part of system.
*Stakeholders and community get visibility to release quality.
*Validate that relevant bugs are fixed in software release.

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

*Delivered features for MeeGo Core OS are working as specified as a part of system.
*MeeGo Core OS is performing well and is reliable.
*Program maturity statement can be given.
For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas article]

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [http://wiki.meego.com/Quality/TestDesignProcessAndGuideline MeeGo QA Common Test Design Process and Guidelines]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [http://wiki.meego.com/Test_Packaging/Test_Plan test definition] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[http://wiki.meego.com/TestCaseTemplate Common Test Case Template] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

MCTS API analysis (http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis) describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the BAT test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
'''MeeGo v1.1'''
* Aava Koski
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com

Quality/Plans/MeeGo Core Test Plan

2010-09-23T11:26:15Z

Ttoropainen: /* Quality requirements for MCTS */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

*New features introduced in specific Core OS weekly release are working as specified as a part of system.
*Stakeholders and community get visibility to release quality.
*Validate that relevant bugs are fixed in software release.

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

*Delivered features for MeeGo Core OS are working as specified as a part of system.
*MeeGo Core OS is performing well and is reliable.
*Program maturity statement can be given.
For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas article]

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline MCTS Development Guidelines]

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

Test Design process follows spirit of process described in http://wiki.meego.com/Quality/TestDesignProcessAndGuideline. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to test definition (http://wiki.meego.com/Test_Packaging/Test_Plan) in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*Common Test Case Template (http://wiki.meego.com/TestCaseTemplate) is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

MCTS API analysis (http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis) describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the BAT test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
'''MeeGo v1.1'''
* Aava Koski
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com

Quality/Plans/MeeGo Core Test Plan

2010-09-23T11:25:40Z

Ttoropainen: /* Introduction */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

*New features introduced in specific Core OS weekly release are working as specified as a part of system.
*Stakeholders and community get visibility to release quality.
*Validate that relevant bugs are fixed in software release.

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

*Delivered features for MeeGo Core OS are working as specified as a part of system.
*MeeGo Core OS is performing well and is reliable.
*Program maturity statement can be given.
For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas article]

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described in: http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

Test Design process follows spirit of process described in http://wiki.meego.com/Quality/TestDesignProcessAndGuideline. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to test definition (http://wiki.meego.com/Test_Packaging/Test_Plan) in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*Common Test Case Template (http://wiki.meego.com/TestCaseTemplate) is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

MCTS API analysis (http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis) describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the BAT test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
'''MeeGo v1.1'''
* Aava Koski
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com

Quality/Plans/MeeGo Core Test Plan

2010-09-23T11:25:17Z

Ttoropainen: /* Introduction */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

*New features introduced in specific Core OS weekly release are working as specified as a part of system.
*Stakeholders and community get visibility to release quality.
*Validate that relevant bugs are fixed in software release.

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

*Delivered features for MeeGo Core OS are working as specified as a part of system.
*MeeGo Core OS is performing well and is reliable.
*Program maturity statement can be given.
For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [ http://wiki.meego.com/Quality#QA_tools_team MeeGo QA Tools Team].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [http://wiki.meego.com/Quality#MeeGo_Handset_Testing Handset UX Test Plan].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [http://wiki.meego.com/Quality/TestAreas test areas article]

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described in: http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

Test Design process follows spirit of process described in http://wiki.meego.com/Quality/TestDesignProcessAndGuideline. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to test definition (http://wiki.meego.com/Test_Packaging/Test_Plan) in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*Common Test Case Template (http://wiki.meego.com/TestCaseTemplate) is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

MCTS API analysis (http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis) describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the BAT test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
'''MeeGo v1.1'''
* Aava Koski
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com

Quality/Plans/MeeGo Core Test Plan

2010-09-23T11:23:35Z

Ttoropainen: /* Objectives */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

*New features introduced in specific Core OS weekly release are working as specified as a part of system.
*Stakeholders and community get visibility to release quality.
*Validate that relevant bugs are fixed in software release.

For these activities MeeGo Core OS QA follows iteration cycle and process described in [http://wiki.meego.com/Release_Engineering/Process Release Engineering’s Process]

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

*Delivered features for MeeGo Core OS are working as specified as a part of system.
*MeeGo Core OS is performing well and is reliable.
*Program maturity statement can be given.
For these activities MeeGo Core OS QA follows release cycle described in [http://wiki.meego.com/Release_Engineering/Release_Timeline Release Engineering’s release timeline]

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by MeeGo QA Tools Team (http://wiki.meego.com/Quality#QA_tools_team).

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see Handset UX Test Plan at: http://wiki.meego.com/Quality#MeeGo_Handset_Testing

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in details at: (http://wiki.meego.com/Quality/TestAreas )

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described in: http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

Test Design process follows spirit of process described in http://wiki.meego.com/Quality/TestDesignProcessAndGuideline. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to test definition (http://wiki.meego.com/Test_Packaging/Test_Plan) in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*Common Test Case Template (http://wiki.meego.com/TestCaseTemplate) is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

MCTS API analysis (http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis) describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the BAT test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
'''MeeGo v1.1'''
* Aava Koski
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com

Quality/Plans/MeeGo Core Test Plan

2010-09-23T11:20:46Z

Ttoropainen: /* What will not Be Tested */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

*New features introduced in specific Core OS weekly release are working as specified as a part of system.
*Stakeholders and community get visibility to release quality.
*Validate that relevant bugs are fixed in software release.

For these activities MeeGo Core OS QA follows iteration cycle and process described in Release Engineering’s release timeline at: http://wiki.meego.com/Release_Engineering/Process

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

*Delivered features for MeeGo Core OS are working as specified as a part of system.
*MeeGo Core OS is performing well and is reliable.
*Program maturity statement can be given.
For these activities MeeGo Core OS QA follows release cycle described in Release Engineering’s release timeline at: http://wiki.meego.com/Release_Engineering/Release_Timeline

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by MeeGo QA Tools Team (http://wiki.meego.com/Quality#QA_tools_team).

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see Handset UX Test Plan at: http://wiki.meego.com/Quality#MeeGo_Handset_Testing

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in details at: (http://wiki.meego.com/Quality/TestAreas )

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described in: http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

*Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
*Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
*QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links
*http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
*http://wiki.meego.com/Quality/TestabilityChecklist

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

Test Design process follows spirit of process described in http://wiki.meego.com/Quality/TestDesignProcessAndGuideline. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to test definition (http://wiki.meego.com/Test_Packaging/Test_Plan) in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*Common Test Case Template (http://wiki.meego.com/TestCaseTemplate) is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

MCTS API analysis (http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis) describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Go to [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.
==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for core components
* Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the BAT test, test team will start the full validation testing for it, following test will be involved:
* Functional Test – which will follow the component test plan to run all tests.
* Non Functional Test – which are defined in the individual component test plan.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://wiki.meego.com/Quality/CoreTestReports MeeGo Quality] wiki

== Environmental Needs ==
=== Hardware Platforms ===
'''MeeGo v1.1'''
* Aava Koski
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com

Quality/Plans/MeeGo Core Test Plan

2010-09-23T11:14:44Z

Ttoropainen: /* Test cases */

Quality/Plans/MeeGo Core Test Plan

2010-09-23T11:13:31Z

Ttoropainen: /* Testability */

Quality/Plans/MeeGo Core Test Plan

2010-09-23T11:12:34Z

Ttoropainen: /* Feature test and System test */

Quality/Plans/MeeGo Core Test Plan

2010-09-23T11:12:17Z

Ttoropainen: /* Feature test and System test */

Quality/Plans/MeeGo Core Test Plan

2010-09-23T11:10:57Z

Ttoropainen: /* Quality requirements for MCTS */

Quality/Plans/MeeGo Core Test Plan

2010-09-23T11:10:13Z

Ttoropainen: /* Test strategy and approach */