Quality/TestSetGuideline

Purpose

The test set guideline could be referred to identify test cases which will be run in different MeeGo test cycle (hourly, daily, weekly, milestone release).

Test Set vs. Test Cycle & Repo

Test Set Definition

• Sanity Test Set

Sanity test set is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in sanity test reflects the most important and basic functionalities of the distribution.

Sanity test set is relatively stable and will be run on daily basis.

• Regression Test Set

Regression test set is conducted to find out what works previously still work in this week release. Regression test set will be selected from MeeGo test suite based on changes in the image between last week and this week.

Regression test set should run once a week against weekly pre-release image.

Steps to define regression test set:

1. Figure out packages/features which have been changed between the image from last week and this week
2. Identify all the related test cases for those affected packages and features. All the auto core test cases could be included in the regression test set since the execution cost is low. (need implement the mechanism in testlink).
3. Auto regression test cases will be run in OTS. Long term goal is to run all the Core auto tests on all the MeeGo devices (Handset, Netbook etc...) automatically in OTS frequently.
4. Manual tests will be run manually on target device

• Feature Test Set

MeeGo test cases will be arranged following features, so it's nature to create feature test set which will be run in weekly testing. When a new feature is initially integrated, we could selectively run e.g. 30% tests for this feature to expose most important bugs. After the feature is fully integrated, we could select to run through all the tests for this feature and report out all the bugs for this feature. The feature test set will be run again in the upcoming milestone or heavily change about this feature.

The new feature test set will be created every week to align with feature integration plan and the test set will be executed in weekly testing cycle.

• System Functional Test

Sysetm functional test aims to evaluate the distribution satisfying the usage model of particular segment. It's designed to test feature interaction following usage scenario. It's suggested to run against official weekly release. System functional test result provides a full view of the distribution capability to end user. It also been referred as system use case test.

• System Performance Test

System performance test belongs to non-functional system test which test the performance of the distribution

• System Reliability Test

System Reliability test belongs to non-functional system test which test the reliablity of system. such as if system could play media for 12 hours, browse web for 12 hours or longer, or continous operation on the system

• Milestone Test Set

To ensure MeeGo milestone (DP, FC, RC and PR)release with acceptance criteria, we would identify test set which should be run before milestone release. The objective is to report out more bugs and close outstanding bugs in each milestone release. The milestone test set will be selected from the above test sets to ensure MeeGo milestone release goal will be achieved. Details TBD.