Meego Wiki
Views

Quality/TestSetGuideline

From MeeGo wiki
< Quality(Difference between revisions)
Jump to: navigation, search
(→Test Set vs. Test Cycle & Repo)
m (→Test Set Definition)
 
(22 intermediate revisions not shown)
Line 1: Line 1:
-
== Test Set Guideline ==
+
== Purpose ==
The test set guideline could be referred to identify test cases which will be run in different MeeGo test cycle (hourly, daily, weekly, milestone release).
The test set guideline could be referred to identify test cases which will be run in different MeeGo test cycle (hourly, daily, weekly, milestone release).
== Test Set vs. Test Cycle & Repo ==
== Test Set vs. Test Cycle & Repo ==
-
{| class="wikitable" border="1" cellpadding="5"
+
[[File:testset-1.jpg]]
-
|-
+
-
{| class="wikitable" border="1" cellpadding="5"
+
-
! '''Test Set''' || Hourly || Daily || Weekly || DP || FC || RC || PR||
+
-
|-
+
-
| Sanity Test Set||  || X ||  ||  ||  ||  ||  ||
+
-
|-
+
-
| Regression Test Set || X (Auto) || || X (Manual) || || || ||
+
-
|-
+
-
| Feature Test Set ||  || || X || || || ||
+
-
|-
+
-
| Use Case Test Set ||  || || X || || || ||
+
-
|-
+
-
| Bug Verification ||  || || X || || || ||
+
-
|-
+
-
| Milestone Test Set || || || || X || X || X || X ||
+
-
|}
+
== Test Set Definition ==
== Test Set Definition ==
* Sanity Test Set
* Sanity Test Set
-
Sanity test set is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in sanity test reflects the most important and basic functionalities of the distribution.
+
** Sanity test set is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in sanity test reflects the most important and basic functionality of the distribution.
 +
** Sanity test set is relatively stable and will be run on daily basis. [[../daily build test|More...]]
-
Sanity test set is relatively stable and will be run on daily basis.
+
* Feature Testing
 +
** Key Feature Test Set is used to verify MeeGo Handset UX most critical key use cases functionalities with well selected basic feature test cases.
 +
** Basic Feature Test Set is verifying MeeGo HandSet UX delivered features with basic feature test cases. Test set is always static to show overall feature functionalities progress and maturity of the entire MeeGo distribution. Based on test results QA is able to identify components with working features to enable extended feature testing and system testing.  
-
* Regression Test Set
+
* Dataflow
-
Regression test set is conducted to find out what works previously still work in this week release. Regression test set will be selected from MeeGo test suite based on changes in the image between last week and this week.
+
** Dataflow is the flow of data to, from and within a device. Inputs are derived from system use cases and each core dataflow test case presents the basic instance of the use case in MeeGo Core stack. The result of Core dataflow could report the maturity level of MeeGo Core OS software and hardware integration on specific device. It’s a weekly testing cycle against weekly build. [[../dataflow|More...]]
-
Regression test set should run once a week against weekly pre-release image.
+
* Extended Feature Test Set is used to verify delivery of features forming full functionality of entirely component. After component is fully integrated all component related test cases will be executed for selected weekly release and report out all the bugs against component and it’s features. Extended feature test set will be run again in the upcoming milestone or when significant changes are applied to component and it’s features.  
-
Steps to define regression test set:
+
* System Functional
-
** Figure out packages/features which have been changed between the image from last week and this week
+
** System Functional Test Set is targeting to evaluate delivered functionalities from system perspective. Test cases are not testing UI or Application itself, instead test cases are testing how whole system is working and interacting with Consumer (end user). Test cases are covering most critical interaction and negative scenarios that consumers will encounter in their daily usage.  
-
** Identify all the related test cases for those affected packages and features. All the auto core test cases could be included in the regression test set since the execution cost is low. (need implement the mechanism in testlink).
+
-
** Auto regression test cases will be run in OTS. Long term goal is to run all the Core auto tests on all the MeeGo devices (Handset, Netbook etc...) automatically in OTS frequently.
+
-
** Manual tests will be run manually on target device
+
-
* Feature Test Set
+
* System Non-Functional
-
MeeGo test cases will be arranged following features, so it's nature to create feature test set which will be run in weekly testing. When a new feature is initially integrated, we could selectively run e.g. 30% tests for this feature to expose most important bugs. After the feature is fully integrated, we could select to run through all the tests for this feature and report out all the bugs for this feature. The feature test set will be run again in the upcoming milestone or heavily change about this feature.
+
** System Performance Test Sets target is to evaluate overall system performance by executing well-selected  set of cases from different test areas - for example response  and reaction times, use times and frame rate measurements. Test set gives a quick view of system performance from end-user point of view.
 +
** System Reliability Test Sets target is to provide an overview to system reliability by executing iterative tests that focus on the most important and most used end-user features of MeeGo.
-
The new feature test set will be created every week to align with feature integration plan and the test set will be executed in weekly testing cycle.
+
* Regression Test Set
 +
** Regression test set is conducted to find out what works previously still work in this week release. Regression test set will be selected from MeeGo test suite based on changes in the image between last week and this week.
 +
** Regression test set should run once a week against weekly pre-release image.
-
* Use Case Test Set
+
Steps to define regression test set:
-
Use case test set aims to evaluate the distribution satisfying the usage of particular segment. Use case is suggested to run against official weekly release. Use Case result provides a full view of the distribution capability to end user.
+
# Figure out packages/features which have been changed between the image from last week and this week
-
 
+
# Identify all the related test cases for those affected packages and features. All the auto core test cases could be included in the regression test set since the execution cost is low. (need implement the mechanism in testlink).
-
* Bug verification Test Set
+
# Auto regression test cases will be run in OTS. Long term goal is to run all the Core auto tests on all the MeeGo devices (Handset, Netbook etc...) automatically in OTS frequently.
-
Ideally we would create a bug list which need be verified this week and generate test set to verify bugs.
+
# Manual tests will be run manually on target device
* Milestone Test Set
* Milestone Test Set
-
To ensure MeeGo milestone (DP, FC, RC and PR)release with acceptance criteria, we would identify test set which should be run before milestone release. The objective is to report out more bugs and close outstanding bugs in each milestone release. The milestone test set will be selected to ensure MeeGo milestone release goal will be achieved. Details TBD.
+
** To ensure MeeGo milestone (DP, FC, RC and PR)release with acceptance criteria, we would identify test set which should be run before milestone release. The objective is to report out more bugs and close outstanding bugs in each milestone release. The milestone test set will be selected from the above test sets to ensure MeeGo milestone release goal will be achieved. Details TBD.

Latest revision as of 07:40, 27 June 2011

Purpose

The test set guideline could be referred to identify test cases which will be run in different MeeGo test cycle (hourly, daily, weekly, milestone release).

Test Set vs. Test Cycle & Repo

Testset-1.jpg

Test Set Definition

  • Sanity Test Set
    • Sanity test set is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in sanity test reflects the most important and basic functionality of the distribution.
    • Sanity test set is relatively stable and will be run on daily basis. More...
  • Feature Testing
    • Key Feature Test Set is used to verify MeeGo Handset UX most critical key use cases functionalities with well selected basic feature test cases.
    • Basic Feature Test Set is verifying MeeGo HandSet UX delivered features with basic feature test cases. Test set is always static to show overall feature functionalities progress and maturity of the entire MeeGo distribution. Based on test results QA is able to identify components with working features to enable extended feature testing and system testing.
  • Dataflow
    • Dataflow is the flow of data to, from and within a device. Inputs are derived from system use cases and each core dataflow test case presents the basic instance of the use case in MeeGo Core stack. The result of Core dataflow could report the maturity level of MeeGo Core OS software and hardware integration on specific device. It’s a weekly testing cycle against weekly build. More...
  • Extended Feature Test Set is used to verify delivery of features forming full functionality of entirely component. After component is fully integrated all component related test cases will be executed for selected weekly release and report out all the bugs against component and it’s features. Extended feature test set will be run again in the upcoming milestone or when significant changes are applied to component and it’s features.
  • System Functional
    • System Functional Test Set is targeting to evaluate delivered functionalities from system perspective. Test cases are not testing UI or Application itself, instead test cases are testing how whole system is working and interacting with Consumer (end user). Test cases are covering most critical interaction and negative scenarios that consumers will encounter in their daily usage.
  • System Non-Functional
    • System Performance Test Sets target is to evaluate overall system performance by executing well-selected set of cases from different test areas - for example response and reaction times, use times and frame rate measurements. Test set gives a quick view of system performance from end-user point of view.
    • System Reliability Test Sets target is to provide an overview to system reliability by executing iterative tests that focus on the most important and most used end-user features of MeeGo.
  • Regression Test Set
    • Regression test set is conducted to find out what works previously still work in this week release. Regression test set will be selected from MeeGo test suite based on changes in the image between last week and this week.
    • Regression test set should run once a week against weekly pre-release image.

Steps to define regression test set:

  1. Figure out packages/features which have been changed between the image from last week and this week
  2. Identify all the related test cases for those affected packages and features. All the auto core test cases could be included in the regression test set since the execution cost is low. (need implement the mechanism in testlink).
  3. Auto regression test cases will be run in OTS. Long term goal is to run all the Core auto tests on all the MeeGo devices (Handset, Netbook etc...) automatically in OTS frequently.
  4. Manual tests will be run manually on target device
  • Milestone Test Set
    • To ensure MeeGo milestone (DP, FC, RC and PR)release with acceptance criteria, we would identify test set which should be run before milestone release. The objective is to report out more bugs and close outstanding bugs in each milestone release. The milestone test set will be selected from the above test sets to ensure MeeGo milestone release goal will be achieved. Details TBD.
Retrieved from "http://wiki.meego.com/Quality/TestSetGuideline"
Personal tools