Revision as of 15:06, 31 January 2011

MeeGo HandSet UX Test Plan

Introduction

This is overall test plan for MeeGo HandSet UX of MeeGo open source project, which defines overall Quality Assurance procedure of validation activities done for MeeGo HandSet UX release. A series of component and system test plans will also be linked in this overall test plan to cover detailed test approaches. This will be joint effort from MeeGo QA Handset UX team.

Objectives

Objectives in MeeGo HandSet UX software testing is to validate the functionality of entire MeeGo HandSet UX software delivery by performing daily and weekly testing for software releases. Target is to ensure that

• Planned and delivered features for MeeGo release HandSet UX are working as specified as a part of system.
• Validate that relevant bugs are fixed in software release.
• Program maturity statement can be and is given.

Goal

The goal is to deliver software release with no open bugs with a priority level of high and a minimal number of open bugs with priority level medium.

Test Strategy and Approach

Introduction

Application is launched from Graphical User Interface and features are used inside application to see that how those are working inside application. Also in system testing applications are used simultaneously to see how applications are interacting as part of system.

Overall procedure in Quality Assurance for MeeGo HandSet UX is as following

• Firstly decompose features to component, each will be associated with one component test plan
• Ensure testability of planned features forming component
• Write test design in component test plan
• Define and store (to Test Link) test cases for features
• Connect test cases to features in test management tool
• Prioritize test cases to form test sets
• Review test plan and test cases
• Automate test cases and add tests to fully automated test infrastructure
• Execute test cases in test sets for software releases following test execution and feature releasing plan
• Report test results and raise relevant bugs
• Provide maturity statement for main releases based on received test results

Feature Test and System Test

QA target is to validate MeeGo distribution

• Feature functionality
• System functionality (Interaction and negative scenarios)
• System performance
• System reliability

Following chart illuminates scope and relationship of feature and system testing.

Feature Testing

• Target is to test full functionality of specified feature forming component (e.g. Dialer) following the features' definition in featurezilla. Test case example: Make a phone call
• Every component (formed by features) basic functionality is tested in feature test set
• Each test component will be documented in component test plan. Test plans will cover all testing aspects for specific component/feature(s).

System Testing

• Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Listening music while receiving incoming call
• Target is to test system testing (performance). Test case example: Open dialer application (Target value 0.1 sec)
• Target is to test system testing (reliability). Test case example: Make 200 calls (Target 199 pass, 1 fail)
• System Test Plans are created as separate test plans containing both Functional and Non-Functional System Testing aspects

Testability

Testability of MeeGo HandSet UX features are ensured at first.

• Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
• Selected Quality Assurance Owners are checking those features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.

Relevant Links

• http://bugs.meego.com/ (MeeGo UX HandSet Features are stored in Bugzilla)
• Quality/TestabilityChecklist
• Quality/HandsetTestabilityStatus

Test Cases

Test Cases are designed by QA owners based on existing features and which have been approved from testability point of view. Common Test Case Template is used when designing test cases. Test cases are released publicly in MeeGo Gitorious under Handset UX Tests part.

• Overall test design process and guideline from features to actual test cases can be found Quality/TestDesignProcessAndGuideline

Relevant Links

• TestCaseTemplate
• http://gitorious.org/qa-tools/
• http://meego.gitorious.org/meego-quality-assurance/handset-ux-tests

Coverage

When features forming components are analysed and test cases are designed based on those also coverage matrix will be created for each component. From coverage matrix it can be seen that what is feature coverage i.e. planned test cases vs. maximum amount of test cases to cover every user scenarios from component/feature.

Relevant Links

• <Coverage Matrix Template>
• <Coverage Matrix for MeeGo HandSet UX component/features>

Features to be Tested

• Overall the MeeGo HandSet UX Testing will cover the MeeGo HandSet UX layer in MeeGo Architecture:

• Specific components/features to be tested will be aligned with the features under MeeGo HandSet Features product in MeeGo Featurezilla @ Bugzilla

Features not to be Tested

• List of exact features not to be tested can be found from Featurezilla @ Bugzilla. One must use Testability query there to have full list identified.
  • Testability in Featurezilla (1.2)

Configurations

MeeGo HandSet UX is tested in a number of reference devices. The public reference configurations used for this release are

• Supported Hardware

Test Sets, Definitions and Priorization

Test sets are formed to Test Management Tool by using specific field inside the tool. Test sets that are formed are

• Acceptance/Sanity Test
  • Acceptance/Sanity test set is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance/sanity test reflects the most important and basic functionalities of the distribution.
  • Acceptance and Sanity test sets are relatively stable and will be run on daily basis.
• Feature Test
  • Key Feature Test Set is used to verify MeeGo Handset UX most critical key use cases functionalities with well selected basic feature test cases.
  • Basic Feature Test Set is verifying MeeGo HandSet UX delivered features with basic feature test cases. Test set is always static to show overall feature functionalities progress and maturity of the entire MeeGo distribution. Based on test results QA is able to identify components with working features to enable extended feature testing and system testing.
  • Extended Feature Test Set is used to verify delivery of features forming full functionality of entirely component. After component is fully integrated all component related test cases will be executed for selected weekly release and report out all the bugs against component and it’s features. Extended feature test set will be run again in the upcoming milestone or when significant changes are applied to component and it’s features.
• System Functional
  • System Functional Test Set is targeting to evaluate delivered functionalities from system perspective. Test cases are not testing UI or Application itself, instead test cases are testing how whole system is working and interacting with Consumer (end user). Test cases are covering most critical interaction and negative scenarios that consumers will encounter in their daily usage.
• System Non-Functional
  • System Performance Test Sets target is to evaluate overall system performance by executing well-selected set of cases from different test areas - for example response and reaction times, use times and frame rate measurements. Test set gives a quick view of system performance from end-user point of view.
  • System Reliability Test Sets target is to provide an overview to system reliability by executing iterative tests that focus on the most important and most used end-user features of MeeGo.

Quality Assurance Owners are setting priorities for Test Cases to form these Test Sets to be used for test execution.

More detailed information: Quality/TestSetGuideline

Test Automation

• Testability driver has been selected as Handset UX automation tool
• Main focus in test automation will be in acceptance, sanity and regression testing automisation
• Automated scripts are released in Gitorius: http://gitorious.org/qa-tools/ under Handset UX Tests part

Requirement Coverage Visibility

• All relevant features are taken from featurezilla @bugzilla and inserted as testing requirements to Test Link-tool requirement interleaf
• Test cases which have been designed against features are then connected under features to show feature coverage

• Target is also to be able to show latest test execution status against features

Test Execution

All automated tests are executed in a MeeGo QA automated environment, and typically test results are available for each build.

Manual tests are executed regularly, but certainly before each release.

In general, MeeGo will be tested from the following different test execution levels.

• Quality/TestSetGuideline

Test Reporting

All MeeGo HandSet UX test results are stored to one place.

• MeeGo Test Repository for HandSet
  • Quality/HandsetTestReport

Use Test Report Templates can be found: TestReportTemplateCollection

Milestone Criteria

• There will be entry and exit criteria defined for each main milestone (Developer Preview, Feature Complete, Release Candidate and Project Release).
• All materials currently related to milestone quality criteria are stored to Release_Engineering/Release_Timeline

Network Environment

• Networking environment needed to conduct testing
  • LAN
  • WiFi network
  • Internet
  • 3G network

Detailed Test Plans

To categorize the production requirements and identify the production functionality that will be tested, the product will be broken down to series of requirement set that QA owners are responsible for the validating.

Component Test Plans

System Test Plans

Dependency and Constraints

• Features' testability is a big dependency for test case design.
• Features' integration time line is another dependency for test case design. If features are integrated late, a lot of test cases' debug will be blocked.

References

• QA main wiki: Quality
• Feature Testability checklist: Quality/TestabilityChecklist
• Testability Status Report: Quality/HandsetTestabilityStatus
• Test Case Design Progress Follow-up: Quality/HandsetTestSuite
• Test Result Reports: Quality/HandsetTestReport
• Test Set Guideline: Quality/TestSetGuideline
• Test Design Process and Guideline: Quality/TestDesignProcessAndGuideline
• MeeGo Architecture http://meego.com/developers/meego-architecture
• MeeGo Bugzilla: http://bugs.meego.com/
• HandSet UX QA Ramp-Up follow up: Quality/HandSetUXRamp-Up1.1