Revision as of 11:01, 1 September 2010

MeeGo 1.1 HandSet UX Test Plan

Introduction

This is overall test plan for MeeGo 1.1 HandSet UX of MeeGo open source project, which defines overall Quality Assurance procedure of validation activities done for MeeGo 1.1 HandSet UX release. A series of component test plans will also be linked in this overall test plan to cover detailed test approaches. This will be joint effort from MeeGo QA.

Objectives

Objectives in MeeGo 1.1 HandSet UX software testing is to validate the functionality of entire MeeGo 1.1 HandSet UX software delivery by performing daily and weekly testing for software releases. Target is to ensure that

• Planned and delivered features for MeeGo 1.1 HandSet UX are working as specified as a part of system.
• Validate that relevant bugs are fixed in software release.
• Program maturity statement can be and is given.

Goal

The goal is to deliver software release with no open bugs with a priority level of high and a minimal number of open bugs with priority level medium.

Test Strategy and Approach

Introduction

Application is launched from Graphical User Interface and features are used inside application to see that how those are working inside application. Also in system testing applications are used simultaneously to see how applications are interacting as part of system.

Overall procedure in Quality Assurance for MeeGo 1.1 HandSet UX is as following

• Firstly decompose features to component, each will be associated with one component test plan
• Ensure testability of planned features forming component
• Write test design in component test plan
• Define and store (to Test Link) test cases for features
• Connect test cases to features in test management tool
• Prioritize test cases to form test sets
• Review component test plan and test cases
• Automate test cases and add tests to fully automated test infrastructure
• Execute test cases in test sets for software releases following test execution and feature releasing plan
• Report test results and raise relevant bugs
• Provide maturity statement for main releases based on received test results

Feature Test and System Test

QA target is to validate MeeGo distribution

• Feature functionality
• System functionality (Interaction and negative scenarios)
• System performance (response time)
• System reliability

Following chart illuminates scope and relationship of feature and system testing.

Feature Testing

• Target is to test full functionality of specified feature forming component (e.g. Dialer) following the features' definition in featurezilla. Test case example: Make a phone call
• Every component (formed by features) basic functionality is tested in feature test set

System Testing

• Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Listening music while receiving incoming call
• Target is to test system testing (performance). Test case example: Open dialer application (Target value 0.1 sec)
• Target is to test system testing (reliability). Test case example: Make 200 calls (Target 199 pass, 1 fail)

• Not tested NFT types: Performance (Throughput, Framerate, Load, Memory Consumption and Power Management) and Reliability (Endurance, Aging, Long Period and Low Resource)

• Each test component will be documented in component test plan. Test plan will cover all testing aspects for specific component/feature(s).

Testability

Testability of MeeGo 1.1 HandSet UX features are ensured at first.

• Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
• Selected Quality Assurance Owners are checking those features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.

Relevant Links

• http://bugs.meego.com/ (MeeGo 1.1 UX HandSet Features are stored in Bugzilla)
• http://wiki.meego.com/Quality/TestabilityChecklist
• http://wiki.meego.com/Quality/HandsetTestabilityStatus

Test Cases

Test Cases are designed by QA owners based on existing features and which have been approved from testability point of view. Test Cases itself are stored to TestLink tool. Common Test Case Template is used when designing test cases.

• Overall test design process and guideline from features to actual test cases can be found http://wiki.meego.com/Quality/TestDesignProcessAndGuideline

Relevant Links

• <Link to Test Link tool>
• http://wiki.meego.com/TestCaseTemplate

Coverage

When features forming components are analysed and test cases are designed based on those also coverage matrix will be created for each component. From coverage matrix it can be seen that what is feature coverage i.e. planned test cases vs. maximum amount of test cases to cover every user scenarios from component/feature.

Relevant Links

• <Coverage Matrix Template>
• <Coverage Matrix for MeeGo 1.1 HandSet UX component/features>

Features to be Tested

• Overall the MeeGo 1.1 HandSet UX Testing will cover the MeeGo 1.1 HandSet UX layer in MeeGo Architecture:

• Specific components/features to be tested will be aligned with the features under MeeGo HandSet Features product in MeeGo Featurezilla @ Bugzilla

Features not to be Tested

• <List of exact features not to be tested can be added after needed query is implemented to Featurezilla @ Bugzilla>

Configurations

MeeGo 1.1 HandSet UX is tested in a number of reference devices. The public reference configurations used for this release are

• N900
• AAVA
• MRST CDK

Test Sets and Priorization

Test sets are formed to TestLink tool by using specific field inside the tool. Test sets that are formed are

• Sanity Test Set
• Regression Test Set
• Feature Test Suite (Basic and Extended)
• System Functional Test Suite (Interactions and negative user scenarios)
• System Non-Functional Test Suite (Performance and Reliability) - Note! Excluded from 1.1 and targeted to 1.2 release.
• Milestone Test Suite

Quality Assurance Owners are setting priorities for Test Cases to form these Test Suites to be used for test execution.

When test suites are in place in public Test Link -tool, then every test suite is reviewed and approved with respective persons.

More detailed information: http://wiki.meego.com/Quality/TestSetGuideline

Note! During MeeGo 1.1 HandSet UX Timeframe QA will not form System Non-Functional Test Suites. Those will be targeted for 1.2 release.

Test Automation

• Testability driver has been selected as Handset UX automation tool

Requirement Coverage Visibility

• All relevant features are taken from featurezilla @bugzilla and inserted as testing requirements to Test Link-tool requirement interleaf
• Test cases which have been designed against features are then connected under features to show feature coverage

• Target is also to be able to show latest test execution status against features

Test Execution

All automated tests are executed in a MeeGo QA automated environment, and typically test results are available for each build.

Manual tests are executed regularly, but certainly before each release.

In general, MeeGo will be tested from the following different test execution levels.

• http://wiki.meego.com/Quality/TestSetGuideline

Test Reporting

All MeeGo 1.1 UX HandSet test results are stored to one place.

• MeeGo 1.1 Test Repository for HandSet
  • http://wiki.meego.com/Quality/HandsetTestReport

Use Test Report Templates can be found: http://wiki.meego.com/TestReportTemplateCollection

Milestone Criteria

• There will be entry and exit criteria defined for each main milestone (Developer Preview, Feature Complete, Release Candidate and Project Release).
• All materials currently related to milestone quality criteria are stored to http://wiki.meego.com/Release_Engineering/Release_Timeline

Network Environment

• Networking environment needed to conduct testing
  • LAN
  • WiFi network
  • Internet
  • 3G network

Detailed Test Plans

To categorize the production requirements and identify the production functionality that will be tested, the product will be broken down to series of requirement set that QA owners are responsible for the validating.

Dependency and Constraints

• Features' testability is a big dependency for test case design.
• Features' integration time line is another dependency for test case design. If features are integrated late, a lot of test cases' debug will be blocked.

References

• QA main wiki: http://wiki.meego.com/Quality
• Feature Testability checklist: http://wiki.meego.com/Quality/TestabilityChecklist
• Testability Status Report: http://wiki.meego.com/Quality/HandsetTestabilityStatus
• Test Case Design Progress Follow-up: http://wiki.meego.com/Quality/HandsetTestSuite
• Test Result Reports: http://wiki.meego.com/Quality/HandsetTestReport
• Test Set Guideline: http://wiki.meego.com/Quality/TestSetGuideline
• Test Design Process and Guideline: http://wiki.meego.com/Quality/TestDesignProcessAndGuideline
• MeeGo Architecture http://meego.com/developers/meego-architecture
• MeeGo Bugzilla: http://bugs.meego.com/
• HandSet UX QA Ramp-Up follow up: http://wiki.meego.com/Quality/HandSetUXRamp-Up1.1