MeeGo 1.1 HandSet UX Test Plan
This is overall test plan for MeeGo 1.1 HandSet UX of MeeGo open source project, which defines overall Quality Assurance procedure of validation activities done for MeeGo 1.1 HandSet UX release. A series of component test plans will also be linked in this overall test plan to cover detailed test approaches. This will be joint effort from MeeGo QA.
Objectives in MeeGo 1.1 HandSet UX software testing is to validate the functionality of entire MeeGo 1.1 HandSet UX software delivery by performing daily and weekly testing for software releases. Target is to ensure that
- Planned and delivered features for MeeGo 1.1 HandSet UX are working as specified as a part of system.
- Validate that relevant bugs are fixed in software release.
- Program maturity statement can be and is given.
The goal is to deliver software release with no open bugs with a priority level of high and a minimal number of open bugs with priority level medium.
Test Strategy and Approach
Application is launched from Graphical User Interface and features are used inside application to see that how those are working inside application. Also in system testing applications are used simultaneously to see how applications are interacting as part of system.
Overall procedure in Quality Assurance for MeeGo 1.1 HandSet UX is as following
- Firstly decompose features to component, each will be associated with one component test plan
- Ensure testability of planned features forming component
- Write test design in component test plan
- Define and store (to Test Link) test cases for features
- Connect test cases to features in test management tool
- Prioritize test cases to form test sets
- Review component test plan and test cases
- Automate test cases and add tests to fully automated test infrastructure
- Execute test cases in test sets for software releases following test execution and feature releasing plan
- Report test results and raise relevant bugs
- Provide maturity statement for main releases based on received test results
Feature Test and System Test
QA target is to validate MeeGo distribution
- Feature functionality
- System functionality (Interaction and negative scenarios)
- System performance (response time)
- System reliability
Following chart illuminates scope and relationship of feature and system testing.
- Target is to test full functionality of specified feature forming component (e.g. Dialer) following the features' definition in featurezilla. Test case example: Make a phone call
- Every component (formed by features) basic functionality is tested in feature test set
- Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Listening music while receiving incoming call
- Target is to test system testing (performance). Test case example: Open dialer application (Target value 0.1 sec)
- Target is to test system testing (reliability). Test case example: Make 200 calls (Target 199 pass, 1 fail)
- Not tested NFT types: Performance (Throughput, Framerate, Load, Memory Consumption and Power Management) and Reliability (Endurance, Aging, Long Period and Low Resource)
- Each test component will be documented in component test plan. Test plan will cover all testing aspects for specific component/feature(s).
Testability of MeeGo 1.1 HandSet UX features are ensured at first.
- Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
- Selected Quality Assurance Owners are checking those features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
Test Cases are designed by QA owners based on existing features and which have been approved from testability point of view. Test Cases itself are stored to TestLink tool. Common Test Case Template is used when designing test cases.
When features forming components are analysed and test cases are designed based on those also coverage matrix will be created for each component. From coverage matrix it can be seen that what is feature coverage i.e. planned test cases vs. maximum amount of test cases to cover every user scenarios from component/feature.
- <Coverage Matrix Template>
- <Coverage Matrix for MeeGo 1.1 HandSet UX component/features>
Features to be Tested
- Overall the MeeGo 1.1 HandSet UX Testing will cover the MeeGo 1.1 HandSet UX layer in MeeGo Architecture:
- Specific components/features to be tested will be aligned with the features under MeeGo HandSet Features product in MeeGo Featurezilla @ Bugzilla
Features not to be Tested
- <List of exact features not to be tested can be added after needed query is implemented to Featurezilla @ Bugzilla>
MeeGo 1.1 HandSet UX is tested in a number of reference devices. The public reference configurations used for this release are
Test Sets and Priorization
Test sets are formed to TestLink tool by using specific field inside the tool. Test sets that are formed are
- Sanity Test Set
- Regression Test Set
- Feature Test Suite (Basic and Extended)
- System Functional Test Suite (Interactions and negative user scenarios)
- System Non-Functional Test Suite (Performance and Reliability) - Note! Excluded from 1.1 and targeted to 1.2 release.
- Milestone Test Suite
Quality Assurance Owners are setting priorities for Test Cases to form these Test Suites to be used for test execution.
When test suites are in place in public Test Link -tool, then every test suite is reviewed and approved with respective persons.
More detailed information: http://wiki.meego.com/Quality/TestSetGuideline
Note! During MeeGo 1.1 HandSet UX Timeframe QA will not form System Non-Functional Test Suites. Those will be targeted for 1.2 release.
- Testability driver has been selected as Handset UX automation tool
Requirement Coverage Visibility
- All relevant features are taken from featurezilla @bugzilla and inserted as testing requirements to Test Link-tool requirement interleaf
- Test cases which have been designed against features are then connected under features to show feature coverage
- Target is also to be able to show latest test execution status against features
All automated tests are executed in a MeeGo QA automated environment, and typically test results are available for each build.
Manual tests are executed regularly, but certainly before each release.
In general, MeeGo will be tested from the following different test execution levels.
All MeeGo 1.1 UX HandSet test results are stored to one place.
- MeeGo 1.1 Test Repository for HandSet
Use Test Report Templates can be found: http://wiki.meego.com/TestReportTemplateCollection
- Networking environment needed to conduct testing
- WiFi network
- 3G network
Detailed Test Plans
To categorize the production requirements and identify the production functionality that will be tested, the product will be broken down to series of requirement set that QA owners are responsible for the validating.
- Telephony Applications: Mika Ikonen
- Web Browser: Petri Jylhä
- Networking: Petri Jylhä
- Instant Messaging and VOIP: Mika Ikonen
- Media Applications: Shuang Wan
- Core UX: Cathy Li
- Email: Yi Fu
- Virtual keyboard and Input method: Yi Fu
- Information Applications: Cathy Li
- Social networking: Cathy Li
- Data Sync: Qin Mu
- Contacts Management Applications: Dayu Yang
- Settings: Dayu Yang
- Calendar: Dayu Yang
Detailed Test Plans
|| Test Coverage
|| Detailed test plan
| Telephony Apps Test Plan || Telephony applications -- Dialer, SMS || <link to detailed test plan>
| Web Browser Test Plan || Fennec based handset browser || <link to detailed test plan>
| Networking Connectivity Test Plan || Connectivity applet and Connman || <link to detailed test plan>
| Instant Messaging Test Plan || Instant Messaging || <link to detailed test plan>
| Core UX Test Plan || Home Screen, app launcher, system UI, display orientation || <link to detailed test plan>
| Media Applications Test Plan || Video player, audio player, photo viewer, camera application || <link to detailed test plan>
| Settings Test Plan || Settings for timezone, bluetooth, telephony etc || <link to detailed test plan>
| Virtual Keyboard and Input method Test Plan || Virtual Keyboard and input method || <link to detailed test plan>
| Email Test Plan || Email Client || <link to detailed test plan>
| Data Sync Test Plan || Sync client || <link to detailed test plan>
| Information Applications Test Plan ||Applets (weather, stock, News feed) || <link to detailed test plan>
| Contacts Management Test Plan || Contacts || <link to detailed test plan>
| Calendar Test Plan || Calendar || <link to detailed test plan>
Dependency and Constraints
- Features' testability is a big dependency for test case design.
- Features' integration time line is another dependency for test case design. If features are integrated late, a lot of test cases' debug will be blocked.