Views
Quality/Plans/Handset UX test plan
From MeeGo wiki
m (→Test Automation) |
(→System Test Plans) |
||
| (5 intermediate revisions not shown) | |||
| Line 90: | Line 90: | ||
==== Features to be Tested ==== | ==== Features to be Tested ==== | ||
* Overall the MeeGo HandSet UX Testing will cover the MeeGo HandSet UX layer in [http://meego.com/developers/meego-architecture MeeGo Architecture]: | * Overall the MeeGo HandSet UX Testing will cover the MeeGo HandSet UX layer in [http://meego.com/developers/meego-architecture MeeGo Architecture]: | ||
| - | [[File: | + | [[File:MeeGo_arch_qa_view.PNG]] |
* Specific components/features to be tested will be aligned with the features under MeeGo HandSet Features product in [http://bugs.meego.com MeeGo Featurezilla @ Bugzilla] | * Specific components/features to be tested will be aligned with the features under MeeGo HandSet Features product in [http://bugs.meego.com MeeGo Featurezilla @ Bugzilla] | ||
| Line 96: | Line 96: | ||
==== Features not to be Tested ==== | ==== Features not to be Tested ==== | ||
* List of exact features not to be tested can be found from Featurezilla @ Bugzilla. One must use Testability query there to have full list identified. | * List of exact features not to be tested can be found from Featurezilla @ Bugzilla. One must use Testability query there to have full list identified. | ||
| - | ** [ | + | ** [https://bugs.meego.com/report.cgi?x_axis_field=cf_testability&y_axis_field=component&z_axis_field=&query_format=report-table&short_desc_type=allwordssubstr&short_desc=&classification=MeeGo+Features&product=MeeGo+Handset+Features&longdesc_type=allwordssubstr&longdesc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&deadlinefrom=&deadlineto=&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1&emailcc2=1&emailtype2=substring&email2=&bug_id_type=anyexact&bug_id=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&format=table&action=wrap&field0-0-0=noop&type0-0-0=noop&value0-0-0= Testability in Featurezilla] |
=== Configurations === | === Configurations === | ||
| Line 171: | Line 171: | ||
{| border="1" | {| border="1" | ||
!| Component | !| Component | ||
| - | |||
| - | |||
!| Detailed test plan | !| Detailed test plan | ||
|- | |- | ||
| - | | Applets | + | | Applets || [[Quality/MeeGo1.2_Handset_Applets_Test_Plan|MeeGo 1.2 Handset Applets Test Plan]] |
|- | |- | ||
| - | | Short Message Service | + | | Short Message Service || [[Quality/MeeGo1.2HandSetUXTestPlanforShortMessageService|MeeGo 1.2 Handset SMS Test Plan]] |
|- | |- | ||
| - | | Dialer | + | | Dialer || [[Quality/MeeGo1.2HandSetUXTestPlanforDialer|MeeGo 1.2 Handset Dialer Test Plan]] |
|- | |- | ||
| - | | Media Applications | + | | Media Applications || [[Quality/Plans/Meego1.2_media_test_plan|MeeGo1.2 Handset Media Applications Test Plan]] |
|- | |- | ||
| - | | Mozilla Fennec Browser | + | | Mozilla Fennec Browser || [[Quality/MeeGo1.2HandSetUXTestPlanforMozillaFennecBrowser|MeeGo 1.2 Handset Mozilla Fennec Browser Test Plan]] |
|- | |- | ||
| - | | Contacts | + | | Contacts|| [[Quality/Meego_contacts_test_plan_v12|MeeGo1.2 Handset Contacts Test Plan]] |
|- | |- | ||
| - | | Core UX (Home, Theme, System UI) | + | | Core UX (Home, Theme, System UI)|| [[Quality/MeeGo1.2_Handset_CoreUX_TestPlan|MeeGo1.2 Handset Core UX Test Plan]] |
|- | |- | ||
| - | | Social Networking | + | | Social Networking || [[Quality/MeeGo1.2_Handset_UX_Social_Networking_TestPlan|MeeGo1.2 Handset Social Networking Test Plan]] |
|- | |- | ||
| - | | Compositing Window Manager | + | | Compositing Window Manager || <link to detailed test plan> |
|- | |- | ||
| - | | Application install/uninstall | + | | Application install/uninstall || No requirement yet. |
|- | |- | ||
| - | | Virtual Keyboard | + | | Virtual Keyboard || [[Quality/Plans/Meego1.2_vkb_test_plan/|VKB Test Plan]] |
|- | |- | ||
| - | | Sync client | + | | Sync client || [[Quality/Meego_Handset_SyncUI_TestPlan_v1.2|SyncUI Test Plan]] |
|- | |- | ||
| - | | Email | + | | Email || [[Quality/Plans/Meego1.2_email_test_plan|Email Test Plan]] |
|- | |- | ||
| - | | Calendar | + | | Calendar || [[Quality/Meego_Handset_Calendar_TestPlan_v12|MeeGo1.2 Handset Calendar Test Plan]] |
|- | |- | ||
| - | | Instant Messaging | + | | Instant Messaging || [[Quality/MeeGo1.2HandSetUXTestPlanforInstantMessaging|MeeGo 1.2 Handset Instant Messaging test Plan]] |
|- | |- | ||
| - | | Connectivity UI | + | | Connectivity UI || Currently only one requirement for Internalisation available - NO testplan needed |
|- | |- | ||
| - | | Settings | + | | Settings || [[Quality/Meego_settings_test_plan_v12|MeeGo1.2 Handset Settings Test Plan]] |
|- | |- | ||
| - | | UI Infrastructure | + | | UI Infrastructure || [[Quality/Plans/Handset_UX_test_planUI_infrastructure|MeeGo 1.2 Handset UI Infrastructure Test Plan]] |
|} | |} | ||
| Line 216: | Line 214: | ||
{| border="1" | {| border="1" | ||
!| System Test Plans | !| System Test Plans | ||
| - | |||
!| Detailed test plan | !| Detailed test plan | ||
|- | |- | ||
| - | | System Functional Test Plan | + | | System Functional Test Plan || [[Quality/MeeGo1.2HandSetUXTestPlanforSyFuTe|MeeGo 1.2 Handset System Functional Testing Test Plan]] |
|- | |- | ||
| - | | System Non-Functional Test Plan | + | | System Non-Functional Test Plan || [[Quality/MeeGo1.2HandSetUXTestPlanforSystemNFT|MeeGo 1.2 Handset System NFT Test Plan]] |
|} | |} | ||
| Line 233: | Line 230: | ||
* Testability Status Report: [[Quality/HandsetTestabilityStatus]] | * Testability Status Report: [[Quality/HandsetTestabilityStatus]] | ||
* Test Case Design Progress Follow-up: [[Quality/HandsetTestSuite]] | * Test Case Design Progress Follow-up: [[Quality/HandsetTestSuite]] | ||
| - | |||
* Test Set Guideline: [[Quality/TestSetGuideline]] | * Test Set Guideline: [[Quality/TestSetGuideline]] | ||
* Test Design Process and Guideline: [[Quality/TestDesignProcessAndGuideline]] | * Test Design Process and Guideline: [[Quality/TestDesignProcessAndGuideline]] | ||
* MeeGo Architecture http://meego.com/developers/meego-architecture | * MeeGo Architecture http://meego.com/developers/meego-architecture | ||
* MeeGo Bugzilla: http://bugs.meego.com/ | * MeeGo Bugzilla: http://bugs.meego.com/ | ||
| - | |||
Latest revision as of 02:41, 20 June 2011
MeeGo HandSet UX Test Plan
Introduction
This is overall test plan for MeeGo HandSet UX of MeeGo open source project, which defines overall Quality Assurance procedure of validation activities done for MeeGo HandSet UX release. A series of component and system test plans will also be linked in this overall test plan to cover detailed test approaches. This will be joint effort from MeeGo QA Handset UX team.
Objectives
Objectives in MeeGo HandSet UX software testing is to validate the functionality of entire MeeGo HandSet UX software delivery by performing daily and weekly testing for software releases. Target is to ensure that
- Planned and delivered features for MeeGo release HandSet UX are working as specified as a part of system.
- Validate that relevant bugs are fixed in software release.
- Program maturity statement can be and is given.
Goal
The goal is to deliver software release with no open bugs with a priority level of high and a minimal number of open bugs with priority level medium.
Test Strategy and Approach
Introduction
Application is launched from Graphical User Interface and features are used inside application to see that how those are working inside application. Also in system testing applications are used simultaneously to see how applications are interacting as part of system.
Overall procedure in Quality Assurance for MeeGo HandSet UX is as following
- Firstly decompose features to component, each will be associated with one component test plan
- Ensure testability of planned features forming component
- Write test design in component test plan
- Define and store (to Test Link) test cases for features
- Connect test cases to features in test management tool
- Prioritize test cases to form test sets
- Review test plan and test cases
- Automate test cases and add tests to fully automated test infrastructure
- Execute test cases in test sets for software releases following test execution and feature releasing plan
- Report test results and raise relevant bugs
- Provide maturity statement for main releases based on received test results
Feature Test and System Test
QA target is to validate MeeGo distribution
- Feature functionality
- System functionality (Interaction and negative scenarios)
- System performance
- System reliability
Following chart illuminates scope and relationship of feature and system testing.
Feature Testing
- Target is to test full functionality of specified feature forming component (e.g. Dialer) following the features' definition in featurezilla. Test case example: Make a phone call
- Every component (formed by features) basic functionality is tested in feature test set
- Each test component will be documented in component test plan. Test plans will cover all testing aspects for specific component/feature(s).
System Testing
- Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Listening music while receiving incoming call
- Target is to test system testing (performance). Test case example: Open dialer application (Target value 0.1 sec)
- Target is to test system testing (reliability). Test case example: Make 200 calls (Target 199 pass, 1 fail)
- System Test Plans are created as separate test plans containing both Functional and Non-Functional System Testing aspects
Testability
Testability of MeeGo HandSet UX features are ensured at first.
- Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
- Selected Quality Assurance Owners are checking those features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
Relevant Links
- http://bugs.meego.com/ (MeeGo UX HandSet Features are stored in Bugzilla)
- Quality/TestabilityChecklist
- Quality/HandsetTestabilityStatus
Test Cases
Test Cases are designed by QA owners based on existing features and which have been approved from testability point of view. Common Test Case Template is used when designing test cases. Test cases are released publicly in MeeGo Gitorious under Handset UX Tests part.
- Overall test design process and guideline from features to actual test cases can be found Quality/TestDesignProcessAndGuideline
Relevant Links
- Test Case Template
- http://meego.gitorious.org/meego-quality-assurance
- http://meego.gitorious.org/meego-quality-assurance/handset-ux-tests
Coverage
When features forming components are analysed and test cases are designed based on those also coverage matrix will be created for each component. From coverage matrix it can be seen that what is feature coverage i.e. planned test cases vs. maximum amount of test cases to cover every user scenarios from component/feature.
Relevant Links
- <Coverage Matrix Template>
- <Coverage Matrix for MeeGo HandSet UX component/features>
Features to be Tested
- Overall the MeeGo HandSet UX Testing will cover the MeeGo HandSet UX layer in MeeGo Architecture:
- Specific components/features to be tested will be aligned with the features under MeeGo HandSet Features product in MeeGo Featurezilla @ Bugzilla
Features not to be Tested
- List of exact features not to be tested can be found from Featurezilla @ Bugzilla. One must use Testability query there to have full list identified.
Configurations
MeeGo HandSet UX is tested in a number of reference devices. The public reference configurations used for this release are
Test Sets, Definitions and Priorization
Test sets are formed to Test Management Tool by using specific field inside the tool. Test sets that are formed are
- Acceptance/Sanity Test
- Acceptance/Sanity test set is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance/sanity test reflects the most important and basic functionalities of the distribution.
- Acceptance and Sanity test sets are relatively stable and will be run on daily basis.
- Feature Test
- Key Feature Test Set is used to verify MeeGo Handset UX most critical key use cases functionalities with well selected basic feature test cases.
- Basic Feature Test Set is verifying MeeGo HandSet UX delivered features with basic feature test cases. Test set is always static to show overall feature functionalities progress and maturity of the entire MeeGo distribution. Based on test results QA is able to identify components with working features to enable extended feature testing and system testing.
- Extended Feature Test Set is used to verify delivery of features forming full functionality of entirely component. After component is fully integrated all component related test cases will be executed for selected weekly release and report out all the bugs against component and it’s features. Extended feature test set will be run again in the upcoming milestone or when significant changes are applied to component and it’s features.
- System Functional
- System Functional Test Set is targeting to evaluate delivered functionalities from system perspective. Test cases are not testing UI or Application itself, instead test cases are testing how whole system is working and interacting with Consumer (end user). Test cases are covering most critical interaction and negative scenarios that consumers will encounter in their daily usage.
- System Non-Functional
- System Performance Test Sets target is to evaluate overall system performance by executing well-selected set of cases from different test areas - for example response and reaction times, use times and frame rate measurements. Test set gives a quick view of system performance from end-user point of view.
- System Reliability Test Sets target is to provide an overview to system reliability by executing iterative tests that focus on the most important and most used end-user features of MeeGo.
Quality Assurance Owners are setting priorities for Test Cases to form these Test Sets to be used for test execution.
More detailed information: Quality/TestSetGuideline
Test Automation
- Testability driver has been selected as Handset UX automation tool
- Main focus in test automation will be in acceptance, sanity and regression testing automisation
- Automated scripts are released in Gitorius: http://meego.gitorious.org/meego-quality-assurance under Handset UX Tests part
Requirement Coverage Visibility
- All relevant features are taken from featurezilla @bugzilla and inserted as testing requirements to Test Link-tool requirement interleaf
- Test cases which have been designed against features are then connected under features to show feature coverage
- Target is also to be able to show latest test execution status against features
Test Execution
All automated tests are executed in a MeeGo QA automated environment, and typically test results are available for each build.
Manual tests are executed regularly, but certainly before each release.
In general, MeeGo will be tested from the following different test execution levels.
Test Reporting
The test reports in the wiki will be deleted soon as requested. MeeGo test reports are now located at qa-reports.meego.com.
Milestone Criteria
- There will be entry and exit criteria defined for each main milestone (Developer Preview, Feature Complete, Release Candidate and Project Release).
- All materials currently related to milestone quality criteria are stored to Release_Engineering/Release_Timeline
Network Environment
- Networking environment needed to conduct testing
- LAN
- WiFi network
- Internet
- 3G network
Detailed Test Plans
To categorize the production requirements and identify the production functionality that will be tested, the product will be broken down to series of requirement set that QA owners are responsible for the validating.
Component Test Plans
| Component | Detailed test plan |
|---|---|
| Applets | MeeGo 1.2 Handset Applets Test Plan |
| Short Message Service | MeeGo 1.2 Handset SMS Test Plan |
| Dialer | MeeGo 1.2 Handset Dialer Test Plan |
| Media Applications | MeeGo1.2 Handset Media Applications Test Plan |
| Mozilla Fennec Browser | MeeGo 1.2 Handset Mozilla Fennec Browser Test Plan |
| Contacts | MeeGo1.2 Handset Contacts Test Plan |
| Core UX (Home, Theme, System UI) | MeeGo1.2 Handset Core UX Test Plan |
| Social Networking | MeeGo1.2 Handset Social Networking Test Plan |
| Compositing Window Manager | <link to detailed test plan> |
| Application install/uninstall | No requirement yet. |
| Virtual Keyboard | VKB Test Plan |
| Sync client | SyncUI Test Plan |
| Email Test Plan | |
| Calendar | MeeGo1.2 Handset Calendar Test Plan |
| Instant Messaging | MeeGo 1.2 Handset Instant Messaging test Plan |
| Connectivity UI | Currently only one requirement for Internalisation available - NO testplan needed |
| Settings | MeeGo1.2 Handset Settings Test Plan |
| UI Infrastructure | MeeGo 1.2 Handset UI Infrastructure Test Plan |
System Test Plans
| System Test Plans | Detailed test plan |
|---|---|
| System Functional Test Plan | MeeGo 1.2 Handset System Functional Testing Test Plan |
| System Non-Functional Test Plan | MeeGo 1.2 Handset System NFT Test Plan |
Dependency and Constraints
- Features' testability is a big dependency for test case design.
- Features' integration time line is another dependency for test case design. If features are integrated late, a lot of test cases' debug will be blocked.
References
- QA main wiki: Quality
- Feature Testability checklist: Quality/TestabilityChecklist
- Testability Status Report: Quality/HandsetTestabilityStatus
- Test Case Design Progress Follow-up: Quality/HandsetTestSuite
- Test Set Guideline: Quality/TestSetGuideline
- Test Design Process and Guideline: Quality/TestDesignProcessAndGuideline
- MeeGo Architecture http://meego.com/developers/meego-architecture
- MeeGo Bugzilla: http://bugs.meego.com/