Meego Wiki
Views

Quality/MeeGo1.2HandSetUXTestPlanforSystemNFT

From MeeGo wiki
< Quality(Difference between revisions)
Jump to: navigation, search
(Created page with "== MeeGo 1.2 HandSet UX Test Plan for Mozilla Fennec Browser == === Component Summary === This test plan is for MeeGo 1.2 HandSet UX Mozilla Fennec Browser. Firefox Mobile is th…")
(→Tested NFT Areas)
 
(17 intermediate revisions not shown)
Line 1: Line 1:
-
== MeeGo 1.2 HandSet UX Test Plan for Mozilla Fennec Browser ==
+
== MeeGo 1.2 HandSet UX Test Plan for System Non-Functional Test Plan ==
-
=== Component Summary ===
+
=== Summary ===
-
This test plan is for MeeGo 1.2 HandSet UX Mozilla Fennec Browser. Firefox Mobile is the Browser built for the way you use the web on your mobile device.
+
This is test plan for MeeGo 1.2 HandSet System Non-Functional Testing (NFT)
=== Features To Be Tested ===
=== Features To Be Tested ===
-
All testable features coming to MeeGo 1.2 HandSet UX Mozilla Fennec Browser will be covered in this test plan.  
+
MeeGo 1.2 release Non-Functional Requirements (NFR) are listed in [http://bugs.meego.com/buglist.cgi?query_format=advanced&short_desc=non-func&short_desc_type=allwordssubstr&classification=MeeGo%20Features FeatureZilla]
 +
 +
We are not testing all of NFRs because many of them are not realistic anymore for current public HW (N900). Especially when release will be run from memory card which decreases overall performance.
-
* Mozilla Fennec browser Features to be tested are summarized as below:
+
These features will be tested:
{| border="1"
{| border="1"
Line 14: Line 16:
!| Comments
!| Comments
|-
|-
-
| Feature [http://bugs.meego.com/show_bug.cgi?id=3103 3103] - [FEA] Browser remember web passwords
+
| Feature [http://bugs.meego.com/show_bug.cgi?id=12089 12089] - [FEA] Non-func - Application start-up time for pre-started applications
-
| Browser should provide support to remember web site passwords and pre-fill those to the web page once re-visited.  
+
| System shall start the most frequently used applications (which are pre-started) immediately (within 100ms) so that device feels responsive.
-
| Moved from release 1.1
+
| Pre-started applications not yet defined. As a basis we will use release 1.1 key applications (Dialer, SMS, Browser, Music Player, and Video Player)
|-
|-
-
| Feature [http://bugs.meego.com/show_bug.cgi?id=3128 3128] - [FEA] Browser web page orientation
+
| Feature [http://bugs.meego.com/show_bug.cgi?id=12090 12090] - [FEA] Non-func - Application start-up time for (non-pre-started) applications
-
| Browser should support of rendering the content optimised both on landscape and portrait modes depending on the orientation of the device (if device orientation supported by the device).  
+
| System shall start all applications (excluding pre-started applications) within 0.5s so that device feels responsive
-
| Moved from release 1.1
+
| Pre-started applications not yet defined. As a basis we will use release 1.1 key applications => other are non-pre-started applications
|-
|-
-
| Feature [http://bugs.meego.com/show_bug.cgi?id=4104 4104] - [FEA] Browser - File download in progress indicator
+
| Feature [http://bugs.meego.com/show_bug.cgi?id=12091 12091] - [FEA] Non-func - Application start-up time for Browser
-
| The device SHALL provide continuous progress feedback during file download so the user knows the download is still in progress. Remaining time or % in progress SHALL be indicated.  
+
| Browser application shall start and display meego.com page within 3s. This requirement assumes Internet connection is already present and active (WLAN 802.11g).
-
| Moved from release 1.1
+
|  
|-
|-
-
| Feature [http://bugs.meego.com/show_bug.cgi?id=4127 4127] - [FEA] Browser - Videos and CSS Layering
+
| Feature [http://bugs.meego.com/show_bug.cgi?id=12095 12095] - [FEA] Non-func - Cold boot
-
| The browser SHALL support the display of HTML elements on top of an embedded video object using standard CSS layering (position and z-index properties).
+
| System shall boot up from cold state as quickly as possible. Maximum boot time shall be 30s, from cold state to UI being completely usable.
-
| Moved from release 1.1
+
|
|-
|-
-
| Feature [http://bugs.meego.com/show_bug.cgi?id=4130 4130] - [FEA] Browser - Deletion of cookies
+
| Feature [http://bugs.meego.com/show_bug.cgi?id=12096 12096] - [FEA] Non-func - Complete shut down
-
| The user SHALL be able to delete his cookies
+
| The system shall power off quickly and without losing any data. The maximum time to complete shut down of the system shall be 10s.
-
| Moved from release 1.1
+
|  
|-
|-
-
| Feature [http://bugs.meego.com/show_bug.cgi?id=9263 9263] - [FEA] [I18N]The Web browser shall be internationalized according to the dependencies listed in Core Master 5068.  
+
| Feature [http://bugs.meego.com/show_bug.cgi?id=12097 12097] - [FEA] Non-func - Screen orientation change
-
| This includes such items as:
+
| System shall render the UI in new orientation immediately after changing the device orientation. The UI shall be re-drawn in new orientation within 0.5s in all use scenarios.
-
  * Unicode support
+
|  
-
  * All translatable strings are externalized
+
|-
-
  * UI adaptability to longer and shorter strings than English
+
| Feature [http://bugs.meego.com/show_bug.cgi?id=12101 12101] - [FEA] Non-func - Video playback and incoming call
-
  * Ability to accept, display, sort, search, copy, paste, <br />    and print accented and double-byte characters
+
| It shall be possible to hear the ringtone within 0.5s during an active video playback.
-
  * No concatenation of translatable strings
+
Note: the behaviour depends on the policy defined by device vendor whereas the purpose of this requirement is to define the required performance once above policy is applicable. MeeGo reference UXs policies are defined at the Policy requirements.
-
  * Support for the POR languages: en_US, en_GB, fr, de, es, it, <br />    fi, sv, pl, ru, nl, pt_BR, ja, ko, zh_TW, zh_CN
+
|
-
| Feature can be tested based on description but extended testing should be done during localization / internationalization testing.  
+
|-
 +
| Feature [http://bugs.meego.com/show_bug.cgi?id=12102 12102] - [FEA] Non-func - Listen music and incoming call
 +
| It shall be possible to hear the ringtone within 0.5s while listening to music playback.
 +
Note: the behaviour depends on the policy defined by device vendor whereas the purpose of this requirement is to define the required performance once above policy is applicable. MeeGo reference UXs policies are defined at the Policy requirements.
 +
|
 +
|-
 +
| Feature [http://bugs.meego.com/show_bug.cgi?id=12103 12103] - [FEA] Non-func - Incoming call indication
 +
| System shall play ringtone for incoming call within 0.5s, when system is in idle state.
 +
|
|}
|}
=== Features Not To Be Tested ===
=== Features Not To Be Tested ===
-
None
+
Some of the NFRs are not realistic anymore for N900 and that's why they are not tested. Here's a list of them:
-
=== Strategy and Approach ===
+
{| border="1"
-
Mozilla Fennec Browser is tested in different test sets. Test sets containing Mozilla Fennec Browser (MFB) related test cases are:
+
!| Feature Name
-
* Sanity Test Set
+
!| Description
-
** Contains few basic MFB test cases that MFB application is part of SW image, can be opened and basic browsing can be made
+
!| Comments
-
* Key Feature Test Set
+
|-
-
** Consists of key features defined by product management. In the begin release 1.1 key features will be used for the test set but after product management defines key features for 1.2 they will be updated to test set. Test set will be run to Preview and Weekly images
+
| Feature [http://bugs.meego.com/show_bug.cgi?id=12092 12092] - [FEA] Non-func - Immediate feedback to user interactions
-
* Feature Basic Acceptance Set (BAT)
+
| System shall respond immediately (within 100ms) to all user actions performed.
-
** 10-30 important browser test cases selected from Basic Feature Test Set. Will be run weekly
+
| We're currently missing a tool for measuring this fast and accurate haptic feedback
-
* Basic Feature Test Set
+
|-
-
** Contains basic feature UI (Functional) test cases for every testable MFB feature. Will be run once for every major milestone
+
| Feature [http://bugs.meego.com/show_bug.cgi?id=12093 12093] - [FEA] Non-func - UI framerate
-
* Extended Feature Test Set
+
| System shall support rendering of animated effects such as transitions at the maximum frame rate supported by the HW. A common reference is minimum 60fps for fluent transitions
-
** Contains bunch of well selected MFB test cases to verify that delivered features (Functional) forming component are working. Will be run once during Feature Development Phase (MM2-MM3) and once in Stabilization Phase (MM3-MM4)
+
| Currently we are missing a proper tool for measurements but that will be hopefully solved
 +
|-
 +
| Feature [http://bugs.meego.com/show_bug.cgi?id=12094 12094] - [FEA] Non-func - Reaction to power off & on
 +
| System shall give immediate feedback when pressing power button on the device
 +
to turn it off. There shall be a visual and possibly other feedback such as
 +
haptic given within 100ms for power switch.
 +
| We're currently missing a tool for measuring this fast and accurate haptic feedback
 +
|-
 +
| Feature [http://bugs.meego.com/show_bug.cgi?id=12098 12098] - [FEA] Non-func - 1080p video decoding performance
 +
| Once 1080p (Full HD) video decoding is supported the system shall provide minimum playback performance of 30FPS, with HW accelerated codecs supported by the system (eg. H.264 and AAC). Video bitrate upto 20Mbps, audio bitrate upto 448kbps.
 +
| Public MeeGo release supports only open source codecs. Not testable
 +
|-
 +
| Feature [http://bugs.meego.com/show_bug.cgi?id=12099 12099] - [FEA] Non-func - Seeking in video playback
 +
| System shall be able to perform seek back- and forward functions within the currently playing video content. System shall resume the playback in new seek position without loosing the visual or audio quality eg. maintaining lip-sync at all times.
 +
| This FEA is tested in Video Player basic feature test set
 +
|-
 +
| Feature [http://bugs.meego.com/show_bug.cgi?id=12100 12100] - [FEA] Non-func - 1080p video encoding performance
 +
| Once 1080p video encoding is supported, the system shall be able to capture video at a minimum of 30FPS using HW-accelerated video codecs present in the system (eg. H.264 and AAC).  Video bitrate upto 20Mbps, audio bitrate upto 448kbps.
 +
| Public MeeGo release supports only open source codecs. Not testable
 +
|}
 +
=== Strategy and Approach ===
 +
==== Tested NFT Areas ====
 +
Non-functional testing (NFT) can be divided in higher level to two different areas: performance and reliability testing. The official NFRs are mostly concentrating to response time but there are also couple ones related to frame rate. Other performance areas and reliability area will be planned and tested based on 1.2 SW maturity status.
 +
Test areas that will be used:
-
Test Set Guideline: http://wiki.meego.com/Quality/TestSetGuideline
+
* Performance testing
 +
** Response Time
 +
*** Response time testing measures the time a system takes to react to given input by end-user; the total time it takes from a user request until response is received
 +
** Frame Rate
 +
*** Measuring how much frames per second (FPS) certain UI graphical actions have. Used especially with video playback
 +
** Power Management
 +
*** Measuring (average) current consumption of the device during a use case / over a period of time
 +
** Data Throughput
 +
*** The (average) rate of successful message delivery over a communication channel in a given period of time
 +
** Memory Consumption
 +
*** Amount of (RAM) memory used during a use case or over a period of time
 +
** Performance Snapshot
 +
*** The purpose of Non-Functional performance snapshot is to give a clear and immediate overview of the software being tested in terms of non-functional errors and efficiency from end user perspective. Test set is consisting of couple test cases from all performance areas defined above
 +
* Reliability testing
 +
** Aging
 +
*** Measures the effect of long term usage for SW reliability and performance
 +
** Iterative
 +
*** Performing multiple iterations of the same test case in order to expose possible problems in sequential runs
-
Note! Nokia Browser team will also perform extensive UI application test set for MFB to ensure that all relevant MFB related bugs are detected and reported.
+
==== Reporting ====
 +
Reporting of NFT will be done to public wiki instead of QA Reports tool. Reporting will focus on trend metrics which will easily show how e.g. response times have been developing. Link to reports will be added here when reporting starts.
-
=== Hardware Platforms ===
+
==== Hardware Platforms ====
* All [http://meego.com/devices/handset/handset-supported-hardware official supported HW's]
* All [http://meego.com/devices/handset/handset-supported-hardware official supported HW's]
=== QA Contact ===
=== QA Contact ===
* [mailto:anssi.takku@nokia.com Anssi Takku]
* [mailto:anssi.takku@nokia.com Anssi Takku]

Latest revision as of 12:11, 31 January 2011

Contents

MeeGo 1.2 HandSet UX Test Plan for System Non-Functional Test Plan

Summary

This is test plan for MeeGo 1.2 HandSet System Non-Functional Testing (NFT)

Features To Be Tested

MeeGo 1.2 release Non-Functional Requirements (NFR) are listed in FeatureZilla

We are not testing all of NFRs because many of them are not realistic anymore for current public HW (N900). Especially when release will be run from memory card which decreases overall performance.

These features will be tested:

Feature Name Description Comments
Feature 12089 - [FEA] Non-func - Application start-up time for pre-started applications System shall start the most frequently used applications (which are pre-started) immediately (within 100ms) so that device feels responsive. Pre-started applications not yet defined. As a basis we will use release 1.1 key applications (Dialer, SMS, Browser, Music Player, and Video Player)
Feature 12090 - [FEA] Non-func - Application start-up time for (non-pre-started) applications System shall start all applications (excluding pre-started applications) within 0.5s so that device feels responsive Pre-started applications not yet defined. As a basis we will use release 1.1 key applications => other are non-pre-started applications
Feature 12091 - [FEA] Non-func - Application start-up time for Browser Browser application shall start and display meego.com page within 3s. This requirement assumes Internet connection is already present and active (WLAN 802.11g).
Feature 12095 - [FEA] Non-func - Cold boot System shall boot up from cold state as quickly as possible. Maximum boot time shall be 30s, from cold state to UI being completely usable.
Feature 12096 - [FEA] Non-func - Complete shut down The system shall power off quickly and without losing any data. The maximum time to complete shut down of the system shall be 10s.
Feature 12097 - [FEA] Non-func - Screen orientation change System shall render the UI in new orientation immediately after changing the device orientation. The UI shall be re-drawn in new orientation within 0.5s in all use scenarios.
Feature 12101 - [FEA] Non-func - Video playback and incoming call It shall be possible to hear the ringtone within 0.5s during an active video playback.

Note: the behaviour depends on the policy defined by device vendor whereas the purpose of this requirement is to define the required performance once above policy is applicable. MeeGo reference UXs policies are defined at the Policy requirements.

Feature 12102 - [FEA] Non-func - Listen music and incoming call It shall be possible to hear the ringtone within 0.5s while listening to music playback.

Note: the behaviour depends on the policy defined by device vendor whereas the purpose of this requirement is to define the required performance once above policy is applicable. MeeGo reference UXs policies are defined at the Policy requirements.

Feature 12103 - [FEA] Non-func - Incoming call indication System shall play ringtone for incoming call within 0.5s, when system is in idle state.

Features Not To Be Tested

Some of the NFRs are not realistic anymore for N900 and that's why they are not tested. Here's a list of them:

Feature Name Description Comments
Feature 12092 - [FEA] Non-func - Immediate feedback to user interactions System shall respond immediately (within 100ms) to all user actions performed. We're currently missing a tool for measuring this fast and accurate haptic feedback
Feature 12093 - [FEA] Non-func - UI framerate System shall support rendering of animated effects such as transitions at the maximum frame rate supported by the HW. A common reference is minimum 60fps for fluent transitions Currently we are missing a proper tool for measurements but that will be hopefully solved
Feature 12094 - [FEA] Non-func - Reaction to power off & on System shall give immediate feedback when pressing power button on the device

to turn it off. There shall be a visual and possibly other feedback such as haptic given within 100ms for power switch.

We're currently missing a tool for measuring this fast and accurate haptic feedback
Feature 12098 - [FEA] Non-func - 1080p video decoding performance Once 1080p (Full HD) video decoding is supported the system shall provide minimum playback performance of 30FPS, with HW accelerated codecs supported by the system (eg. H.264 and AAC). Video bitrate upto 20Mbps, audio bitrate upto 448kbps. Public MeeGo release supports only open source codecs. Not testable
Feature 12099 - [FEA] Non-func - Seeking in video playback System shall be able to perform seek back- and forward functions within the currently playing video content. System shall resume the playback in new seek position without loosing the visual or audio quality eg. maintaining lip-sync at all times. This FEA is tested in Video Player basic feature test set
Feature 12100 - [FEA] Non-func - 1080p video encoding performance Once 1080p video encoding is supported, the system shall be able to capture video at a minimum of 30FPS using HW-accelerated video codecs present in the system (eg. H.264 and AAC). Video bitrate upto 20Mbps, audio bitrate upto 448kbps. Public MeeGo release supports only open source codecs. Not testable

Strategy and Approach

Tested NFT Areas

Non-functional testing (NFT) can be divided in higher level to two different areas: performance and reliability testing. The official NFRs are mostly concentrating to response time but there are also couple ones related to frame rate. Other performance areas and reliability area will be planned and tested based on 1.2 SW maturity status. Test areas that will be used:

  • Performance testing
    • Response Time
      • Response time testing measures the time a system takes to react to given input by end-user; the total time it takes from a user request until response is received
    • Frame Rate
      • Measuring how much frames per second (FPS) certain UI graphical actions have. Used especially with video playback
    • Power Management
      • Measuring (average) current consumption of the device during a use case / over a period of time
    • Data Throughput
      • The (average) rate of successful message delivery over a communication channel in a given period of time
    • Memory Consumption
      • Amount of (RAM) memory used during a use case or over a period of time
    • Performance Snapshot
      • The purpose of Non-Functional performance snapshot is to give a clear and immediate overview of the software being tested in terms of non-functional errors and efficiency from end user perspective. Test set is consisting of couple test cases from all performance areas defined above
  • Reliability testing
    • Aging
      • Measures the effect of long term usage for SW reliability and performance
    • Iterative
      • Performing multiple iterations of the same test case in order to expose possible problems in sequential runs

Reporting

Reporting of NFT will be done to public wiki instead of QA Reports tool. Reporting will focus on trend metrics which will easily show how e.g. response times have been developing. Link to reports will be added here when reporting starts.

Hardware Platforms

QA Contact

Retrieved from "http://wiki.meego.com/Quality/MeeGo1.2HandSetUXTestPlanforSystemNFT"
Personal tools