Meego Wiki
Views

Quality/HandsetUXQAGettingStarted

From MeeGo wiki
< Quality
Revision as of 06:50, 13 December 2010 by Maritavesterinen (Talk | contribs)
Jump to: navigation, search

Contents

Getting Started with MeeGo.com Testing

WORK IN PROGRESS

The purpose of this wiki is to gather basic information how to get started with Handset testing in MeeGo.com.


What we do in MeeGo.com testing?

Meego Quality: http://wiki.meego.com/Quality

Accounts & Access Rights

Here are described where to register and what mailing lists are needed to get started in MeeGo Handset UXQA testing.

Register into Accounts

Requires authentication credentials from contact person: Nokia: NN Intel: NN

Subscribe Mailing Lists

Subscribe following mailing lists.

Mailing Lists Subscriptions from MeeGo.com. Login into MeeGo.com and select 'My Account' and 'Mailing Lists Subscriptions':

  • meego-community: A list devoted to cultivating and supporting the diverse MeeGo community
  • meego-handset: This list is used by the people involved in management and development of the MeeGo Handset
  • meego-releases: Collaboration list on daily activities for Release Engineering and Quality Assurance teams:

Mailing list subscription for MeeGo QA test reporting:

Mailing list for MeeGo ARM releases, requires PMO account:

Join into IRC channels

IRC meeting schedules: http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule

Meetings

QA Meeting

http://wiki.meego.com/Quality/meetings

MeeGo Software

Handset UX is tested with different Architectures and each Architecture has different hardware.

SW Repositories

External repostories:

Internal repository:

Needed Hardware and Software

Installing Software

MeeGo 1.1 SW installation:http://meego.com/downloads/releases/1.1/meego-v1.1-handset

Testing Tools

Testing Activites

We test the Handset UX for different architectures and each architecture has different hardware.

Test Areas & Test Set Definitions

Test areas: http://wiki.meego.com/Quality/TestAreas

Test set defintions: http://wiki.meego.com/Quality/TestSetGuideline

Test Case Design

Test desing porcess and guideline: http://wiki.meego.com/Quality/TestDesignProcessAndGuideline

Test Case Naming

Test case naming convention, descriptive way of naming.


Test Case Template

Test case templates: http://wiki.meego.com/TestCaseTemplate

Test Case Release Process

Link from Fan Zhao

Test Case Review Process

Test Case Change Process

Test Execution

Spirit of Testing

Gives the right spirit how the different type of testing should be done, guides to determinate the correct perspective of testing and aims that tester should understand when test case should be marked as passed/failed/na etc. Spirit of Testing would also help the tester’s to follow a sequence of logical steps while trying to resolve a conflict during test execution of a specific test set/type with specific examples of what is to be done when random panics occur or when TC fails once and passes other times etc.

Remember End-user Perspective Approach in Testing

Testing is done from end-user perspective. If test case can't be executed directly from UI then test case should be marked as Failed/Blocked/Na. Executing test case using command line, with simulator or other workaround which is not end-user action shall not be used in Handset testing.

Example test case 1: Connect to secure web site which requires certificate installation

  • Tester finds out that secure site certificate installation fails.
  • Test case verdict is FAILED. Do NOT install certificate using command line!

Example test case 2: Switch Bluetooth on

  • Tester finds out that Bluetooth interface can not be set up.
  • Test case verdict is FAILED. Do NOT configure Bluetooth interface using command line!
Exception

When testing is started in the early phase of product development eg. Acceptance and Sanity testing, sometimes work around methods are needed to test functionality. If test case verdict is passed because of workaround, then workaround method is commented in the test report.

Example:

  • Wifi modem is not working, Ethernet adapter or GPRS needs to be used to test browser

Test Case Verdict

http://wiki.meego.com/Quality/Glossary#Test_case_verdict

Basic rules for test case pass or fail

Functional Testing
  • Test Case = Fail when fail in execution two times fail + fail => Fail
  • Test Case = PASS if only 1 fail in multiple repetitions, but the failure is mentioned during commenting.
  • If there is a UI bug blocking further testing, then BUG is raised
  • If any test step fails, if expected outcome / intent of test case is not impacted heavily, then pass, else fail and but BUG ID will be raised.
  • If the pre-condition fails, the test cases are marked as “fail”. But if the precondition fails and TC can be executed on consecutive attempts, then only the respective TC should be marked as failed and other TC’s should be tested accordingly.
Non-Functional Testing
  • Response time: Three measurements. Median used as a result. Passed when all steps in test case passed successfully and the target was reached (if any and if not separate fail limit is available)
  • Load, power management and memory consumption: One measurement. Passed when all steps in test case passed successfully and the target was reached (if any and if not separate fail limit is available)
  • If test case cannot be executed (pre-condition or some of the steps fail) because of functional problem status will be “N/A” and explanation shall be given.

Crashes & Reboots Categories

If handset jams, reboots or handset needs to be restarted during test case execution these occurrences shall be counted and reported as Bugs.

  • System reset - Handset makes reset and recovers to idle
  • System freeze - Handset UX freezes and battery needs to be removed or handset resets after a while
  • Application freeze - Application freezes, other functionality is Handset is ok
  • SW needs to be installed again - Handset does not boot-up and needs to be installed again

Actions during Random Crashes/Panics

  • During testing if random crash or a single time panic occurs (unable to reproduce again), then raise a Bug, but pass the Test case
  • If the panic is reproducible consistently, then fail the test case
  • If the panic is preventing the test case from being executed, then fail the test case and raise a bug
  • If it’s a random crash not related to TC, then raise a bug, but TC is not failed

Bug Reporting

Link collection how to report defects found during Handset UX QA testing, what bug reporting tool is used and how bugs are followed.

Bug Reporting Tool

Bug Reporting How to

  • Bug Workflow
  • Bug Verification
  • When re-opening bug

Items which dhould be added to How to report bugs wiki

  • How to choose right component, if tester is not sure which one should be used for the failure?
  • Changing Platform to already reported bug?
  • Are we using special keywords in handset UX QA?
  • What data needs to be attached into bug report: Traces, crashes, reboots?

Test Reporting

After test execution is done and bugs written test results are reported into MeeGo Handset reporting wiki and test results are distributed via e-mail. H


Test Reporting Wiki

Create new report into wiki page under correct test set:

New test reporting tool coming later

Test Report Naming

Name test report following test report naming convention http://wiki.meego.com/TestReportTemplateCollection

Test Report Templates

Detailed Test Results

Use color coding for grading in the detailed test results table:

Low risk, most functions work well, minor bugs or limited normal bugs, Run Rate: 100% and Total Pass Rate: over 90%
Medium risk, main function work, normal bugs and limited major bugs, Run Rate: 80% and Total Pass Rate: over 60%
High risk, main functions loss, a few critical/major bugs, Run Rate: less than 80% and Pass Rate: less than 60%
Not available/Not integrated

Example of Detailed test result table with grading File:Detailed test results.JPG

Test Report Storage

Test Report Email

Follow same template in test report e-mail as in test report wiki. Do not attach large documents into test report E-mail.

  • Subject for test report e-mail: Follow same naming convention as in test report wiki

Test Report Distribution

Send test report e-mail to MeeGo QA distribution list: 'meego-qa@lists.meego.com'

Link Library

Retrieved from "http://wiki.meego.com/Quality/HandsetUXQAGettingStarted"
Personal tools