Getting Started with MeeGo.com Testing
WORK IN PROGRESS
The purpose of this wiki is to gather basic information how to get started with Handset testing in MeeGo.com.
What we do in MeeGo.com testing?
Meego Quality: http://wiki.meego.com/Quality
Accounts & Access Rights
Here are described where to register and what mailing lists are needed to get started in MeeGo Handset UXQA testing.
Register into Accounts
Subscribe Mailing Lists
Subscribe following mailing lists.
Mailing Lists Subscriptions from MeeGo.com. Login into MeeGo.com and select 'My Account' and 'Mailing Lists Subscriptions':
- meego-community: A list devoted to cultivating and supporting the diverse MeeGo community
- meego-handset: This list is used by the people involved in management and development of the MeeGo Handset
- meego-releases: Collaboration list on daily activities for Release Engineering and Quality Assurance teams:
Mailing list subscription for MeeGo QA test reporting:
Mailing list for MeeGo ARM releases, requires PMO account:
Join into IRC channels
IRC meeting schedules: http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule
http://webchat.freenode.net/?channels=meego-meeting
Meetings
QA Meeting
http://wiki.meego.com/Quality/meetings
MeeGo Software
Handset UX is tested with different Architectures and each Architecture has different hardware.
SW Repositories
External repostories:
Internal repository:
Needed Hardware and Software
Installing Software
MeeGo 1.1 SW installation:http://meego.com/downloads/releases/1.1/meego-v1.1-handset
Testing Tools
Testing Activites
We test the Handset UX for different architectures and each architecture has different hardware.
Test Areas & Test Set Definitions
Test areas: http://wiki.meego.com/Quality/TestAreas
Test set defintions: http://wiki.meego.com/Quality/TestSetGuideline
Test Case Design Process
Follow this guideline http://wiki.meego.com/Quality/TestDesignProcessAndGuideline for creating high quality test cases for common usage in MeeGo quality assurance teams.
Test Case Template
Test case templates: http://wiki.meego.com/TestCaseTemplate
Test Execution
Spirit of Testing
Gives the right spirit how the different type of testing should be done, guides to determinate the correct perspective of testing and aims that tester should understand when test case should be marked as passed/failed/na etc. Spirit of Testing would also help the tester’s to follow a sequence of logical steps while trying to resolve a conflict during test execution of a specific test set/type with specific examples of what is to be done when random panics occur or when TC fails once and passes other times etc.
Remember End-user Perspective Approach in Testing
Testing is done from end-user perspective. If test case can't be executed directly from UI then test case should be marked as Failed/Blocked/Na. Executing test case using command line, with simulator or other workaround which is not end-user action shall not be used in Handset testing.
Example test case 1: Connect to secure web site which requires certificate installation
- Tester finds out that secure site certificate installation fails.
- Test case verdict is FAILED. Do NOT install certificate using command line!
Example test case 2: Switch Bluetooth on
- Tester finds out that Bluetooth interface can not be set up.
- Test case verdict is FAILED. Do NOT configure Bluetooth interface using command line!
Exception
When testing is started in the early phase of product development eg. Acceptance and Sanity testing, sometimes work around methods are needed to test functionality. If test case verdict is passed because of workaround, then workaround method is commented in the test report.
Example:
- Wifi modem is not working, Ethernet adapter or GPRS needs to be used to test browser
Test Case Verdict
http://wiki.meego.com/Quality/Glossary#Test_case_verdict
Basic rules for test case pass or fail
Functional Testing
- Test Case = Fail when fail in execution two times fail + fail => Fail
- Test Case = PASS if only 1 fail in multiple repetitions, but the failure is mentioned during commenting.
- If there is a UI bug blocking further testing, then BUG is raised
- If any test step fails, if expected outcome / intent of test case is not impacted heavily, then pass, else fail and but BUG ID will be raised.
- If the pre-condition fails, the test cases are marked as “fail”. But if the precondition fails and TC can be executed on consecutive attempts, then only the respective TC should be marked as failed and other TC’s should be tested accordingly.
Non-Functional Testing
- Response time: Three measurements. Median used as a result. Passed when all steps in test case passed successfully and the target was reached (if any and if not separate fail limit is available)
- Load, power management and memory consumption: One measurement. Passed when all steps in test case passed successfully and the target was reached (if any and if not separate fail limit is available)
- If test case cannot be executed (pre-condition or some of the steps fail) because of functional problem status will be “N/A” and explanation shall be given.
Crashes & Reboots Categories
If handset jams, reboots or handset needs to be restarted during test case execution these occurrences shall be counted and reported as Bugs.
- System reset - Handset makes reset and recovers to idle
- System freeze - Handset UX freezes and battery needs to be removed or handset resets after a while
- Application freeze - Application freezes, other functionality is Handset is ok
- SW needs to be installed again - Handset does not boot-up and needs to be installed again
Actions during Random Crashes/Panics
- During testing if random crash or a single time panic occurs (unable to reproduce again), then raise a Bug, but pass the Test case
- If the panic is reproducible consistently, then fail the test case
- If the panic is preventing the test case from being executed, then fail the test case and raise a bug
- If it’s a random crash not related to TC, then raise a bug, but TC is not failed
Bug Reporting
Link collection how to report defects found during Handset UX QA testing, what bug reporting tool is used and how bugs are followed.
Bug Reporting Tool
Bug Reporting How to
- Bug Workflow
- Bug Verification
- When re-opening bug
Items which dhould be added to How to report bugs wiki
- How to choose right component, if tester is not sure which one should be used for the failure?
- Changing Platform to already reported bug?
- Are we using special keywords in handset UX QA?
- What data needs to be attached into bug report: Traces, crashes, reboots?
Test Reporting
After test execution is done and bugs written test results are reported into MeeGo Handset reporting wiki and test results are distributed via e-mail.
Test Reporting Tool
QA test reporting tool: http://qa-reports.meego.com
Test Report Naming
Name test report following test report naming convention http://wiki.meego.com/TestReportTemplateCollection
Test Report Templates
Test Report Storage
Test Report Email
Follow same template in test report e-mail as in test report wiki. Do not attach large documents into test report E-mail.
- Subject for test report e-mail: Follow same naming convention as in test report wiki
Test Report Distribution
Send test report e-mail to MeeGo QA distribution list: 'meego-qa@lists.meego.com'
Link Library