(→Test Case Verdict for Functional Testing) |
(→Test Case Verdict for Non-Functional Testing) |
||
| Line 156: | Line 156: | ||
- If it’s a random crash not related to TC, then raise a bug, but TC is not failed | - If it’s a random crash not related to TC, then raise a bug, but TC is not failed | ||
| - | |||
| - | ==== | + | ==== Non-Functional basic rules for test case pass or fail ==== |
| - | + | ||
| - | + | ||
| - | + | ||
| - | + | ||
| - | + | ||
| - | + | ||
| - | + | ||
| - | + | ||
* Response time: Three measurements. Median used as a result. Passed when all steps in test case passed successfully and the target was reached (if any and if not separate fail limit is available) | * Response time: Three measurements. Median used as a result. Passed when all steps in test case passed successfully and the target was reached (if any and if not separate fail limit is available) | ||
Contents
|
WORK IN PROGRESS
The purpose of this wiki is to gather basic information how to get started with Handset testing in MeeGo.com.
Meego Quality: http://wiki.meego.com/Quality
Here are described where to register and what mailing lists are needed to get started in MeeGo Handset UXQA testing.
Requires authentication credentials from contact person: Nokia: NN Intel: NN
Subscribe following mailing lists.
Mailing Lists Subscriptions from MeeGo.com. Login into MeeGo.com and select 'My Account' and 'Mailing Lists Subscriptions':
Mailing list subscription for MeeGo QA test reporting:
Mailing list for MeeGo ARM releases, requires PMO account:
IRC meeting schedules: http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule
http://wiki.meego.com/Quality/meetings
Handset UX is tested with different Architectures and each Architecture has different hardware.
Internal repository for Weekly releases: https://files.maemo.org/exchange/mercury-6SeNoU/
External repository for Daily releases: http://repo.meego.com/MeeGo/builds
MeeGo 1.1 SW installation:http://meego.com/downloads/releases/1.1/meego-v1.1-handset
We test the Handset UX for different architectures and each architecture has different hardware.
Test areas: http://wiki.meego.com/Quality/TestAreas
Test set defintions: http://wiki.meego.com/Quality/TestSetGuideline
Test desing porcess and guideline: http://wiki.meego.com/Quality/TestDesignProcessAndGuideline
Test case naming convention, descriptive way of naming.
Test case templates: http://wiki.meego.com/TestCaseTemplate
Link from Fan Zhao
Gives the right spirit how the different type of testing should be done, guides to determinate the correct perspective of testing and aims that tester should understand when test case should be marked as passed/failed/na etc. Spirit of Testing would also help the tester’s to follow a sequence of logical steps while trying to resolve a conflict during test execution of a specific test set/type with specific examples of what is to be done when random panics occur or when TC fails once and passes other times etc.
Testing is done from end-user perspective. If test case can't be executed directly from UI then test case should be marked as Failed/Blocked/Na. Executing test case using command line, with simulator or other workaround which is not end-user action shall not be used in Handset testing.
Example test case 1: Connect to secure web site which requires certificate installation
Example test case 2: Switch Bluetooth on
http://wiki.meego.com/Quality/Glossary#Test_case_verdict
- Test Case = Fail when fail in execution two times fail + fail => Fail - Test Case = PASS if only 1 fail in multiple repetitions, but the failure is mentioned during commenting. - If there is a UI bug blocking further testing, then BUG is raised - If any test step fails, if expected outcome / intent of test case is not impacted heavily, then pass, else fail and but BUG ID will be raised.
If the pre-condition fails, the test cases are marked as “fail”. But if the precondition fails and TC can be executed on consecutive attempts, then only the respective TC should be marked as failed and other TC’s should be tested accordingly.
- During testing if random crash or a single time panic occurs (unable to reproduce again), then raise a Bug, but pass the Test case - If the panic is reproducible consistently, then fail the test case - If the panic is preventing the test case from being executed, then fail the test case and raise a bug - If it’s a random crash not related to TC, then raise a bug, but TC is not failed
If handset jams, reboots or handset needs to be restarted during test case execution these occurrences shall be counted and reported as Bugs.
Link collection how to report defects found during Handset UX QA testing, what bug reporting tool is used and how bugs are followed.
Items which dhould be added to How to report bugs wiki
After test execution is done and bugs written test results are reported into MeeGo Handset reporting wiki and test results are distributed via e-mail. H
Create new report into wiki page under correct test set:
New test reporting tool coming later
Name test report following test report naming convention http://wiki.meego.com/TestReportTemplateCollection
Use color coding for grading in the detailed test results table:
| Low risk, most functions work well, minor bugs or limited normal bugs, Run Rate: 100% and Total Pass Rate: over 90% | |
| Medium risk, main function work, normal bugs and limited major bugs, Run Rate: 80% and Total Pass Rate: over 60% | |
| High risk, main functions loss, a few critical/major bugs, Run Rate: less than 80% and Pass Rate: less than 60% | |
| Not available/Not integrated |
Example of Detailed test result table with grading File:Detailed test results.JPG
Follow same template in test report e-mail as in test report wiki. Do not attach large documents into test report E-mail.
Send test report e-mail to MeeGo QA distribution list: 'meego-qa@lists.meego.com'
