Latest revision as of 12:58, 16 March 2011

Getting Started with MeeGo.com Testing

The purpose of this wiki is to gather basic information how to get started with Handset testing in MeeGo.com.

What we do in MeeGo.com testing?

Meego Quality: Quality

Accounts & Access Rights

Here are described where to register and what mailing lists are needed to get started in MeeGo Handset UXQA testing.

Register into Accounts

• Register into MeeGo.com http://meego.com
• Register into Project Maemo Org (PMO)https://account.projects.maemo.org/
• Create account in Bug reporting tool: http://bugs.meego.com
• Meego SW internal repository https://files.maemo.org/exchange/mercury-6SeNoU/
• Test Management tool Testlink: http://tl.meego.com

Subscribe Mailing Lists

Subscribe following mailing lists.

Mailing Lists Subscriptions from MeeGo.com. Login into MeeGo.com and select 'My Account' and 'Mailing Lists Subscriptions':

• meego-community: A list devoted to cultivating and supporting the diverse MeeGo community
• meego-handset: This list is used by the people involved in management and development of the MeeGo Handset
• meego-releases: Collaboration list on daily activities for Release Engineering and Quality Assurance teams:

Mailing list subscription for MeeGo QA test reporting:

• meego-qa: http://lists.meego.com/listinfo/meego-qa

Mailing list for MeeGo ARM releases, requires PMO account:

• meego-arm: https://projects.maemo.org/cgi-bin/mailman/listinfo/meego-arm

Join into IRC channels

Read carefully IRC guideline: IRC guidelines

Configure Freenode IRC client: http://blog.freenode.net/2008/04/registering-a-channel-on-freenode/ or use Freenode web client: http://webchat.freenode.net/

Meetings

QA and other IRC meeting schedules can be found here: MeeGo-Meeting_IRC_Schedule

QA Meeting

Quick link to QA meeting: http://webchat.freenode.net/?channels=meego-meeting

QA meeting minutes: Quality/Meetings

MeeGo Software

Handset UX is tested with different Architectures and each Architecture has different hardware.

SW Repositories

External repostories:

• Daily images: http://download.meego.com/trunk-daily/builds/
• Weekly images: http://repo.meego.com/MeeGo/builds/

Internal repository:

• Weekly releases: https://files.maemo.org/exchange/mercury-6SeNoU/

Needed Hardware and Software

• Handset
• SD micro card
• USB stick for SD micro card
• Ubuntu release installed into PC
• MeeGo flasher: ARM/N900/tools/flasher

Installing Software

MeeGo 1.1 SW installation:http://meego.com/downloads/releases/1.1/meego-v1.1-handset

• N900 ARM/N900/Install/MMC

Testing Tools

Test Case Storage

Test cases and test sets are designed and stored in Test Link:

http://tl.meego.com/

Test sets are also available in Gitorious:

http://meego.gitorious.org/meego-quality-assurance/handset-ux-tests

Test Planner

Test Planner is a UI tool for creating and maintaining test plans and cases:

http://meegoqatools.wordpress.com/2011/01/12/creating-test-plans-with-meego-testplanner/

Test Runner

Test Runner is a UI tool for running test cases:

http://meegoqatools.wordpress.com/2010/12/13/how-to-do-host-based-testing-part-1/

Testing Activites

We test the Handset UX for different architectures and each architecture has different hardware.

Test Areas & Test Set Definitions

Test areas: Quality/TestAreas

Test set defintions: Quality/TestSetGuideline

Test Case Design Process

Follow this guideline Quality/TestDesignProcessAndGuideline for creating high quality test cases for common usage in MeeGo quality assurance teams.

Test Case Template

Test case templates: Quality/Test case template

Test Execution

Spirit of Testing

Gives the right spirit how the different type of testing should be done, guides to determinate the correct perspective of testing and aims that tester should understand when test case should be marked as passed/failed/na etc. Spirit of Testing would also help the tester’s to follow a sequence of logical steps while trying to resolve a conflict during test execution of a specific test set/type with specific examples of what is to be done when random panics occur or when TC fails once and passes other times etc.

Remember End-user Perspective Approach in Testing

Testing is done from end-user perspective. If test case can't be executed directly from UI then test case should be marked as Failed/Blocked/Na. Executing test case using command line, with simulator or other workaround which is not end-user action shall not be used in Handset testing.

Example test case 1: Connect to secure web site which requires certificate installation

• Tester finds out that secure site certificate installation fails.
• Test case verdict is FAILED. Do NOT install certificate using command line!

Example test case 2: Switch Bluetooth on

• Tester finds out that Bluetooth interface can not be set up.
• Test case verdict is FAILED. Do NOT configure Bluetooth interface using command line!

Exception

When testing is started in the early phase of product development eg. Acceptance and Sanity testing, sometimes work around methods are needed to test functionality. If test case verdict is passed because of workaround, then workaround method is commented in the test report.

Example:

• Wifi modem is not working, Ethernet adapter or GPRS needs to be used to test browser

Test Case Verdict

Quality/Glossary#Test_case_verdict

Basic Rules for Test Case Pass or Fail

Functional Testing

• Test Case = Fail when fail in execution two times fail + fail => Fail
• Test Case = PASS if only 1 fail in multiple repetitions, but the failure is mentioned during commenting.
• If there is a UI bug blocking further testing, then BUG is raised
• If any test step fails, if expected outcome / intent of test case is not impacted heavily, then pass, else fail and but BUG ID will be raised.

• If the pre-condition fails, the test cases are marked as “fail”. But if the precondition fails and TC can be executed on consecutive attempts, then only the respective TC should be marked as failed and other TC’s should be tested accordingly.

Crashes & Reboots Categories

If handset jams, reboots or handset needs to be restarted during test case execution these occurrences shall be counted and reported as Bugs.

• System reset - Handset makes reset and recovers to idle
• System freeze - Handset UX freezes and battery needs to be removed or handset resets after a while
• Application freeze - Application freezes, other functionality is Handset is ok
• SW needs to be installed again - Handset does not boot-up and needs to be installed again

Actions during Random Crashes/Panics

• During testing if random crash or a single time panic occurs (unable to reproduce again), then raise a Bug, but pass the Test case
• If the panic is reproducible consistently, then fail the test case
• If the panic is preventing the test case from being executed, then fail the test case and raise a bug
• If it’s a random crash not related to TC, then raise a bug, but TC is not failed

Bug Reporting

Link collection how to report defects found during Handset UX QA testing, what bug reporting tool is used and how bugs are followed.

Bug Reporting Tool

• Bug reporting tool: http://bugs.meego.com/

Bug Reporting How to

• Bug reporting how to: Quality/How_To_Report_Bugs

• Bug Workflow
• Bug Verification
• When re-opening bug

Test Reporting

After test execution is done and bugs written test results are reported into QA Test Reporting Tool.

QA Test Reporting Tool

http://qa-reports.meego.com

Test Report Feeds

You can subscribe to RSS feeds of newly created reports with any RSS reader.

Quality/QA-tools/QAReports/Basic_Workflow#Subscribing_to_RSS_Feeds