Revision as of 11:07, 23 September 2010

Proposal For MeeGo 1.2. This plan is under review. actively revising and updating

MeeGo Core Overall Test Plan

Introduction

This is overall test plan for MeeGo Core of MeeGo open source project, which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

Objectives

Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

• New features introduced in specific Core OS weekly release are working as specified as a part of system.
• Stakeholders and community get visibility to release quality.
• Validate that relevant bugs are fixed in software release.

For these activities MeeGo Core OS QA follows iteration cycle and process described in Release Engineering’s release timeline at: http://wiki.meego.com/Release_Engineering/Process

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

• Delivered features for MeeGo Core OS are working as specified as a part of system.
• MeeGo Core OS is performing well and is reliable.
• Program maturity statement can be given.

For these activities MeeGo Core OS QA follows release cycle described in Release Engineering’s release timeline at: http://wiki.meego.com/Release_Engineering/Release_Timeline

Test strategy and approach

Introduction

Quality requirements for MCTS

Feature test and System test

Feature Testing

System Testing

Testability

Test cases

Features to Be Tested

Overall the MeeGo Core Testing will cover the MeeGo OS Middlewares layer and MeeGo OS Base layer in MeeGo Architecture:

Specific features to be tested will be aligned with the features under MeeGo Core OS Features product in MeeGo Featurezilla

What will not Be Tested

Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.

• Any proprietary component which is not included in the open source MeeGo release
• Certification testing like: Ofono and Bluetooth

Test Coverage Definition

In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

• Thorough
  • Full testing service for API
  • Target is to have 100% function/method coverage provided by API
  • Parameter coverage defined by using testing techniques as follows:
    • All Pairs (Pair wise)
    • Equivalence Partitioning
    • Boundary Value Analysis
    • Subjective test case selection (according to Expert opinion or similar)

• Average
  • Between Thorough and Light
  • Full API coverage (according to rules for Thorough) for new functionality.
  • Integration level testing for legacy features.

• Light
  • Integration level testing
  • Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

• Not Covered
  • No Middleware testing, covered indirectly by UX Testing
  • There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

Test Coverage Measurement

Component Test Plans and Coverage

Go to MeeGo Core Test Suite for details

Test Cycle

MeeGo Core will be tested from the following different test execution levels.

Hourly Testing

It's for Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

Sanity (Daily) Testing

It's for Trunk. It will run all fully automated test cases for core components and aims to identify the regressions as early as possible. It will be conducted under OTS (Open Test System).

Weekly Testing

Weekly Testing is a test cycle against the weekly images released from distribution team. Usually the test frequency is twice a week, which aligns with distribution's weekly image release cadence.

It's an incremental testing and includes two parts in general: test the basic distribution health to report out regression, test new features when they are ready for testing. We will also do bug verification and regression test which is set of tests to verify that changes from the last build (code enhancements, bug fixes) don't introduce new issues to the previous working code. The purpose of weekly testing is to early test the new features, to track critical/major issues got fixed in timely fashion, and to report out the overall quality status of the whole image so that people could understand where we are.

Full Pass (Milestone) Testing

In the formal test cycle for milestone test, after a new build passed the BAT test, test team will start the full validation testing for it, following test will be involved:

• Functional Test – which will follow the component test plan to run all tests.
• Non Functional Test – which are defined in the individual component test plan.

Regression Testing

This is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.

The regression test will be taken in following test cycle:

• In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
• In the intermediate regular release, only the modified features need to be heavily tested.
• Bug verification on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Test Report

• Send Test report to mailing list
  • Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
  • Distribution Milestone Full Pass Test report
• Archive reports at MeeGo Quality wiki

Environmental Needs

Hardware Platforms

MeeGo v1.1

• Aava Koski
• N900
• MRST CDK
• Netbook

Test enviromental requirements

References

• MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
• MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
• MeeGo Architecture: http://meego.com/developers/meego-architecture
• MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
• MeeGo bugzilla: http://bugs.meego.com