Revision as of 04:10, 27 September 2010

Proposal For MeeGo 1.2. This plan is under review. actively revising and updating

MeeGo Core Overall Test Plan

Introduction

This is overall test plan for MeeGo Core of MeeGo open source project, which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

Objectives

Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

• New features introduced in specific Core OS weekly release are working as specified as a part of system.
• Stakeholders and community get visibility to release quality.
• Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in Release Engineering’s Process

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

• Delivered features for MeeGo Core OS are working as specified as a part of system.
• MeeGo Core OS is performing well and is reliable.
• Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in MCTS Development Guidelines.

For these activities MeeGo Core OS QA follows release cycle described in Release Engineering’s release timeline

Test strategy and approach

Introduction

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

• Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
• Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
• If test cases are available in upstream project they will be used in order to keep maintainability.
• Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
• Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
• MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

• Component tests. Unit/module like test cases verifying API’s of specific component.
• System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by MeeGo QA Tools Team.

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see Handset UX Test Plan.

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in test areas

Development teams and MeeGo Core OS QA

Quality requirements for MCTS

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

• Fast test execution time & feedback
• Enables high automation rate (less resource hungry)
• Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
• Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described MCTS Development Guidelines

Feature test and System test

QA target is to validate MeeGo distribution

• Feature functionality
• System functionality (Interaction and negative scenarios)
• System performance (Data Throughput, Latencies, Framerate, etc.)
• System reliability

Feature Testing

• In this context known also as Component Testing.
• Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
• Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

System Testing

• Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
• Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
• Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
• Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

Testability

Testability of MeeGo 1.2 Core OS features is ensured.

• Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
• Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
• QA does not does not accept that features without proper information to ensure testability are assigned to development.

Relevant Links

• http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
• http://wiki.meego.com/Quality/TestabilityChecklist

Test cases

Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

• Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
• Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
• Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
• The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of MeeGo QA Common Test Design Process and Guidelines. Specifics being:

• Test Cases are designed by test asset developers based on QA Owners input.
• QA Owners input is based on existing features and which have been approved from testability point of view.
• Test Cases are described according to test definition in tests.xml files
• Tests.xml files and are stored to MeeGo Gitorious.
• Common Test Case Template is used when designing test cases.

Features to Be Tested

Overall the MeeGo Core Testing will cover the MeeGo OS Middlewares layer and MeeGo OS Base layer in MeeGo Architecture:

Specific features to be tested will be aligned with the features under MeeGo Core OS Features product in MeeGo Featurezilla

What will not be tested

Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.

• Any proprietary component which is not included in the open source MeeGo release
• Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.

Test Coverage Definition

In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

• Thorough
  • Full testing service for API
  • Target is to have 100% function/method coverage provided by API
  • Parameter coverage defined by using testing techniques as follows:
    • All Pairs (Pair wise)
    • Equivalence Partitioning
    • Boundary Value Analysis
    • Subjective test case selection (according to Expert opinion or similar)

• Average
  • Between Thorough and Light
  • Full API coverage (according to rules for Thorough) for new functionality.
  • Integration level testing for legacy features.

• Light
  • Integration level testing
  • Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

• Not Covered
  • No Middleware testing, covered indirectly by UX Testing
  • There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

Test Coverage Measurement

In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

MCTS API analysis describes methods to be used for test coverage measurement.

Component Test Plans and Coverage

Go to MeeGo Core Test Suite for details

Test Cycle

MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

Hourly Testing

Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

Sanity (Daily) Testing

Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionalities which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

• Set of fully automated test cases for core components
• Set of fully automated system level test cases verifying functionalities from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

• Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
• How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

Weekly Testing

Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence. Weekly testing is incremental testing and target for weekly testing is to:

• Test the new features early
• Bug verification in order to track critical/major bugs got fixed in timely fashion
• Measure regression

New features are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

Regression test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round. Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation. The regression test will be taken in following test cycle:

• In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
• In the intermediate regular release, only the modified features need to be heavily tested.

Bug verification on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

Full Pass (Milestone) Testing

In the formal test cycle for milestone test, after a new build passed the weekly test, test team will start the full validation testing for it, following test will be involved:

• Functional Test – which will follow the component test plan to run all tests.
• Non Functional Test – which are defined in the individual component test plan.

Test Report

• Send Test report to mailing list
  • Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
  • Distribution Milestone Full Pass Test report
• Archive reports at MeeGo Quality wiki

Environmental Needs

Hardware Platforms

MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:

• Aava DV1/DV2
• N900
• MRST CDK
• Netbook

Test enviromental requirements

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

References

• MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
• MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
• MeeGo Architecture: http://meego.com/developers/meego-architecture
• MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
• MeeGo bugzilla: http://bugs.meego.com
• MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release