MeeGo Conference2010/MCTS CSA Future

MeeGo Core Test Suite Current State and future workshop

MeeGo Core Test Suite (MCTS) [1] is set of test components intended for verification of MeeGo Core OS according to test plan [2] covering functional and non-functional (efficiency and reliability) aspects [3]. MCTS is actively used by MeeGo Core OS QA. Currently MCTS offers more than 2000 test cases for various components including component level test cases and system level, E2E type test cases.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test cases are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

Currently we have had contributions to MCTS from different sources and there are lots of variations in methods how test cases are created, test case quality and how test cases are documented. Also there are overlapping test cases e.g. currently BlueZ test cases can be found from three different test components.

For the future of MCTS I see following as key points.

• At the end of the day we shall have on unified MCTS applicable for any reference HW.
• Every test case has been described according to test definition [4]
• MCTS will have only one test component for each library
• MCTS has only one Common library providing services for all MCTS components
• All test cases in MCTS have been developed according to guidelines [5] at:
• API coverage has been defined according to defined methods [6] and published with test assets [7].
• Each component has good descriptions in Test Specification [8]
• Each Component has a test plan [9]

Target of this activity is that MeeGo Developers can trust and rely on MCTS results with any hardware

As addition to these I see one more quite comprehensive element and it is freedom of test developers to choose language they want Python, Perl, C++, and so forth). In order to make usage of MCTS easy for anyone it is important to unify the ways test cases are defined, documented and so forth.

[1] MeeGo Core Test Suite - MCTS - http://gitorious.org/qa-tools/mcts

[2] MeeGo 1.2 Core OS Test Plan - http://wiki.meego.com/Quality/TestPlan/MeeGo_Core_Test_Plan

[3] Test Areas describing quality characteristics - http://wiki.meego.com/Quality/TestAreas

[4] Test definition - http://wiki.meego.com/Test_Packaging/Test_Plan

[5] MCTS Development Guideline - http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline

[6] API analysis for functional test case design - http://wiki.meego.com/Quality/TestSuite MCTS/MCTS_API_analysis)

[7] Coverage analysis spreadsheets - http://gitorious.org/qa-tools/mcts-coverage

[8] Example of test specification of MCTS test suite - http://wiki.meego.com/QualityTestSuiteVideo_Playback_Driver_Test_Specification

[9] Template for test plan - http://wiki.meego.com/CompTestPlanTemplate)

Kindest Regards,

Toropainen