Quality/QA-tools/Test packaging

Introduction

Test packaging is intended to provide flexible and consistent ways to run tests and get results. You may select testing tools of your choice.

Test packaging is set of simple rules to wrap tests with test plan inside rpm package. Tool support for validating test plan and automating test plan execution is provided with testrunner. Following describes rpm-version of test packaging.

Open Testing Service Architecture

The context of test package usage is described in Open Testing Service (OTS) architecture.

Test packaging rules

A source package with test cases must:

• build binary rpm package with name which ends with "-tests" (this is a test package)

A test package must:

• contain all tests, scripts and configuration files required to run tests
• define dependencies - the ones it tests, the test tools and test data it depends on (if any)
• contain test plan located at /usr/share/<packagename>-tests/tests.xml [1]

For details on creating a test package, see Creating a test package.

Test plan definition

tests.xml describes what test cases this package has. The file must conform to the specification described in Test Definition XML.

Here are values for some of the key fields:

• Level: [Component|Feature|System|Product] - Most common ones for the Meego.com QA use are the Feature (for feature acceptance tests agreed with product management) and System (for Use Case based system tests). Projects might use Component value for the component and component integration tests.
• Type: [Functional|Integration|Performance|Benchmark|Resource utilisation|Throughput|Latency|Framerate|Robustness|Iterative|Low-resource|Long-lasting|Recoverability|Usability] - This is now hybrid of test areas and types currently in use (see Test areas). (Component) Integration testing might be the one on which Projects and Release Engineering are having interest.
• Domain: [Security|Data Management|Software Management|System|Location|Kernel|Personal Information Management|Multimedia|Essentials|Communications|Qt|Graphics] - This is now aligned with architectural domains (see http://meego.com/developers/meego-architecture/meego-architecture-domain-view MeeGo architecture domain view).
• Component: [Accounts|Single Sign-On|Integrity Protection Framework|Certificate Manager|Software Distribution Security|Access Control Framework|Security Adaptation|Content Framework|Package Manager|System Control|Resource Policy|Startup Services|Context Framework|Sensor Framework|Sensor Adaptation|Device Mode Adaptation|Haptics and Vibra Adaptation|Location Framework|Location Adaptation|Linux Kernel|Calendar Engine|Contact Engine|Email Engine|Backup Framework|Synchronization Framework|Imaging and Video Adaptation|Camera Adaptation|UPnP|Gstreamer|Audio Adaptation|Pulse Audio|Base Essentials|IP Telephony|Instant Messaging|Presence|Cellular Framework|ConnMan|Bluetooth|Communication Adaptation|Qt|Qt Mobility|Qt Webkit|Web Runtime|Font Management|Input Adaptation|X11|OpenGL ES|Display and Graphics Adaptation] - This is now aligned with architectural components (see http://meego.com/developers/meego-architecture/meego-architecture-domain-view MeeGo architecture domain view).
• Feature: For this there is various lists and ideas - please suggest and contribute on this.
• Requirement: List of numeric values of the Bugzilla IDs which test case is validating.

Tool support

OTS takes care of automated testing with test packages in cloud. Highest level of automation in this context means one-button test automation triggered with version control update, ending up in test run results in target hardware published.

Tool chain covers all the bits and pieces required for that.