Revision as of 12:14, 20 December 2010

!!!!!!!!!!! WORK IN PROGRESS !!!!!!!!!!!

Introduction

Purpose

This document provides a technical architecture and system design for MCTS Test Automation environment. The purpose is to introduce a new subsystem to the existing MeeGo test automation environment (OTS), where external devices can be connected and controlled on test case or test step level. In this context an external device can for example be, a network simulator, a WLAN analyzer or even another DUT. The new subsystem utilizes test tools already in place in MeeGo by extending their functionality, and also introduces additional components to provide needed new functionality. Suitable open-source will also be utilized as much as it is rationale to do. Intended readers for this document are the developers contributing MeeGo QA Tools area and MCTS test asset developers. Contributions to this document are welcome.

Scope

This document first gives an insight to the overall system with some background information before moving to the system architecture and detailed design chapters. Document focuses on solving following design issues:

• Control and interaction between simulators, analyzers and DUT(s)
• Interfaces of different architectural layers and components
• Adaptation of devices to the test automation environment

Glossary

System Overview

Current OTS environment allows to execute test cases in a single DUT. When executing automated functional test cases in a simulated live environment, it is usually required to verify the result from another device. An example from such test case would be a voice call between two or more DUTs. This is a typical master – slave test case, where master DUT is used to initiate MO call, which is then verified in the receiving slave device. Another situation, when test case verdict needs to be verified from external device is for example tests related to audio playback. In this case the result is fetched from audio analyzer for example. This design will introduce new messaging and communication method to the system, which is used to handle interaction between devices. The below picture illustrates the system overview.

System Overview

Design Considerations

Assumptions and Dependencies

Any assumptions, dependencies and requirements regarding the system and operating environment are listed here.

• External device connecting to the test automation environment can be any type from any vendor
• Communication
  • Devices need to be controlled from host and target DUT
  • Test case/ testrunner-lite/ single component can communicate with other devices via messaging
• Message types
  • Request-Reply
  • Events/ signals
  • Variable exchange in messages
  • synchronous/ asynchronous messages
• Both parallel and sequential controlling of the devices shall be possible
• Interaction (events) can happen any time between devices
• External devices can be connected to a remote host
• In case of automated execution is not possible, manual execution mode must be available as a backup
• System needs to be able to run in both Linux and Windows (Priority is in Linux)

Design Constraints

Any global limitations or constraints that have a significant impact on the design of the system's software are listed here.

• Usage of proprietary software components is not allowed
• System needs to extensible and distributable
• API for communicating with the devices has to be generic enough covering at least the basic and selected complex uses cases
• Backwards compatibility with the current test definitions MUST be maintained

System Architecture

For controlling multiple devices and executing commands either sequential or parallel, some short of event/mesaging mechanism is clearly needed. Implementing such subsystem from scratch would require too much effort. Thus existing open solutions are studied and utilized for this. Also extending some of the existing test tools (e.g. OTS) with such functionality would make the architecture too complex. To keep architecture modular, the component controlling handling the communication should be its own subsystem, which then will be connected to OTS.

Architectural Design

This section describes the chosen architecture for the system. Also, the alternative architectures, which have been considered are discussed here.

Layered Architecture View

The below picture illustrates the layered architecture of the system. NOTE: only the essential components are shown in the picture.

Layered View

• Test Tools Layer is the top most layer in the architecture. It contains the tools for execution test binaries both in host and device side and recipe for executing commands.
• Test Case Layer contains the test binaries executed on the host side. Test cases itself can contain lots of state information and logic, which would be impossible to handle on tools side. Therefore, a support for host side test cases is provied.
• Device API Layer contains the different APIs, which are used to communicate with external devices. This layer contains an API, which common for all devices and device type specific APIs.
• Messaging Layer is a vertical layer providing the communication and message routing between devices
• Device Adaptation Layer contains the adaptations to the device drivers.

Component Architecture View

The picture below, illustrates the component view of the architecture. Each component in the diagram represents an individual subsystem (executable, library or data). For simplicity, only the relevant component and interfaces are described. Note: component names are preliminary.

Component View

test-definition (existing)

• Purpose: Collection of XML schema files, which are used to validate test plan xml files.
• Provides: Means to validate test plan XML syntax
• Uses: -
• Changes Needed:
  • Support for:
    • test step level attributes, to mark, in which device command is executed
    • measurement values
    • fetching additional log files from different devices
  • Test plan syntax needs be to extended by evolving current testdefinition-complex.xsd or by providing new even more complex schema.
• Type: Data

testrunner-lite (existing)

• Purpose: Runs on host test system. Parses test plan XML and executes commands (test binaries) on host, remote device (over ssh) and external devices over messaging service.
• Provides: Universal way to execute test cases, automated or manual.
• Uses:
  • test-defininition to validate test plan XML
  • libxml2 for XML parsing
  • ssh for communicating with the device
  • libcurl for HTTP logging
  • Messaging API for external device communication
• Changes Needed:
  • support for parsing new attributes from test plan XML
  • execute test commands in the device instructed in test plan XML
  • fetch additional log files from devices
  • for test case development, support for simulating external devices is needed
  • send/ receive AMQP messages
• Type: Executable

testrunner (existing)

• Purpose: Executing test binaries manually
• Provides: UI for the manual execution
• Uses:
  • testrunner-lite to execute test plan
  • Messaging API for external device communication
• Changes Needed:
  • send/ receive AMQP messages
• Type: Executable

Messaging API (new)

• Purpose: Hides the complexity of AMQP protocol from the client applications
• Provides: Simple API for handling AMQP messaging
• Uses: [OpenAMQ C API]
• Type: Library

Sub-system Architecture

Design Rationale

Rationale, why this approach was chosen.

Design Alternatives

This section describes architectures which have been considered.

Detailed System Design

Meetings/ Planning

This section is meant for meeting and planning activities related to this work.

• Workshop on November 30th 2010
• Planning the PoC
• PoC Implementation