Revision as of 08:28, 31 December 2010

!!!!!!!!!!! WORK IN PROGRESS !!!!!!!!!!!

Introduction

Purpose

This document provides a technical architecture and system design for MCTS Test Automation environment. The purpose is to introduce a new subsystem to the existing MeeGo test automation environment (OTS), where external devices can be connected and controlled on test case or test step level. In this context an external device can for example be, a network simulator, a WLAN analyzer or even another DUT. The new subsystem utilizes test tools already in place in MeeGo by extending their functionality, and also introduces additional components to provide needed new functionality. Suitable open-source will also be utilized as much as it is rationale to do. Intended readers for this document are the developers contributing MeeGo QA Tools area and MCTS test asset developers. Contributions to this document are welcome.

Scope

This document first gives an insight to the overall system with some background information before moving to the system architecture and detailed design chapters. Document focuses on solving following design issues:

• Control and interaction between simulators, analyzers and DUT(s)
• Interfaces of different architectural layers and components
• Adaptation of devices to the test automation environment

Glossary

System Overview

Current OTS environment allows to execute test cases in a single DUT. When executing automated functional test cases in a simulated live environment, it is usually required to verify the result from another device. An example from such test case would be a voice call between two or more DUTs. This is a typical master – slave test case, where master DUT is used to initiate MO call, which is then verified in the receiving slave device. Another situation, when test case verdict needs to be verified from external device is for example tests related to audio playback. In this case the result is fetched from audio analyzer for example. This design will introduce new messaging and communication method to the system, which is used to handle interaction between devices. The system overview is presented in Picture 1.

Picture 1 - System Overview

Design Considerations

Assumptions and Dependencies

Any assumptions, dependencies and requirements regarding the system and operating environment are listed here.

• External device connecting to the test automation environment can be any type from any vendor
• Communication
  • Devices need to be controlled from host and target DUT
  • Test case/ testrunner-lite/ single component can communicate with other devices via messaging
• Message types
  • Request-Reply
  • Events/ signals
  • Variable exchange in messages
  • synchronous/ asynchronous messages
• Both parallel and sequential controlling of the devices shall be possible
• Interaction (events) can happen any time between devices
• External devices can be connected to a remote host
• In case of automated execution is not possible, manual execution mode must be available as a backup
• System needs to be able to run in both Linux and Windows (Priority is in Linux)

Design Constraints

Any global limitations or constraints that have a significant impact on the design of the system's software are listed here.

• Usage of proprietary software components is not allowed
• System needs to extensible and distributable
• API for communicating with the devices has to be generic enough covering at least the basic and selected complex uses cases
• Backwards compatibility with the current test definitions MUST be maintained

System Architecture

For controlling multiple devices and executing commands either sequential or parallel, some short of event/mesaging mechanism is clearly needed. Implementing such subsystem from scratch would require too much effort. Thus existing open solutions are studied and utilized for this. Also extending some of the existing test tools (e.g. OTS) with such functionality would make the architecture too complex. To keep architecture modular, the component controlling handling the communication should be its own subsystem, which then will be connected to OTS.

Architectural Design

This section describes the chosen architecture for the system. Also, the alternative architectures, which have been considered are discussed here.

Layered Architecture View

Picture 2 illustrates the layered architecture of the system. NOTE: only the essential components are shown in the picture.

Picture 2 - Layered View

• Test Tools Layer is the top most layer in the architecture. It contains the tools for execution test binaries both in host and device side and recipe for executing commands.
• Test Case Layer contains the test binaries executed on the host side. Test cases itself can contain lots of state information and logic, which would be impossible to handle on tools side. Therefore, a support for host side test cases is provied.
• Device API Layer contains the different APIs, which are used to communicate with external devices. This layer contains an API, which common for all devices and device type specific APIs.
• Messaging Layer is a vertical layer providing the communication and message routing between devices
• Device Adaptation Layer contains the adaptations to the device drivers.

Component Architecture View

Picture 3 illustrates the component view of the architecture. Each component in the diagram represents an individual subsystem (executable, library or data). For simplicity, only the relevant component and interfaces are described. Note: component names are preliminary.

Picture 3 - Component View

test-definition (existing)

• Purpose: Collection of XML schema files, which are used to validate test plan xml files.
• Provides: Means to validate test plan XML syntax
• Uses: -
• Changes Needed:
  • Support for:
    • test step level attributes, to mark, in which device command is executed
    • measurement values
    • fetching additional log files from different devices
  • Test plan syntax needs be to extended by evolving current testdefinition-complex.xsd or by providing new even more complex schema.
• Type: Data

testrunner-lite (existing)

• Purpose: Runs on host test system. Parses test plan XML and executes commands (test binaries) on host, remote device (over ssh) and external devices over messaging service.
• Provides: Universal way to execute test cases, automated or manual.
• Uses:
  • test-defininition to validate test plan XML
  • libxml2 for XML parsing
  • ssh for communicating with the device
  • libcurl for HTTP logging
  • Messaging API for external device communication
• Changes Needed:
  • support for parsing new attributes from test plan XML
  • execute test commands in the device instructed in test plan XML
  • fetch additional log files from devices
  • for test case development, support for simulating external devices is needed
  • send/ receive AMQP messages
• Type: Executable

testrunner (existing)

• Purpose: Executing test binaries manually
• Provides: UI for the manual execution
• Uses:
  • testrunner-lite to execute test plan
  • Messaging API for external device communication
• Changes Needed:
  • send/ receive AMQP messages
• Type: Executable

Messaging API (new)

• Purpose: Hides the complexity of AMQP protocol from the client applications
• Provides: Simple API for handling AMQP messaging
• Uses: AMQP Broker Client API
• Type: Library

Device API(s) (new)

• Purpose: Daemon process, which loads libraries/ plugins communicating with the devices
• Provides: Converts AMQP messages to device API calls
• Uses: Messaging API to produce/ consume AMQP messages
• Type: Executable

Device Adaptation (new)

• Purpose: Calls the actual device driver
• Provides: Communication with the device
• Uses: Function or socket API for up- and downstream communication
• Type: Library

Messaging (Existing - [Apache Qpid])

• Purpose: Implements AMQP
• Provides: Message Broker and language bindings
• Uses: -
• Type: Executable/ library

Sub-system Architecture

Messaging Architecture

For communication between devices, system will be using Advanced Message Queuing Protocol (AMQP).

Selected components for the message exchange are:

• [Apache Qpid]) AMQP broker, protocol and client API's for communicating with the broker.
• For parameters exchange, system will use [JSON]. JSON object is transformed to a string presentation and carried in the AMQP message payload.

Messaging Sub-system

Qpid broker supports currently AMQP 0-10. Qpid provides a C++ version of the Client API for communicating with broker (http://qpid.apache.org/apis/0.8/cpp/html/). [Messaging API] abstracts the the AMQP protocol specific functionality very well. This allows the developers to put their focus on message sending and receiving without needing to worry about the underlying AMQP. Downside of the Qpid is, that it doesn't provide C-language version of the API. This is clearly needed since many of the test tools and cases are implemented in C. Therefore a C-wrapper for the Qpid C++ Messaging API is provided.

Draft version of wrapper can be found from here: http://gitorious.org/qpid-c-wrapper

Picture 4 illustrates the messaging sub-system architecture.

Picture 4 - Messaging sub-system

• Common library provides an interface for creating and extracting messages etc. It uses:
  • JSON library for packing/unpacking arguments to/from AMQP message content
  • Base64 encoding/decoding binary data to / from message
• Client in the picture represents a test case or test tool (e.g. testrunner-lite) which communicates via Qpid
• libqpidc is the C-wrapper for the Qpid C++ Client API

Qpid uses addresses to identify message senders and receiver. Both are create to a specific address which is just a string. Client API supports currently to types of adresses: queues and topics.

• Messages sent to queues are stored to the broker and delivered, when there is a subscriber for the message. Each message is allocated to a one subscriber
• Messages sent to topics are not stored, and delivered to all subscribers

In AMQP, addresses can have a subject and a routing key is use to deliver the message to a correct subscriber.

In this context we can map address type: queue or topic and subject in the following way to achieve the needed messaging patterns:

• address is a name of a service providing some specific functionality e.g. call_service providing services for answering, creating phone calls etc.
  • To send a message to a queue type of address, a request - response messaging pattern is achieved for requesting some service.
  • To send a message to a topic type of address, a publish - subscribe messaging pattern is achieved for requesting some service from multiple services or receiving an event when something has happened.
• subject is a name of the request or event provided by the service.

Address - subject pairs are described with the following syntax: Address / [subject] e.g. call_service/create_phone_call.

Opening and Closing Broker Connection

Picture 5 illustrates the sequence for opening and closing the connection to Qpid broker using the libqpidc. Wrapper function names maps closely to the C++ Messaging API.

Picture 5 - Open and Close Broker Connection

Publish and Subscribe

Picture 6 - Publish - Subscribe

Design Rationale

Rationale, why this approach was chosen.

Design Alternatives

This section describes architectures which have been considered.

Detailed System Design

Test Definition Design

This chapter describes the changes needed to test definition to meet the requirements.

Event Markup in Test Plan XML

Test definition needs to be extended by introducing new markup for events on test plan XML level. This is achieved by defining new event -element. This element can be a child of pre_steps, case, post_steps and get -elements.

Components affected: testrunner-lite, testrunner

Element event can have the following attributes:

Example:

<event type="send" name="event1"/>

Attribute type of the, will be used to define the type of the event. Value can be any of the following:

Send Event

Send event is used to request service from another device via AMQP. This event type causes testrunner-lite to send event to message queue. This event type can have zero or more param child elements.

Example:

<?xml version="1.0" encoding="ISO-8859-1"?>
 <testdefinition version="0.1">
  <suite name="example-suite">
   <set name="example-set" description="example tests">
    <case name="example-case" description="event tests">
     <!-- Send event1 with 2 parameters -->
     <event type="send" name="event1">
      <param type="string">foo</param>
      <param type="number">123</param>
     </event>
    </case>
   </set>
  </suite>
 </testdefinition>

Parameters (param -element) has a mandatory type attribute, which corresponds to [JSON] data types (except binary type, which is not supported be JSON).

Type attribute can have the following values:

• string

<param type="string">foo</param>

• number

<param type="number">123</param>

• boolean

<param type="boolean">true</param>

• object

Object type corresponds to JSON object structure forming key/value pairs. Type object SHALL have 1 or more item child elements, which have a MANDATORY name and type -attributes.

<param type="object">
 <item name="firstname" type="string">John</item>
 <item name="lastname" type="string">Doe</item>
</param>

• array

Array type corresponds to JSON array structure forming ordered collection of values. Type array SHALL have 1 or more item child elements, which have a MANDATORY type -attribute.

<param type="array">
 <item type="string">test</item>
 <item type="number">1111</item>
</param>

• binary

JSON doesn't support presenting binary format. Thus, a binary -type introduced is used to present base64 encoded binary data.

<param type="binary">/9j/4AAQSkZJRgABAQAMKKM3c18pHlu0djQPahC5OrN9JlT1S7msOo/LyLRSUbImhkTGsaNTWgNHYF7oQus17be9ghCEICEIQH/9k=</param>

Wait Event

Event type wait causes testrunner-lite to wait for a certain event. This can be used to synchronize test execution between devices.

<event type="wait" name="event2"/>

• Wait event has a MANDATORY name-attribute, which defines the event to wait and OPTIONAL timeout-attribute defining the max milliseconds to wait for the to receive. If timeout trigger causes event to fail.

<event type="wait" name="event2" timeout="30000"/>

Sequential Execution

With the event support on test plan XML level a sequential test case utilizing multiple devices can be defined as follows:

<?xml version="1.0" encoding="ISO-8859-1"?>
 <testdefinition version="0.1">
  <suite name="example-suite">
   <set name="seq-example-set" description="sequential example tests">
    <case name="seq-example-case" description="sequential event tests">
     <!-- Execute in DUT. -->
     <step>/usr/bin/test1</step> 
     <!-- Send event1 with 2 parameters. -->
     <event type="send" name="event1">
      <param type="string">foo</param>
      <param type="number">123</param>
     </event>
     <!-- Wait for event2. -->
     <event type="wait" name="event2"/>
     <!-- Execute in DUT. -->
     <step>/usr/bin/test2</step> 
    </case>
   </set>
  </suite>
 </testdefinition>

Parallel Execution

To support parallel test execution between two testrunner-lite instances for example, a parallel -element is introduced.

<?xml version="1.0" encoding="ISO-8859-1"?>
<testdefinition version="0.1">
 <suite name="example-suite">
  <set name="paral-example-set" description="parallel example tests">
   <parallel> 
    <case name="paral-example-case1" description="parallel tests">
     <step>/usr/bin/test1</step>
     <step event="wait">event1</step>
     <step event="wait">event2</step>
     <step event="send">myevent1</step> 
     <step event="send">myevent2</step>
    </case>
    <case name="paral-example-case2" description="parallel tests">
     <step event="send">event1</step>
     <step event="send">event2</step> 
     <step event="wait">myevent1</step>
     <step event="wait">myevent2</step>
    </case>
   </parallel>
  </set>
 </suite>
</testdefinition>

Validating Test Environment

System shall support validating the environment before executing any tests. This is needed to ensure that the environment has the required devices to execute test cases. Test environment is described in the test plan XML enclosed in environment -element. Environment can be defined on testdefinition, suite or set level. Environment MUST have one or more capability -element.

• TBC: what kind of capabilities are needed.

Components affected: testrunner-lite, testrunner, qa-reports, (OTS)

• tests.xml

<?xml version="1.0" encoding="ISO-8859-1"?>
 <testdefinition version="0.1">
  <environment>
    <capability>sms</capability>
    <capability>phone_call</capability>
    <capability>power_measurement</capability>
  <environment>
 <suite name="example-suite">
  <!-- set, case ... -->
 </suite>
</testdefinition>

Executing Test Binaries on Host

Test plan XML needs to support steps executed on host. This frees testrunner-lite from responsibility of being aware of state information and other logic in test cases. Logic is then implemented in the test case running on host machine. An OPTIONAL target -attribute will be added to step -element to achieve this. If target-attribute is not provided in the test plan, command is executed in the device to maintain backward compatibility.

Components affected: testrunner-lite

<?xml version="1.0" encoding="ISO-8859-1"?>
 <testdefinition version="0.1">
  <suite name="example-suite">
   <set name="example-set" description="example tests">
    <case name="example-case" description="event tests">
     <step target="host">/usr/bin/test1</step> 
     <step target="target">/usr/bin/test2</step> 
     <step>/usr/bin/test2</step> 
   </case>
  </set>
 </suite>
</testdefinition>

Examples

Test Case: Receive Voice Call

Picture 5 - Receiving voice call

Meetings/ Planning

This section is meant for meeting and planning activities related to this work.

• Workshop on November 30th 2010
• Planning the PoC
• PoC Implementation
• PoC follow-up 2010-11-30