Latest revision as of 12:17, 21 January 2011

Introduction

Purpose

This document provides a technical architecture and system design for MCTS Test Automation environment. The purpose is to introduce a new subsystem to the existing MeeGo test automation environment (OTS), where external devices can be connected and controlled on test case or test step level. In this context an external device can for example be, a network simulator, a WLAN analyzer or even another DUT. The new subsystem utilizes test tools already in place in MeeGo by extending their functionality, and also introduces additional components to provide needed new functionality. Suitable open-source will also be utilized as much as it is rationale to do. Intended readers for this document are the developers contributing MeeGo QA Tools area and MCTS test asset developers.

Scope

This document first gives an insight to the overall system with some background information before moving to the system architecture and detailed design chapters. Document focuses on solving following design issues:

• Control and interaction between simulators, analyzers and DUT(s)
• Interfaces of different architectural layers and components
• Adaptation of devices to the test automation environment

Glossary

System Overview

Current OTS environment allows to execute test cases in a single DUT. When executing automated functional test cases in a simulated live environment, it is usually required to verify the result from another device. An example from such test case would be a voice call between two or more DUTs. This is a typical master – slave test case, where master DUT is used to initiate MO call, which is then verified in the receiving slave device. Another situation, when test case verdict needs to be verified from external device is for example tests related to audio playback. In this case the result is fetched from audio analyzer for example. This design will introduce new messaging and communication method to the system, which is used to handle interaction between devices. The system overview is presented in Picture 1.

Picture 1 - System Overview

Design Considerations

Assumptions and Dependencies

Any assumptions, dependencies and requirements regarding the system and operating environment are listed here.

• External device connecting to the test automation environment can be any type from any vendor
• Communication
  • Devices need to be controlled from host and target DUT
  • Test case/ testrunner-lite/ single component can communicate with other devices via messaging
• Message types
  • Request-Reply
  • Events/ signals
  • Variable exchange in messages
  • synchronous/ asynchronous messages
• Both parallel and sequential controlling of the devices shall be possible
• Interaction (events) can happen any time between devices
• External devices can be connected to a remote host
• In case of automated execution is not possible, manual execution mode must be available as a backup
• System needs to be able to run in both Linux and Windows (Priority is in Linux)

Design Constraints

Any global limitations or constraints that have a significant impact on the design of the system's software are listed here.

• Usage of proprietary software components is not allowed
• System needs to extensible and distributable
• API for communicating with the devices has to be generic enough covering at least the basic and selected complex uses cases
• Backwards compatibility with the current test definitions MUST be maintained

System Architecture

For controlling multiple devices and executing commands either sequential or parallel, some short of event/mesaging mechanism is clearly needed. Implementing such subsystem from scratch would require too much effort. Thus existing open solutions are studied and utilized for this. Also extending some of the existing test tools (e.g. OTS) with such functionality would make the architecture too complex. To keep architecture modular, the component controlling handling the communication should be its own subsystem, which then will be connected to OTS.

Architectural Design

This section describes the chosen architecture for the system. Also, the alternative architectures, which have been considered are discussed here.

Layered Architecture View

Picture 2 illustrates the layered architecture of the system. NOTE: only the essential components are shown in the picture.

Picture 2 - Layered View

• QA Tools Layer is the top most layer in the architecture. It contains the tools for execution test binaries both in host and device side and recipe for executing commands.
• Test Case Layer contains the test binaries executed on the host or DUT side. Test cases itself can contain lots of state information and logic, which would be impossible to handle on tools side. Therefore, a support for host side test cases is provided.
• Device Services Layer contains the different services providing access to some specific device over AMQP Messaging API.
• Messaging Layer is a vertical layer providing the communication and AMQP message routing between devices
• Device Adaptation Layer contains the adaptations to the device drivers.

Component Architecture View

Picture 3 illustrates the component view of the architecture. Each component in the diagram represents an individual subsystem (executable, library or data). For simplicity, only the relevant component and interfaces are described. Note: component names are preliminary.

Picture 3 - Component View

Responsibilities of the component "owned" by MeeGo QA Tools are presented below.

test-definition

• Purpose: Collection of XML schema files, which are used to validate test plan xml files.
• Provides: Means to validate test plan XML syntax
• Uses: -
• Changes Needed:
  • Support for event type of mark-up (and event parameters) in test plan XML
  • Support for executing test binaries on host PC
  • Test plan syntax needs be to extended by evolving current testdefinition-complex.xsd or by providing new even more complex schema.
• Type: Data

testrunner-lite

• Purpose: Runs on host test system. Parses test plan XML and executes commands (test binaries) on host, remote device (over ssh) and external devices over messaging service.
• Provides: Universal way to execute test cases, automated or manual.
• Uses:
  • test-defininition to validate test plan XML
  • libxml2 for XML parsing
  • ssh for communicating with the device
  • libcurl for HTTP logging
  • libcqpid for external device communication
• Changes Needed:
  • support for parsing new elements/ attributes from test plan XML
  • execute test commands in the device instructed in test plan XML
  • for test case development, support for simulating external devices is needed
  • send/ receive AMQP messages using libcqpid
• Type: Executable

testrunner

• Purpose: Executing test binaries manually
• Provides: UI for the manual execution
• Uses:
  • testrunner-lite to execute test plan
  • Qpid C++ Client API for commmunication
• Changes Needed:
  • send/ receive AMQP messages
• Type: Executable

Qpid Client API C-wrapper (libcqpid)

• Purpose: C Client API for Qpid
• Provides: C-wrapper for Qpid C++ Messaging API
• Uses: Messaging namespace from Qpid C++ Client API
• Type: Library

Sub-system Architecture

Messaging Architecture

For communication between devices, system will be using Advanced Message Queuing Protocol (AMQP).

Selected components for the message exchange are:

• [Apache Qpid]) AMQP broker, protocol and client API's for communicating with the broker.
• For parameters exchange, system will use [JSON]. JSON object is transformed to a string presentation and carried in the AMQP message payload.

Messaging Sub-system

Qpid broker supports currently AMQP 0-10. Qpid provides a C++ version of the Client API for communicating with broker (http://qpid.apache.org/apis/0.8/cpp/html/). [Messaging API] abstracts the the AMQP protocol specific functionality very well. This allows the developers to put their focus on message sending and receiving without needing to worry about the underlying AMQP. Downside of the Qpid is, that it doesn't provide C-language version of the API. This is clearly needed since many of the test tools and cases are implemented in C. Therefore a C-wrapper for the Qpid C++ Messaging API is provided.

Draft version of wrapper can be found from here: http://gitorious.org/qpid-c-wrapper

Picture 4 illustrates the messaging sub-system architecture.

Picture 4 - Messaging sub-system

• Common library provides an interface for creating and extracting messages etc. It uses:
  • JSON library for packing/unpacking arguments to/from AMQP message content
  • Base64 encoding/decoding binary data to / from message
• Client in the picture represents a test case or test tool (e.g. testrunner-lite) which communicates via Qpid
• libcqpid is the C-wrapper for the Qpid C++ Client API

Qpid uses addresses to identify message senders and receiver. Both are create to a specific address which is just a string. Client API supports currently to types of adresses: queues and topics.

• Messages sent to queues are stored to the broker and delivered, when there is a subscriber for the message. Each message is allocated to a one subscriber
• Messages sent to topics are not stored, and delivered to all subscribers

In AMQP, addresses can have a subject and a routing key is use to deliver the message to a correct subscriber.

In this context we can map address type: queue or topic and subject in the following way to achieve the needed messaging patterns:

• address is a name of a service providing some specific functionality e.g. call_service providing services for answering, creating phone calls etc.
  • To send a message to a queue type of address, a request - response messaging pattern is achieved for requesting some service.
  • To send a message to a topic type of address, a publish - subscribe messaging pattern is achieved for requesting some service from multiple services or receiving an event when something has happened.
• subject is a name of the request or event provided by the service.

Address - subject pairs are described with the following syntax: Address / [subject] e.g. call_service/create_phone_call.

Opening and Closing Broker Connection

Picture 5 illustrates the sequence for opening and closing the connection to Qpid broker using the libcqpid. Wrapper function names maps closely to the C++ Messaging API.

Picture 5 - Open and Close Broker Connection

Request and Response

Picture 6 illustrates request - response messaging pattern using the libcqpid and Qpid broker. For simplicity, some of the components are left out from the picture. Client, in the picture, represents e.g. a test case executed on host DUT. Service is a device service providing some functionality.

Picture 6 - Request - Response

Pre-condition: Connection and session are created.

1. Service creates a receiver to specified address or address/subject using the session handle. Address is a queue type of in this case. This call can create the queue on-demand or it can be created beforehand using qpid-config
2. Client creates a sender to the same queue
3. Temporary response queue is allocated.
4. Client creates a receiver to the response queue. Service will send response message to this address.
5. Sender is utilized to...
6. ... send message to broker exchange
7. Service utilizes receiver...
8. ... to fetch messages from the queue
9. Service processes request (get reply address and parse arguments)
10. Request is forwarded to the device
11. After request has been processed, service creates a reply message and responds to the client

Device services

With a term device service, in this context, is meant any daemon process which:

• provides some specified functionality accessible through messaging API
• connects and communicates through Qpid broker dedicated for the system

It's an implementation specific detail how this is adapted, but few examples are given:

• testrunner-lite can be started as a daemon. It provides the same functionality through messaging API as currently is provided via Command Line Interface. It uses libcqpid to communicate with broker.
• HAT-control provides the same functionality through messaging API as currently is provided via Command Line Interface. It uses libcqpid to communicate with broker.

Design Rationale

AMQP is chosen because:

• it's an open standard for messaging middleware
• it's rapid
• it's robust (broker based == guaranteed message delivery)

Apache Qpid (implementation of AMQP) is chosen because:

• provides the required messaging scenarios:
  • point-to-point communication
  • one-to-many communication (broadcast)
• easiest to build/ setup of the ones evaluated
• provides a simple messaging API
• implements enough features from the AMQP standard
• it's free from memory leaks
• broker event logging to file, stderr, stdout or syslog

Detailed System Design

Testrunner-lite API

Picture 7 illustrates component view of the testrunner-lite API. Purpose of this API is to provided similar functionality accessible via messaging than currently is available from Command Line Interface.

Picture 7 - Testrunner-lite API

Controlling Devices from Test Plan XML

This chapter describes the changes needed to test definition and testrunner-lite to meet the requirements for controlling devices on test plan XML level.

Event Markup in Test Plan XML

To use event system from testrunner-lite, test definition needs to be extended by introducing new markup for events on test plan XML level. This is achieved by defining new event -element. This element can be a child of pre_steps, case, post_steps and get -elements.

Components affected: testrunner-lite, testrunner

Element event can have the following attributes:

Attribute type, will be used to define the type of the event. Value can be any of the following:

Picture 8 illustrates how different event types in test plan XML mapping to libcqpid calls from testrunner-lite.

Picture 8 - Testrunner-lite event handling

Subscribe Event (Optional)

TBC: Is this type of functionality needed on test plan XML level? This is difficult to implement on testrunner-lite.

Event element with type-attribute having subscribe -value can be used to subscribe some event explicitly, which is then later received.

<?xml version="1.0" encoding="ISO-8859-1"?>
 <testdefinition version="0.1">
  <suite name="example-suite">
   <set name="example-set" description="example tests">
    <case name="example-case" description="event tests">
     <event type="subscribe" resource="example-service"/>
    </case>
   </set>
  </suite>
 </testdefinition>

This will trigger testrunner-lite to create on-demand topic called example-service.

Send Event

Send event is used to request service from another device via Qpid broker. This event type causes testrunner-lite to send request message to queue and optionally wait for response. Send event can have zero or more param child elements.

Example:

<?xml version="1.0" encoding="ISO-8859-1"?>
 <testdefinition version="0.1">
  <suite name="example-suite">
   <set name="example-set" description="example tests">
    <case name="example-case" description="event tests">
     <!-- Send event1 with 2 parameters -->
     <event type="send" resource="service/event1">
      <param name="foo" type="string">foo</param>
      <param name="number" type="number">123</param>
     </event>
    </case>
   </set>
  </suite>
 </testdefinition>

Parameters (param -element) has MANDATORY name and type and attributes, which corresponds to [JSON] data types (except base64 type, which is not supported be JSON).

Type attribute can have the following values:

• string

<param name="test" type="string">foo</param>

• number

<param name="result" type="number">123</param>

• boolean

<param name="enabled" type="boolean">true</param>

• object

Object type corresponds to JSON object structure forming key/value pairs. Type object SHALL have 1 or more item child elements, which have a MANDATORY name and type -attributes.

<param name="person" type="object">
 <item name="firstname" type="string">John</item>
 <item name="lastname" type="string">Doe</item>
</param>

• array

Array type corresponds to JSON array structure forming ordered collection of values. Type array SHALL have 1 or more item child elements, which have a MANDATORY type -attribute.

<param name="person" "type="array">
 <item name="firsname" type="string">Jane</item>
 <item name=""lastname" "number">Doe</item>
</param>

Wait Event

Event type wait causes testrunner-lite to wait for a certain event subscribed earlier. This can be used to synchronize test execution between devices.

<event type="wait" resource="my-service/event2"/>

Wait event has a MANDATORY resource-attribute, which defines the event to wait and OPTIONAL timeout-attribute defining the max milliseconds to wait for the to receive. If timeout trigger causes event to fail.

<event type="wait" resource="my-service/event2" timeout="30000"/>

Sequential Execution

With the event support on test plan XML level a sequential test case utilizing multiple devices can be defined as follows:

<?xml version="1.0" encoding="ISO-8859-1"?>
 <testdefinition version="0.1">
  <suite name="example-suite">
   <set name="seq-example-set" description="sequential example tests">
    <case name="seq-example-case" description="sequential event tests">
     <!-- Execute in DUT. -->
     <step>/usr/bin/test1</step> 
     <!-- Send request with 2 parameters. -->
     <event type="send" resource="my-service/my-event">
       <param name="foo" type="string">foo</param>
       <param name="number" type="number">123</param>
     </event>
     <!-- Wait for event1. -->
     <event type="wait" resource="my-service/event1"/>
     <!-- Execute in DUT. -->
     <step>/usr/bin/test2</step> 
    </case>
   </set>
  </suite>
 </testdefinition>

Parallel Execution

To support parallel test execution between two testrunner-lite instances for example, a parallel -element is introduced.

<?xml version="1.0" encoding="ISO-8859-1"?>
<testdefinition version="0.1">
 <suite name="example-suite">
  <set name="paral-example-set" description="parallel example tests">
   <parallel> 
    <case name="paral-example-case1" description="parallel tests">
     <step>/usr/bin/test1</step>
     <event type="wait" resource="tr-lite-service/event1"/>
     <event type="wait" resource="tr-lite-service/event2"/>
     <event type="send" resource="tr-lite-service/myevent1"/>
     <event type="send" resource="tr-lite-service/myevent2"/>
    </case>
    <case name="paral-example-case2" description="parallel tests">
     <event type="send" resource="tr-lite-service/event1"/>
     <event type="send" resource="tr-lite-service/event2"/>
     <event type="wait" resource="tr-lite-service/myevent1"/>
     <event type="wait" resource="tr-lite-service/myevent2"/>
    </case>
   </parallel>
  </set>
 </suite>
</testdefinition>

Picture 9 - Parallel execution

Validating Test Environment

System shall support validating the environment before executing any tests. This is needed to ensure that the environment has the required devices to execute test cases. Test environment is described in the test plan XML enclosed in environment -element. Environment can be defined on testdefinition, suite or set level. Environment MUST have one or more service -element.

<?xml version="1.0" encoding="ISO-8859-1"?>
 <testdefinition version="0.1">
  <environment>
    <service>sms_service</service>
    <service>call_service</service>
    <service>power_measurement</service>
  <environment>
 <suite name="example-suite">
  <!-- set, case ... -->
 </suite>
</testdefinition>

Validation can be done on OTS or testrunner-lite level. It publishes messages to a predefined service topic, in which the device services have subscribed. Devices reply to the message based on the subject with type of the device. Environment information will be added to results.xml and shown in qa-reports.

Picture 10 - Test environment validation

Executing Test Binaries on Host

Test plan XML needs to support steps executed on host. This frees testrunner-lite from responsibility of being aware of state information and other logic in test cases. Logic is then implemented in the test case running on host machine. An OPTIONAL target -attribute will be added to step -element to achieve this. If target-attribute is not provided in the test plan, command is executed in the device to maintain backward compatibility.

Components affected: testrunner-lite

<?xml version="1.0" encoding="ISO-8859-1"?>
 <testdefinition version="0.1">
  <suite name="example-suite">
   <set name="example-set" description="example tests">
    <case name="example-case" description="event tests">
     <step target="host">/usr/bin/test1</step> 
     <step target="target">/usr/bin/test2</step> 
     <step>/usr/bin/test2</step> 
   </case>
  </set>
 </suite>
</testdefinition>

Controlling Devices from Test Cases

This chapter presents the examples, how framework can be utilized on test case level to control devices.

Example: Receiving Voice Call

Picture 11 illustrates a sequence of a voice call test case between two devices (e.g. between two DUTs). Test case here is a binary executed by testrunner-lite on host or device. This sequence utilizes the publish - subscribe pattern (topic address). For simplicity some of the components e.g. libcqpid are left out from the picture.

Picture 11 - Receiving voice call

1. Session interface is utilized to create receiver by the Device service. In this example topic with a subject is created on-demand utilizing the create-policy. After this call clients can subscribe and send messages to call_service/receive_phone_call.
2. Test case creates a sender to the same address - subject call_service/receive_phone_call
3. Test case utilizes common library to create JSON representation of the arguments and wrap it to a string
4. ... which is then added to AMQP message content and sent
5. Device service receives the message and ...
6. ... processes it based on the subject (receive_phone_call)
7. If the message did not contain reply address it is rejected and test shall fail
8. Valid message causes service to forward message to the device, which initializes underlying device for receiving phone calls
9. The particular 'receive_phone_call -request would respond with a phone number in the message
10. Service utilizes session to create sender to the temporary reply queue
11. Reply message is sent to the client
12. ... which receives it
13. At this point, broker informed that message has been acknowledged
14. Test case needs to be informed somehow, when other end receives phone call. To do this it subscribes itself to call_initiated event
15. Test case continues to initiate phone call
16. When device receives call, it signals the service which sends call_initiated event

The same sequence would be possible utilizing queues. In this case Device service would have supplied "call_service/receive_phone_call; {create: always}" as parameter, when creating receiver.

Open Issues/ Ideas

Test Plan XML

• Is there a need to separate request and response messages/ parameters for send-event?
• Does wait-event need to support parameters?
• Is there a need for subscribe-event? Used to subscribe events explicitly.
• Is there a need to support binary parameters? JSON doesn't support this.

Manual Execution Mode

• Shall we use testrunner(-ui) for this?

Bugzilla

• [Architecture and System Design for BLTS test automation environment]
• [Proof-of-Concept for controlling external devices in test environment]

Meetings/ Planning

This section is meant for meeting and planning activities related to this work.

• Workshop on November 30th 2010
• Planning the PoC
• PoC Implementation
• PoC follow-up 2010-11-30
• Architecture and Proof-Of-Concept Review 2011-01-14