Revision as of 12:49, 30 December 2010 (view source) Rha (Talk | contribs) ← Older edit		Revision as of 12:49, 30 December 2010 (view source) Rha (Talk | contribs) (→Examples) Newer edit →
Line 405:		Line 405:
	= Examples =		= Examples =
-	Test Case: Receive Voice Call	+	== Test Case: Receive Voice Call ==
	= Meetings/ Planning =		= Meetings/ Planning =

---

Revision as of 12:49, 30 December 2010

!!!!!!!!!!! WORK IN PROGRESS !!!!!!!!!!!

Contents 1 Introduction 1.1 Purpose 1.2 Scope 1.3 Glossary 2 System Overview 3 Design Considerations 3.1 Assumptions and Dependencies 3.2 Design Constraints 4 System Architecture 4.1 Architectural Design 4.1.1 Layered Architecture View 4.1.2 Component Architecture View 4.2 Sub-system Architecture 4.2.1 Messaging Architecture 4.2.1.1 Messaging Sub-system 4.2.1.2 Opening and Closing Broker Connection 4.2.1.3 Publish and Subscribe 4.3 Design Rationale 4.4 Design Alternatives 5 Detailed System Design 5.1 Test Definition Design 5.1.1 Event Markup in Test Plan XML 5.1.1.1 Send Event 5.1.1.2 Wait Event 5.1.1.3 Sequential Execution 5.1.1.4 Parallel Execution 5.1.2 Validating Test Environment 5.1.3 Executing Test Binaries on Host 6 Examples 6.1 Test Case: Receive Voice Call 7 Meetings/ Planning

Introduction

Purpose

This document provides a technical architecture and system design for MCTS Test Automation environment. The purpose is to introduce a new subsystem to the existing MeeGo test automation environment (OTS), where external devices can be connected and controlled on test case or test step level. In this context an external device can for example be, a network simulator, a WLAN analyzer or even another DUT. The new subsystem utilizes test tools already in place in MeeGo by extending their functionality, and also introduces additional components to provide needed new functionality. Suitable open-source will also be utilized as much as it is rationale to do. Intended readers for this document are the developers contributing MeeGo QA Tools area and MCTS test asset developers. Contributions to this document are welcome.

Scope

This document first gives an insight to the overall system with some background information before moving to the system architecture and detailed design chapters. Document focuses on solving following design issues:

• Control and interaction between simulators, analyzers and DUT(s)
• Interfaces of different architectural layers and components
• Adaptation of devices to the test automation environment

Glossary

Term	Definition
DUT	Device Under Test
OTS	Link to wiki
test-definition	Link to wiki
testrunner-lite	Link to wiki
Testrunner	Link to wiki
AMQP	[Advanced Message Queuing Protocol]

System Overview

Current OTS environment allows to execute test cases in a single DUT. When executing automated functional test cases in a simulated live environment, it is usually required to verify the result from another device. An example from such test case would be a voice call between two or more DUTs. This is a typical master – slave test case, where master DUT is used to initiate MO call, which is then verified in the receiving slave device. Another situation, when test case verdict needs to be verified from external device is for example tests related to audio playback. In this case the result is fetched from audio analyzer for example. This design will introduce new messaging and communication method to the system, which is used to handle interaction between devices. The system overview is presented in Picture 1.

Design Considerations

Assumptions and Dependencies

Any assumptions, dependencies and requirements regarding the system and operating environment are listed here.

• External device connecting to the test automation environment can be any type from any vendor
• Communication
  • Devices need to be controlled from host and target DUT
  • Test case/ testrunner-lite/ single component can communicate with other devices via messaging
• Message types
  • Request-Reply
  • Events/ signals
  • Variable exchange in messages
  • synchronous/ asynchronous messages
• Both parallel and sequential controlling of the devices shall be possible
• Interaction (events) can happen any time between devices
• External devices can be connected to a remote host
• In case of automated execution is not possible, manual execution mode must be available as a backup
• System needs to be able to run in both Linux and Windows (Priority is in Linux)

Design Constraints

Any global limitations or constraints that have a significant impact on the design of the system's software are listed here.

• Usage of proprietary software components is not allowed
• System needs to extensible and distributable
• API for communicating with the devices has to be generic enough covering at least the basic and selected complex uses cases
• Backwards compatibility with the current test definitions MUST be maintained

System Architecture

For controlling multiple devices and executing commands either sequential or parallel, some short of event/mesaging mechanism is clearly needed. Implementing such subsystem from scratch would require too much effort. Thus existing open solutions are studied and utilized for this. Also extending some of the existing test tools (e.g. OTS) with such functionality would make the architecture too complex. To keep architecture modular, the component controlling handling the communication should be its own subsystem, which then will be connected to OTS.

Architectural Design

This section describes the chosen architecture for the system. Also, the alternative architectures, which have been considered are discussed here.

Layered Architecture View

Picture 2 illustrates the layered architecture of the system. NOTE: only the essential components are shown in the picture.

• Test Tools Layer is the top most layer in the architecture. It contains the tools for execution test binaries both in host and device side and recipe for executing commands.
• Test Case Layer contains the test binaries executed on the host side. Test cases itself can contain lots of state information and logic, which would be impossible to handle on tools side. Therefore, a support for host side test cases is provied.
• Device API Layer contains the different APIs, which are used to communicate with external devices. This layer contains an API, which common for all devices and device type specific APIs.
• Messaging Layer is a vertical layer providing the communication and message routing between devices
• Device Adaptation Layer contains the adaptations to the device drivers.

Component Architecture View

Picture 3 illustrates the component view of the architecture. Each component in the diagram represents an individual subsystem (executable, library or data). For simplicity, only the relevant component and interfaces are described. Note: component names are preliminary.

---

test-definition (existing)

• Purpose: Collection of XML schema files, which are used to validate test plan xml files.
• Provides: Means to validate test plan XML syntax
• Uses: -
• Changes Needed:
  • Support for:
    • test step level attributes, to mark, in which device command is executed
    • measurement values
    • fetching additional log files from different devices
  • Test plan syntax needs be to extended by evolving current testdefinition-complex.xsd or by providing new even more complex schema.
• Type: Data

testrunner-lite (existing)

• Purpose: Runs on host test system. Parses test plan XML and executes commands (test binaries) on host, remote device (over ssh) and external devices over messaging service.
• Provides: Universal way to execute test cases, automated or manual.
• Uses:
  • test-defininition to validate test plan XML
  • libxml2 for XML parsing
  • ssh for communicating with the device
  • libcurl for HTTP logging
  • Messaging API for external device communication
• Changes Needed:
  • support for parsing new attributes from test plan XML
  • execute test commands in the device instructed in test plan XML
  • fetch additional log files from devices
  • for test case development, support for simulating external devices is needed
  • send/ receive AMQP messages
• Type: Executable

testrunner (existing)

• Purpose: Executing test binaries manually
• Provides: UI for the manual execution
• Uses:
  • testrunner-lite to execute test plan
  • Messaging API for external device communication
• Changes Needed:
  • send/ receive AMQP messages
• Type: Executable

Messaging API (new)

• Purpose: Hides the complexity of AMQP protocol from the client applications
• Provides: Simple API for handling AMQP messaging
• Uses: AMQP Broker Client API
• Type: Library

Device API(s) (new)

• Purpose: Daemon process, which loads libraries/ plugins communicating with the devices
• Provides: Converts AMQP messages to device API calls
• Uses: Messaging API to produce/ consume AMQP messages
• Type: Executable

Device Adaptation (new)

• Purpose: Calls the actual device driver
• Provides: Communication with the device
• Uses: Function or socket API for up- and downstream communication
• Type: Library

Messaging (Existing - [Apache Qpid])

• Purpose: Implements AMQP
• Provides: Message Broker and language bindings
• Uses: -
• Type: Executable/ library

Sub-system Architecture

Messaging Architecture

For communication between devices, system will be using Advanced Message Queuing Protocol (AMQP).

Selected components for the message exchange are:

• [Apache Qpid]) AMQP broker, protocol and client API's for communicating with the broker.
• For parameters exchange, system will use [JSON]. JSON object is transformed to a string presentation and carried in the AMQP message payload.

Messaging Sub-system

System uses OpenAMQ as the message broker and C WireAPI for client communication. WireAPI is provided by OpenAMQ and closely maps the function names to AMQP standard. Downside of this API is, that it requires understanding of AMQP, which can be quite complex and test case developers are not expected to have this knowledge. Thus, a messaging API (preliminary name) is provided, which offers a simple interface for connecting to, disconnecting from broker, producing and consuming messages. API is used by test case(s) in host and DUT and test tools. Picture 4 illustrates the messaging sub-system architecture.

Opening and Closing Broker Connection

Publish and Subscribe

Design Rationale

Rationale, why this approach was chosen.

Design Alternatives

This section describes architectures which have been considered.

Detailed System Design

Test Definition Design

This chapter describes the changes needed to test definition to meet the requirements.

Event Markup in Test Plan XML

Test definition needs to be extended by introducing new markup for events on test plan XML level. This is achieved by defining new event -element. This element can be a child of pre_steps, case, post_steps and get -elements.

Components affected: testrunner-lite, testrunner

Element event can have the following attributes:

Attribute	Mandatory/ optional	Description
type	Mandatory	Defines the event type.
name	Mandatory, if type=send or type=wait.	Specifies the name of the event to produce or wait.
timeout	Optional	Can be used to together type=get or type=wait. Fails, if response is not received within the specified time.

Example:

<event type="send" name="event1"/>

Attribute type of the, will be used to define the type of the event. Value can be any of the following:

Value	Description
send	Produces event and sends it to exchange (asynchronous). Used for requesting service from
wait	Waits for event. Blocks test run until event is received or timeout occurs.

Send Event

Send event is used to request service from another device via AMQP. This event type causes testrunner-lite to send event to message queue. This event type can have zero or more param child elements.

Example:

<?xml version="1.0" encoding="ISO-8859-1"?>
 <testdefinition version="0.1">
  <suite name="example-suite">
   <set name="example-set" description="example tests">
    <case name="example-case" description="event tests">
     <!-- Send event1 with 2 parameters -->
     <event type="send" name="event1">
      <param type="string">foo</param>
      <param type="number">123</param>
     </event>
    </case>
   </set>
  </suite>
 </testdefinition>

Parameters (param -element) has a mandatory type attribute, which corresponds to [JSON] data types (except binary type, which is not supported be JSON).

Type attribute can have the following values:

• string

<param type="string">foo</param>

• number

<param type="number">123</param>

• boolean

<param type="boolean">true</param>

• object

Object type corresponds to JSON object structure forming key/value pairs. Type object SHALL have 1 or more item child elements, which have a MANDATORY name and type -attributes.

<param type="object">
 <item name="firstname" type="string">John</item>
 <item name="lastname" type="string">Doe</item>
</param>

• array

Array type corresponds to JSON array structure forming ordered collection of values. Type array SHALL have 1 or more item child elements, which have a MANDATORY type -attribute.

<param type="array">
 <item type="string">test</item>
 <item type="number">1111</item>
</param>

• binary

JSON doesn't support presenting binary format. Thus, a binary -type introduced is used to present base64 encoded binary data.

<param type="binary">/9j/4AAQSkZJRgABAQAMKKM3c18pHlu0djQPahC5OrN9JlT1S7msOo/LyLRSUbImhkTGsaNTWgNHYF7oQus17be9ghCEICEIQH/9k=</param>

Wait Event

Event type wait causes testrunner-lite to wait for a certain event. This can be used to synchronize test execution between devices.

<event type="wait" name="event2"/>

• Wait event has a MANDATORY name-attribute, which defines the event to wait and OPTIONAL timeout-attribute defining the max milliseconds to wait for the to receive. If timeout trigger causes event to fail.

<event type="wait" name="event2" timeout="30000"/>

Sequential Execution

With the event support on test plan XML level a sequential test case utilizing multiple devices can be defined as follows:

<?xml version="1.0" encoding="ISO-8859-1"?>
 <testdefinition version="0.1">
  <suite name="example-suite">
   <set name="seq-example-set" description="sequential example tests">
    <case name="seq-example-case" description="sequential event tests">
     <!-- Execute in DUT. -->
     <step>/usr/bin/test1</step> 
     <!-- Send event1 with 2 parameters. -->
     <event type="send" name="event1">
      <param type="string">foo</param>
      <param type="number">123</param>
     </event>
     <!-- Wait for event2. -->
     <event type="wait" name="event2"/>
     <!-- Execute in DUT. -->
     <step>/usr/bin/test2</step> 
    </case>
   </set>
  </suite>
 </testdefinition>

Parallel Execution

To support parallel test execution between two testrunner-lite instances for example, a parallel -element is introduced.

<?xml version="1.0" encoding="ISO-8859-1"?>
<testdefinition version="0.1">
 <suite name="example-suite">
  <set name="paral-example-set" description="parallel example tests">
   <parallel> 
    <case name="paral-example-case1" description="parallel tests">
     <step>/usr/bin/test1</step>
     <step event="wait">event1</step>
     <step event="wait">event2</step>
     <step event="send">myevent1</step> 
     <step event="send">myevent2</step>
    </case>
    <case name="paral-example-case2" description="parallel tests">
     <step event="send">event1</step>
     <step event="send">event2</step> 
     <step event="wait">myevent1</step>
     <step event="wait">myevent2</step>
    </case>
   </parallel>
  </set>
 </suite>
</testdefinition>

Validating Test Environment

System shall support validating the environment before executing any tests. This is needed to ensure that the environment has the required devices to execute test cases. Test environment is described in the test plan XML enclosed in environment -element. Environment can be defined on testdefinition, suite or set level. Environment MUST have one or more capability -element.

• TBC: what kind of capabilities are needed.

Components affected: testrunner-lite, testrunner, qa-reports, (OTS)

• tests.xml

<?xml version="1.0" encoding="ISO-8859-1"?>
 <testdefinition version="0.1">
  <environment>
    <capability>sms</capability>
    <capability>phone_call</capability>
    <capability>power_measurement</capability>
  <environment>
 <suite name="example-suite">
  <!-- set, case ... -->
 </suite>
</testdefinition>

Executing Test Binaries on Host

Test plan XML needs to support steps executed on host. This frees testrunner-lite from responsibility of being aware of state information and other logic in test cases. Logic is then implemented in the test case running on host machine. An OPTIONAL target -attribute will be added to step -element to achieve this. If target-attribute is not provided in the test plan, command is executed in the device to maintain backward compatibility.

Components affected: testrunner-lite

<?xml version="1.0" encoding="ISO-8859-1"?>
 <testdefinition version="0.1">
  <suite name="example-suite">
   <set name="example-set" description="example tests">
    <case name="example-case" description="event tests">
     <step target="host">/usr/bin/test1</step> 
     <step target="target">/usr/bin/test2</step> 
     <step>/usr/bin/test2</step> 
   </case>
  </set>
 </suite>
</testdefinition>

Examples

Test Case: Receive Voice Call

Meetings/ Planning

This section is meant for meeting and planning activities related to this work.

• Workshop on November 30th 2010
• Planning the PoC
• PoC Implementation
• PoC follow-up 2010-11-30