Views
Quality/QA-tools/MCTS test automation design
!!!!!!!!!!! WORK IN PROGRESS !!!!!!!!!!!
Contents |
Introduction
Purpose
This document provides a technical architecture and system design for MCTS Test Automation environment. The purpose is to introduce a new subsystem to the existing MeeGo test automation environment (OTS), where external devices can be connected and controlled on test case or test step level. In this context an external device can for example be, a network simulator, a WLAN analyzer or even another DUT. The new subsystem utilizes test tools already in place in MeeGo by extending their functionality, and also introduces additional components to provide needed new functionality. Suitable open-source will also be utilized as much as it is rationale to do. Intended readers for this document are the developers contributing MeeGo QA Tools area and MCTS test asset developers.
Scope
This document first gives an insight to the overall system with some background information before moving to the system architecture and detailed design chapters. Document focuses on solving following design issues:
- Control and interaction between simulators, analyzers and DUT(s)
- Interfaces of different architectural layers and components
- Adaptation of devices to the test automation environment
Glossary
| Term | Definition |
|---|---|
| DUT | Device Under Test |
| OTS | Link to wiki |
| test-definition | Link to wiki |
| testrunner-lite | Link to wiki |
| Testrunner | Link to wiki |
| AMQP | [Advanced Message Queuing Protocol] |
System Overview
Current OTS environment allows to execute test cases in a single DUT. When executing automated functional test cases in a simulated live environment, it is usually required to verify the result from another device. An example from such test case would be a voice call between two or more DUTs. This is a typical master – slave test case, where master DUT is used to initiate MO call, which is then verified in the receiving slave device. Another situation, when test case verdict needs to be verified from external device is for example tests related to audio playback. In this case the result is fetched from audio analyzer for example. This design will introduce new messaging and communication method to the system, which is used to handle interaction between devices. The system overview is presented in Picture 1.
Design Considerations
Assumptions and Dependencies
Any assumptions, dependencies and requirements regarding the system and operating environment are listed here.
- External device connecting to the test automation environment can be any type from any vendor
- Communication
- Devices need to be controlled from host and target DUT
- Test case/ testrunner-lite/ single component can communicate with other devices via messaging
- Message types
- Request-Reply
- Events/ signals
- Variable exchange in messages
- synchronous/ asynchronous messages
- Both parallel and sequential controlling of the devices shall be possible
- Interaction (events) can happen any time between devices
- External devices can be connected to a remote host
- In case of automated execution is not possible, manual execution mode must be available as a backup
- System needs to be able to run in both Linux and Windows (Priority is in Linux)
Design Constraints
Any global limitations or constraints that have a significant impact on the design of the system's software are listed here.
- Usage of proprietary software components is not allowed
- System needs to extensible and distributable
- API for communicating with the devices has to be generic enough covering at least the basic and selected complex uses cases
- Backwards compatibility with the current test definitions MUST be maintained
System Architecture
For controlling multiple devices and executing commands either sequential or parallel, some short of event/mesaging mechanism is clearly needed. Implementing such subsystem from scratch would require too much effort. Thus existing open solutions are studied and utilized for this. Also extending some of the existing test tools (e.g. OTS) with such functionality would make the architecture too complex. To keep architecture modular, the component controlling handling the communication should be its own subsystem, which then will be connected to OTS.
Architectural Design
This section describes the chosen architecture for the system. Also, the alternative architectures, which have been considered are discussed here.
Layered Architecture View
Picture 2 illustrates the layered architecture of the system. NOTE: only the essential components are shown in the picture.
- Test Tools Layer is the top most layer in the architecture. It contains the tools for execution test binaries both in host and device side and recipe for executing commands.
- Test Case Layer contains the test binaries executed on the host side. Test cases itself can contain lots of state information and logic, which would be impossible to handle on tools side. Therefore, a support for host side test cases is provied.
- Device API Layer contains the different APIs, which are used to communicate with external devices. This layer contains an API, which common for all devices and device type specific APIs.
- Messaging Layer is a vertical layer providing the communication and message routing between devices
- Device Adaptation Layer contains the adaptations to the device drivers.
Component Architecture View
Picture 3 illustrates the component view of the architecture. Each component in the diagram represents an individual subsystem (executable, library or data). For simplicity, only the relevant component and interfaces are described. Note: component names are preliminary.
Responsibilities of the component "owned" by MeeGo QA Tools are presented below.
test-definition
- Purpose: Collection of XML schema files, which are used to validate test plan xml files.
- Provides: Means to validate test plan XML syntax
- Uses: -
- Changes Needed:
- Support for event type of mark-up (and event parameters) in test plan XML
- Support for executing test binaries on host PC
- Test plan syntax needs be to extended by evolving current testdefinition-complex.xsd or by providing new even more complex schema.
- Type: Data
testrunner-lite
- Purpose: Runs on host test system. Parses test plan XML and executes commands (test binaries) on host, remote device (over ssh) and external devices over messaging service.
- Provides: Universal way to execute test cases, automated or manual.
- Uses:
- test-defininition to validate test plan XML
- libxml2 for XML parsing
- ssh for communicating with the device
- libcurl for HTTP logging
- libqpidc for external device communication
- Changes Needed:
- support for parsing new elements/ attributes from test plan XML
- execute test commands in the device instructed in test plan XML
- for test case development, support for simulating external devices is needed
- send/ receive AMQP messages using libqpidc
- Type: Executable
testrunner
- Purpose: Executing test binaries manually
- Provides: UI for the manual execution
- Uses:
- testrunner-lite to execute test plan
- Qpid C++ Client API for commmunication
- Changes Needed:
- send/ receive AMQP messages
- Type: Executable
Qpid Client API C-wrapper (libqpidc)
- Purpose: C Client API for Qpid
- Provides: C-wrapper for Qpid C++ Messaging API
- Uses: Messaging namespace from Qpid C++ Client API
- Type: Library
Sub-system Architecture
Messaging Architecture
For communication between devices, system will be using Advanced Message Queuing Protocol (AMQP).
Selected components for the message exchange are:
- [Apache Qpid]) AMQP broker, protocol and client API's for communicating with the broker.
- For parameters exchange, system will use [JSON]. JSON object is transformed to a string presentation and carried in the AMQP message payload.
Messaging Sub-system
Qpid broker supports currently AMQP 0-10. Qpid provides a C++ version of the Client API for communicating with broker (http://qpid.apache.org/apis/0.8/cpp/html/). [Messaging API] abstracts the the AMQP protocol specific functionality very well. This allows the developers to put their focus on message sending and receiving without needing to worry about the underlying AMQP. Downside of the Qpid is, that it doesn't provide C-language version of the API. This is clearly needed since many of the test tools and cases are implemented in C. Therefore a C-wrapper for the Qpid C++ Messaging API is provided.
Draft version of wrapper can be found from here: http://gitorious.org/qpid-c-wrapper
Picture 4 illustrates the messaging sub-system architecture.
- Common library provides an interface for creating and extracting messages etc. It uses:
- JSON library for packing/unpacking arguments to/from AMQP message content
- Base64 encoding/decoding binary data to / from message
- Client in the picture represents a test case or test tool (e.g. testrunner-lite) which communicates via Qpid
- libqpidc is the C-wrapper for the Qpid C++ Client API
Qpid uses addresses to identify message senders and receiver. Both are create to a specific address which is just a string. Client API supports currently to types of adresses: queues and topics.
- Messages sent to queues are stored to the broker and delivered, when there is a subscriber for the message. Each message is allocated to a one subscriber
- Messages sent to topics are not stored, and delivered to all subscribers
In AMQP, addresses can have a subject and a routing key is use to deliver the message to a correct subscriber.
In this context we can map address type: queue or topic and subject in the following way to achieve the needed messaging patterns:
- address is a name of a service providing some specific functionality e.g. call_service providing services for answering, creating phone calls etc.
- To send a message to a queue type of address, a request - response messaging pattern is achieved for requesting some service.
- To send a message to a topic type of address, a publish - subscribe messaging pattern is achieved for requesting some service from multiple services or receiving an event when something has happened.
- subject is a name of the request or event provided by the service.
Address - subject pairs are described with the following syntax: Address / [subject] e.g. call_service/create_phone_call.
Opening and Closing Broker Connection
Picture 5 illustrates the sequence for opening and closing the connection to Qpid broker using the libqpidc. Wrapper function names maps closely to the C++ Messaging API.
Request and Response
Picture 6 illustrates request - response messaging pattern using the libqpidc and Qpid broker. For simplicity, some of the components are left out from the picture. Client, in the picture, represents e.g. a test case executed on host DUT. Service is a device service providing some functionality.
Pre-condition: Connection and session are created.
- Service creates a receiver to specified address or address/subject using the session handle. Address is a queue type of in this case. This call can create the queue on-demand or it can be created beforehand using qpid-config
- Client creates a sender to the same queue
- Temporary response queue is allocated.
- Client creates a receiver to the response queue. Service will send response message to this address.
- Sender is utilized to...
- ... send message to broker exchange
- Service utilizes receiver...
- ... to fetch messages from the queue
- Service processes request (get reply address and parse arguments)
- Request is forwarded to the device
- After request has been processed, service creates a reply message and responds to the client
Device services
With a term device service, in this context, is meant any daemon process which:
- provides some specified functionality accessible through messaging API
- connects and communicates through Qpid broker dedicated for the system
It's an implementation specific detail how this is adapted, but few examples are given:
- testrunner-lite can be started as a daemon. It provides the same functionality through messaging API as currently is provided via Command Line Interface. It uses libqpidc to communicate with broker.
- HAT-control provides the same functionality through messaging API as currently is provided via Command Line Interface. It uses libqpidc to communicate with broker.
Design Rationale
AMQP is chosen because:
- it's an open standard for messaging middleware
- it's rapid
- it's robust (broker based == guaranteed message delivery)
Apache Qpid (implementation of AMQP) is chosen because:
- provides the required messaging scenarios:
- point-to-point communication
- one-to-many communication (broadcast)
- easiest to build/ setup of the ones evaluated
- provides a simple messaging API
- implements enough features from the AMQP standard
- free from memory leaks
Detailed System Design
Testrunner-lite API
Controlling Devices from Test Plan XML
This chapter describes the changes needed to test definition and testrunner-lite to meet the requirements for controlling devices on test plan XML level.
Event Markup in Test Plan XML
To use event system from testrunner-lite, test definition needs to be extended by introducing new markup for events on test plan XML level. This is achieved by defining new event -element. This element can be a child of pre_steps, case, post_steps and get -elements.
Components affected: testrunner-lite, testrunner
Element event can have the following attributes:
| Attribute | Mandatory/ optional | Description |
|---|---|---|
| type | Mandatory | Defines the event type. |
| resource | Mandatory | Specifies the address / [subject]. For example my-service/my-message |
| timeout | Optional | Fails, the event step, if response is not received within the specified time. |
Attribute type, will be used to define the type of the event. Value can be any of the following:
| Value | Description |
|---|---|
| subscribe | Forces testrunner-lite to subscribe some event sent by a specified service. |
| send | Produces request and sends it to exchange. Optionally waits for response. Used for requesting service from some device. |
| wait | Waits for event. Blocks test run until event is received or timeout occurs. |
Picture 7 illustrates how different event types in test plan XML mapping to libqpidc calls from testrunner-lite.
Subscribe Event
Event element with type-attribute having subscribe -value can be used to subscribe some event, which is later received.
<?xml version="1.0" encoding="ISO-8859-1"?>
<testdefinition version="0.1">
<suite name="example-suite">
<set name="example-set" description="example tests">
<case name="example-case" description="event tests">
<event type="subscribe" resource="example-service"/>
</case>
</set>
</suite>
</testdefinition>
This will trigger testrunner-lite to create on-demand topic called example-service.
Send Event
Send event is used to request service from another device via Qpid broker. This event type causes testrunner-lite to send request message to queue and optionally wait for response. Send event can has a MANDATORY request -element and OPTIONAL response -element. These can have zero or more param child elements.
Example:
<?xml version="1.0" encoding="ISO-8859-1"?>
<testdefinition version="0.1">
<suite name="example-suite">
<set name="example-set" description="example tests">
<case name="example-case" description="event tests">
<!-- Send event1 with 2 parameters -->
<event type="send" resource="service/event1">
<request>
<param name="foo" type="string">foo</param>
<param name="number" type="number">123</param>
</request>
<response>
<param name="result" type="string">ok</param>
</response>
</event>
</case>
</set>
</suite>
</testdefinition>
Parameters (param -element) has MANDATORY name and type and attributes, which corresponds to [JSON] data types (except base64 type, which is not supported be JSON).
Type attribute can have the following values:
- string
<param name="test" type="string">foo</param>
- number
<param name="result" type="number">123</param>
- boolean
<param name="enabled" type="boolean">true</param>
- object
Object type corresponds to JSON object structure forming key/value pairs. Type object SHALL have 1 or more item child elements, which have a MANDATORY name and type -attributes.
<param name="person" type="object"> <item name="firstname" type="string">John</item> <item name="lastname" type="string">Doe</item> </param>
- array
Array type corresponds to JSON array structure forming ordered collection of values. Type array SHALL have 1 or more item child elements, which have a MANDATORY type -attribute.
<param name="person" "type="array"> <item name="firsname" type="string">Jane</item> <item name=""lastname" "number">Doe</item> </param>
- base64
JSON doesn't support presenting binary format. Thus, a base64 -type introduced is used to present base64 encoded binary data.
<param type="base64">/9j/4AAQSkZJRgABAQAMKKM3c18pHlu0djQPahC5OrN9JlT1S7msOo/LyLRSUbImhkTGsaNTWgNHYF7oQus17be9ghCEICEIQH/9k=</param>
Wait Event
Event type wait causes testrunner-lite to wait for a certain event subscribed earlier. This can be used to synchronize test execution between devices.
<event type="wait" resource="my-service/event2"/>
Wait event has a MANDATORY resource-attribute, which defines the event to wait and OPTIONAL timeout-attribute defining the max milliseconds to wait for the to receive. If timeout trigger causes event to fail.
<event type="wait" resource="my-service/event2" timeout="30000"/>
TBC: Does wait event type need to have support for parameters?
Sequential Execution
With the event support on test plan XML level a sequential test case utilizing multiple devices can be defined as follows:
<?xml version="1.0" encoding="ISO-8859-1"?>
<testdefinition version="0.1">
<suite name="example-suite">
<set name="seq-example-set" description="sequential example tests">
<case name="seq-example-case" description="sequential event tests">
<!--- Subscribe to event. --->
<event type="subscribe" resource="my-service/event1"/>
<!-- Execute in DUT. -->
<step>/usr/bin/test1</step>
<!-- Send request with 2 parameters. -->
<event type="send" resource="my-service/my-event">
<request>
<param name="foo" type="string">foo</param>
<param name="number" type="number">123</param>
</request>
</event>
<!-- Wait for event1. -->
<event type="wait" resource="my-service/event1"/>
<!-- Execute in DUT. -->
<step>/usr/bin/test2</step>
</case>
</set>
</suite>
</testdefinition>
Parallel Execution
To support parallel test execution between two testrunner-lite instances for example, a parallel -element is introduced.
<?xml version="1.0" encoding="ISO-8859-1"?>
<testdefinition version="0.1">
<suite name="example-suite">
<set name="paral-example-set" description="parallel example tests">
<parallel>
<case name="paral-example-case1" description="parallel tests">
<step>/usr/bin/test1</step>
<event type="subscribe" resource="tr-lite-service/event1"/>
<event type="subscribe" resource="tr-lite-service/event2"/>
<event type="wait" resource="tr-lite-service/event1"/>
<event type="wait" resource="tr-lite-service/event2"/>
<event type="send" resource="tr-lite-service/myevent1"/>
<event type="send" resource="tr-lite-service/myevent2"/>
</case>
<case name="paral-example-case2" description="parallel tests">
<event type="subscribe" resource="tr-lite-service/myevent1"/>
<event type="subscribe" resource="tr-lite-service/myevent2"/>
<event type="send" resource="tr-lite-service/event1"/>
<event type="send" resource="tr-lite-service/event2"/>
<event type="wait" resource="tr-lite-service/myevent1"/>
<event type="wait" resource="tr-lite-service/myevent2"/>
</case>
</parallel>
</set>
</suite>
</testdefinition>
Validating Test Environment
System shall support validating the environment before executing any tests. This is needed to ensure that the environment has the required devices to execute test cases. Test environment is described in the test plan XML enclosed in environment -element. Environment can be defined on testdefinition, suite or set level. Environment MUST have one or more service -element.
<?xml version="1.0" encoding="ISO-8859-1"?>
<testdefinition version="0.1">
<environment>
<service>sms_service</service>
<service>call_service</service>
<service>power_measurement</service>
<environment>
<suite name="example-suite">
<!-- set, case ... -->
</suite>
</testdefinition>
Validation can be done on OTS or testrunner-lite level. It publishes messages to a predefined service topic, in which the device services have subscribed. Devices reply to the message based on the subject with type of the device. Environment information will be added to results.xml and shown in qa-reports.
Executing Test Binaries on Host
Test plan XML needs to support steps executed on host. This frees testrunner-lite from responsibility of being aware of state information and other logic in test cases. Logic is then implemented in the test case running on host machine. An OPTIONAL target -attribute will be added to step -element to achieve this. If target-attribute is not provided in the test plan, command is executed in the device to maintain backward compatibility.
Components affected: testrunner-lite
<?xml version="1.0" encoding="ISO-8859-1"?>
<testdefinition version="0.1">
<suite name="example-suite">
<set name="example-set" description="example tests">
<case name="example-case" description="event tests">
<step target="host">/usr/bin/test1</step>
<step target="target">/usr/bin/test2</step>
<step>/usr/bin/test2</step>
</case>
</set>
</suite>
</testdefinition>
Controlling Devices from Test Cases
This chapter presents the examples, how framework can be utilized on test case level to control devices.
Example: Receiving Voice Call
Picture 9 illustrates a sequence of a voice call test case between two devices (e.g. between two DUTs). Test case here is a binary executed by testrunner-lite on host or device. This sequence utilizes the publish - subscribe pattern (topic address). For simplicity some of the components e.g. libqpidc are left out from the picture.
- Session interface is utilized to create receiver by the Device service. In this example topic with a subject is created on-demand utilizing the create-policy. After this call clients can subscribe and send messages to call_service/receive_phone_call.
- Test case creates a sender to the same address - subject call_service/receive_phone_call
- Test case utilizes common library to create JSON representation of the arguments and wrap it to a string
- ... which is then added to AMQP message content and sent
- Device service receives the message and ...
- ... processes it based on the subject (receive_phone_call)
- If the message did not contain reply address it is rejected and test shall fail
- Valid message causes service to forward message to the device, which initializes underlying device for receiving phone calls
- The particular 'receive_phone_call -request would respond with a phone number in the message
- Service utilizes session to create sender to the temporary reply queue
- Reply message is sent to the client
- ... which receives it
- At this point, broker informed that message has been acknowledged
- Test case needs to be informed somehow, when other end receives phone call. To do this it subscribes itself to call_initiated event
- Test case continues to initiate phone call
- When device receives call, it signals the service which sends call_initiated event
The same sequence would be possible utilizing queues. In this case Device service would have supplied "call_service/receive_phone_call; {create: always}" as parameter, when creating receiver.
Meetings/ Planning
This section is meant for meeting and planning activities related to this work.