Contents 1 Proposals for test definition (tests.xml) and for result definition (results.xml) !DRAFT! 1.1 1. Not designed to be a test plan 1.2 2. Doesn't support host and device commands in the same tests.xml 1.3 3. Doesn't support device control commands, like reboot 1.4 4. Measurement values 1.5 5. Plug-ins to conductor/testrunner-lite 1.6 6. Coverage Analyze parameters 2 Architecture and effect to components 2.1 Current model 2.2 Option 1 2.3 Option 2 2.4 Option 3 2.5 Option 4 2.6 Responsibilities 3 N900 image creation 4 Bugzilla testing

Proposals for test definition (tests.xml) and for result definition (results.xml) !DRAFT!

1. Not designed to be a test plan

"As a tester, I want to create a test plan which includes test cases from several test definitions".

This can be done by copying parts from the several test definitions and pasting them as a new tests.xml file, but this can cause error situations when the original test cases are changed.
To solve this problem, we can introduce a new tag element link to the test definition: <link testpackage="" filter="" />

One extra option is needed when using XML as test plan, property parameter is for OTS server. Property parameter can be defined for suite, set, case and link elements.

Another option could be to introduce a separated test plan format, but this option didn't get many votes. We want to have only one test definition format.

Example for <link> -tag and for property parameter:

<testdefinition version="1.1">
  <suite name="MeeGo_Core_Set" domain="Core" device_properties="devicegroup:common_n900">
    <set name="Connectivity" description="Connectivity test cases for Core" feature="Connectivity">
      <link testpackage="mcts-connman" filter="testset='Core'" />
      <link testpackage="mcts-bluetooth" filter="testcase='NFT-BT-Obex_file_transfer-NFT'"/>
    </set>
    <set name="Multimedia" description="Multimedia test cases for Core" feature="Multmedia" device_properties="devicegroup:multimedia_n900">
      <link testpackage="mcts-gstreamer" filter="testset='Core_audio_pack_1'" />
      <link testpackage="mcts-gstreamer" filter="testset='Core_audio_pack_2'" />
      <link testpackage="mcts-gstreamer" filter="testset='Core_audio_pack_3'" />
      <link testpackage="mcts-qtm-multimedia" filter="testset='Audio_recording'" />
      <link testpackage="mcts-qtm-multimedia" filter="testset='Video_local_playback'" />
      <link testpackage="mcts-qtm-multimedia" filter="testset='Video_stream_playback'" device_properties="devicegroup:multimedia_n900 devicename:stream"/>
    </set>
    <set name="System" description="System test cases for Core" feature="System">
      <link testpackage="mwts-filesystem-scripts" filter="testcase='NFT-FS-Write_Data_TMP-THRO'" />
      <link testpackage="mwts-filesystem-scripts" filter="testcase='NFT-FS-Read_Data_TMP-THRO'" />
      <link testpackage="mwts-sensors" filter="testset='Core'" />
    </set>
  </suite>
</testdefinition>

This change will effect to the testplanner #9763, testrunner-lite and to OTS.

For OTS, the test definitions have been part of the test assets, but this change would require that the user can give his/her own tests.xml to the OTS.

2. Doesn't support host and device commands in the same tests.xml

Currently tests.xml steps are executed in the host or in the device, but XML doesn't support that there could be steps for host and steps for device.
Extending the current step-element to have parameter target could solve this problem. If the target is not defined the step is executed like before, so the backward compatibility will remain.

This would effect to the testrunner-lite mainly. Host-based commands can be used only then when testrunner-lite is executed as host based.

Example for host and device commands together:

    <set name="Power_management" description="Measure power management during the features" feature="Power">
      <case name="NFT-PWR-Power_concumption_during_MP3_playback" description="Measure power consumption during MP3 playback">
        <step target="host">start_power_measuring audio_playback</step>
        <step target="device">min -ct 'FUTE-Audio-Play_MP3'</step>
        <step target="host">stop_power_measuring</step>
      </case>
      <case name="NFT-PWR-Power_concumption_during_voice_call" description="Measure power consumption during voice call">
        <step target="host">start_power_measuring voice_call</step>
        <step>min -ct 'FUTE-VoiceCall-Create_voice_call'</step>
        <step target="host">stop_power_measuring</step>
      </case>
    </set>

3. Doesn't support device control commands, like reboot

There has been a request for feature that test case could say when to reboot. Device control commands are that kind of commands which testrunner-lite cannot handle, so the responsible component would be then OTS's conductor. This would require some kind of testrunner-lite python API for conductor, or creating python testrunner for conductor.

Anyway, device control commands would be:

• flash, if test case does something irreversible to the device, test case can reflash the device
• reboot, reboots the device and continues to next step
• install, install package from the repository or separated package
• uninstall, uninstall package from the device
• upload, upload files from the host pc to the device
• download, download files from device to the host pc

 <!-- if necessary to reflash the device during the executions -->
 <flash />

 <!-- reboot the device, sshsetup parameter for checking ssh connection -->
 <reboot sshsetup="false" />

 <!-- install new package from the repository/local file system -->
 <install timeout="120">mcts-usb-massstorage</install>

 <!-- install new package from the repository/local file system -->
 <uninstall timeout="120">telepathy-ring</uninstall>

 <!-- upload file(s)/directory from the local host to the device -->
 <put timeout="120" dest="/usr/share/testdata/MP3/">/usr/testdata/MP3/test.mp3</upload>

 <!-- download file(s)/directory to the device -->
 <!-- if 'dest' not defined, downloads to working directory -->
 <get timeout="120" dest="/tmp/">/home/meego/Pictures/*.jpg</download>

4. Measurement values

Test case can measure FPS, CPU, MEM, throughput etc. during the executions. We should specify a measurement value format (XML or CSV) and way how to test case can report those values to testrunner/conductor.

Measurement value format can be simply:
name;value;unit;target;failure;

New tag can be used in the tests.xml to get there measurement values.

Examples

tests.xml

      <case name="NFT-BT-Upload_10MB" description="Upload 10MB file and measure throughput">
        <step>min -ct 'NFT-BT-Upload_10MB'</step>
        <get measurement="true">/var/log/tests/NFT-BT-Upload_10MB.result</get>
      </case>

bluetooth_measurement.txt

bt.upload;1.58;MBit/s;2.2;1.2;

results.xml

     <case name="NFT-BT-Upload_10MB" description="Upload 10MB file and measure throughput" manual="false" insignificant="false" result="PASS">
      <step command="min -ct 'NFT-BT-Upload_10MB'" result="PASS">
       <expected_result>0</expected_result>
       <return_code>0</return_code>
       <start>2010-11-17 09:04:09</start>
       <end>2010-11-17 09:04:09</end>
       <stdout></stdout>
       <stderr></stderr>
      </step>
      <measurement name="bt.upload" value="1.58" unit="MBit/s" target="2.2" fail="1.2" />
     </case>

5. Plug-ins to conductor/testrunner-lite

TODO.

6. Coverage Analyze parameters

Mark Halmagiu requested two new optional parameters to case-element: watchmethod and watchexpression. These elements are used in CovAn tool.

      <case name="NFT-BT-Upload_10MB" description="Upload 10MB file and measure throughput" watchmethod="ltrace" watchexpression="libluetooth3">
        <step></step>
      </case>

Architecture and effect to components

Current model

The figure above shows current flow model of test automation. All components on the left side are running on the host PC.
Tests.xml doesn't include any device controlling stuff, defines only test cases instructions.

Pros

Testrunner executes only tests.xmls

Tests.xml is simple

Cons

Many different components handling the device

No possibility to handle device controlling, like reflashing / rebooting from the testrunner

Option 1

Custom flasher is replaced with wider implementation, which is responsible of all hardware specified stuff like flashing, reboots etc.
Testrunner would have API for python and testrunner would inform conductor when reflashing or rebooting is required.

Pros

Testrunner can handle all other commands than reboot and flashing

Only one testrunner implementation

More detailed status info from the execution

Cons

Reasonable to write extra command handling with C/C++ instead of python

Complex structure

Tests.xml includes information that testrunner cannot handle as standalone

Plugins to testrunner for handling external devices?

Complex tests.xml

Option 2

Using same kind of device control structure than in option 1. Only change would be that testrunner would use directly device controller.

Pros

Only one testrunner implementation

Only one component controls the device

Python code for extra command handling

Device control would be easy

Cons

How testrunner works standalone

Complex structure

Plugins to testrunner for handling external devices?

Complex tests.xml

Option 3

In this option testrunner is replaced with python version. Python testrunner would be plug-in to the conductor and would be easy to replace with own executer.
The device control would remain the same, so it would handle all the HW commands.

Pros

All components are using same language

Easy to maintain, replace and add components

More detailed status info available

Device control would be easy

Cons

Two testrunner

Complex tests.xml

Option 4

Option 4 is pretty much similar than option 3, except tests.xml would stay as it is and using separated format for test plans.
Test plan includes information what to execute (links to tests.xmls) and host base functions. Test plan also includes information how to distribute test cases and how to handle error situations.

Pros

Simple structure, testrunner executes tests.xml and conductor the test plan

Test plan could be executed as standalone in the host machine

Cons

Two xml formats

Responsibilities

N900 image creation

We used Fedora 13 in VirtualBox to create images for N900. Ubuntu 10.10 has a bug for building images, somehow Ubuntu rpm cannot build rpm database correctly.
You can install VirtualBox into the server or some another computer. Http server is used for sharing the image.

1. Download Fedora 13 i386 image

http://mirrors.fedoraproject.org/publiclist/Fedora/13/i386/

2. Install VirtualBox

Add VirtualBox repositories

sudo nano /etc/apt/sources.list.d/virtualbox.list
deb http://download.virtualbox.org/virtualbox/debian maverick non-free

Get cerfitication key and install VirtualBox

wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add -
sudo apt-get update
sudo apt-get install virtualbox-3.2

3. Start VirtualBox

VirtualBox

4. Create new virtual environment for Fedora
5. Attach download image to IDE secondary master
6. Change network adapter to use Bridged Adapter
7. Start virtual machine and install Fedora
8. Install mic2 tools (original instruction http://wiki.meego.com/Image_Creation )

Open terminal

su
yum install wget
cd /etc/yum.repos.d/
wget http://repo.meego.com/MeeGo/tools/repos/fedora/13/meego-tools.repo
yum --nogpgcheck install mic2
rpm -Uvh http://download.meego.com/live/Tools:/Building/openSUSE_11.3/i586/qemu-arm-static-0.12.3-1.2.i586.rpm

9. Download .ks file for N900

cd /root/
wget http://repo.meego.com/MeeGo/builds/1.1.80/1.1.80.6.20101116.1/core/images/meego-core-armv7l-n900/meego-core-armv7l-n900-1.1.80.6.20101116.1.ks

10. Modify .ks file

Add to the package list next packages

eat-device
testrunner-lite-tests

11. Create image for N900

mic-image-creator --config=meego-core-armv7l-n900-1.1.80.6.20101116.1.ks --arch=armv7l --format=raw --cache=meego-core-armv7l --save-kernel --compress-disk-image=bz2

12. Install apache

yum install httpd

Remove default view

rm /etc/httpd/conf.d/welcome.conf

13. Copy image to /var/www/html 14. Open firewall ports

Use Firewall interface from System | Administrator | Firewall

Enable WWW tcp 80

14. Test that webserver is visible from other machine

Bugzilla testing

OTS 0.8.1 release status

13196	[META] OTS 0.8.1 release
13196 ⇐ 9035 [FEA] Support for test package custom distribution models - RELEASED	[META] OTS 0.8.1 release
13196 ⇐ 13135 Change OTS python namespace to match packaging name - VERIFIED	[META] OTS 0.8.1 release
13196 ⇐ 13200 [FEA] input plugin support for ots 0.8 - RESOLVED	[META] OTS 0.8.1 release
13196 ⇐ 13141 Support for device groups and parallel testruns - VERIFIED	[META] OTS 0.8.1 release