"As a tester, I want to create a test plan which includes test cases from several test definitions".
This can be done by copying parts from the several test definitions and pasting them as a new tests.xml file, but this can cause error situations when the original test cases are changed.
To solve this problem, we can introduce a new tag element link to the test definition: <link testpackage="" filter="" />
One extra option is needed when using XML as test plan, property parameter is for OTS server. Property parameter can be defined for suite, set, case and link elements.
Another option could be to introduce a separated test plan format, but this option didn't get many votes. We want to have only one test definition format.
Example for <link> -tag and for property parameter:
<testdefinition version="1.1"> <suite name="MeeGo_Core_Set" domain="Core" device_properties="devicegroup:common_n900"> <set name="Connectivity" description="Connectivity test cases for Core" feature="Connectivity"> <link testpackage="mcts-connman" filter="testset='Core'" /> <link testpackage="mcts-bluetooth" filter="testcase='NFT-BT-Obex_file_transfer-NFT'"/> </set> <set name="Multimedia" description="Multimedia test cases for Core" feature="Multmedia" device_properties="devicegroup:multimedia_n900"> <link testpackage="mcts-gstreamer" filter="testset='Core_audio_pack_1'" /> <link testpackage="mcts-gstreamer" filter="testset='Core_audio_pack_2'" /> <link testpackage="mcts-gstreamer" filter="testset='Core_audio_pack_3'" /> <link testpackage="mcts-qtm-multimedia" filter="testset='Audio_recording'" /> <link testpackage="mcts-qtm-multimedia" filter="testset='Video_local_playback'" /> <link testpackage="mcts-qtm-multimedia" filter="testset='Video_stream_playback'" device_properties="devicegroup:multimedia_n900 devicename:stream"/> </set> <set name="System" description="System test cases for Core" feature="System"> <link testpackage="mwts-filesystem-scripts" filter="testcase='NFT-FS-Write_Data_TMP-THRO'" /> <link testpackage="mwts-filesystem-scripts" filter="testcase='NFT-FS-Read_Data_TMP-THRO'" /> <link testpackage="mwts-sensors" filter="testset='Core'" /> </set> </suite> </testdefinition>
This change will effect to the testplanner #9763, testrunner-lite and to OTS.
For OTS, the test definitions have been part of the test assets, but this change would require that the user can give his/her own tests.xml to the OTS.
Currently tests.xml steps are executed in the host or in the device, but XML doesn't support that there could be steps for host and steps for device.
Extending the current step-element to have parameter target could solve this problem. If the target is not defined the step is executed like before, so the backward compatibility will remain.
This would effect to the testrunner-lite mainly. Host-based commands can be used only then when testrunner-lite is executed as host based.
Example for host and device commands together:
<set name="Power_management" description="Measure power management during the features" feature="Power"> <case name="NFT-PWR-Power_concumption_during_MP3_playback" description="Measure power consumption during MP3 playback"> <step target="host">start_power_measuring audio_playback</step> <step target="device">min -ct 'FUTE-Audio-Play_MP3'</step> <step target="host">stop_power_measuring</step> </case> <case name="NFT-PWR-Power_concumption_during_voice_call" description="Measure power consumption during voice call"> <step target="host">start_power_measuring voice_call</step> <step>min -ct 'FUTE-VoiceCall-Create_voice_call'</step> <step target="host">stop_power_measuring</step> </case> </set>
There has been a request for feature that test case could say when to reboot. Device control commands are that kind of commands which testrunner-lite cannot handle, so the responsible component would be then OTS's conductor. This would require some kind of testrunner-lite python API for conductor, or creating python testrunner for conductor.
Anyway, device control commands would be:
<!-- if necessary to reflash the device during the executions --> <flash /> <!-- reboot the device, sshsetup parameter for checking ssh connection --> <reboot sshsetup="false" /> <!-- install new package from the repository/local file system --> <install timeout="120">mcts-usb-massstorage</install> <!-- install new package from the repository/local file system --> <uninstall timeout="120">telepathy-ring</uninstall> <!-- upload file(s)/directory from the local host to the device --> <put timeout="120" dest="/usr/share/testdata/MP3/">/usr/testdata/MP3/test.mp3</upload> <!-- download file(s)/directory to the device --> <!-- if 'dest' not defined, downloads to working directory --> <get timeout="120" dest="/tmp/">/home/meego/Pictures/*.jpg</download>
Test case can measure FPS, CPU, MEM, throughput etc. during the executions. We should specify a measurement value format (XML or CSV) and way how to test case can report those values to testrunner/conductor.
Measurement value format can be simply:
New tag can be used in the tests.xml to get there measurement values.
<case name="NFT-BT-Upload_10MB" description="Upload 10MB file and measure throughput"> <step>min -ct 'NFT-BT-Upload_10MB'</step> <get measurement="true">/var/log/tests/NFT-BT-Upload_10MB.result</get> </case>
<case name="NFT-BT-Upload_10MB" description="Upload 10MB file and measure throughput" manual="false" insignificant="false" result="PASS"> <step command="min -ct 'NFT-BT-Upload_10MB'" result="PASS"> <expected_result>0</expected_result> <return_code>0</return_code> <start>2010-11-17 09:04:09</start> <end>2010-11-17 09:04:09</end> <stdout></stdout> <stderr></stderr> </step> <measurement name="bt.upload" value="1.58" unit="MBit/s" target="2.2" fail="1.2" /> </case>
Mark Halmagiu requested two new optional parameters to case-element: watchmethod and watchexpression. These elements are used in CovAn tool.
<case name="NFT-BT-Upload_10MB" description="Upload 10MB file and measure throughput" watchmethod="ltrace" watchexpression="libluetooth3"> <step></step> </case>
The figure above shows current flow model of test automation. All components on the left side are running on the host PC.
Tests.xml doesn't include any device controlling stuff, defines only test cases instructions.
Custom flasher is replaced with wider implementation, which is responsible of all hardware specified stuff like flashing, reboots etc.
Testrunner would have API for python and testrunner would inform conductor when reflashing or rebooting is required.
Using same kind of device control structure than in option 1. Only change would be that testrunner would use directly device controller.
In this option testrunner is replaced with python version. Python testrunner would be plug-in to the conductor and would be easy to replace with own executer.
The device control would remain the same, so it would handle all the HW commands.
Option 4 is pretty much similar than option 3, except tests.xml would stay as it is and using separated format for test plans.
Test plan includes information what to execute (links to tests.xmls) and host base functions. Test plan also includes information how to distribute test cases and how to handle error situations.
We used Fedora 13 in VirtualBox to create images for N900. Ubuntu 10.10 has a bug for building images, somehow Ubuntu rpm cannot build rpm database correctly.
You can install VirtualBox into the server or some another computer. Http server is used for sharing the image.
1. Download Fedora 13 i386 image
2. Install VirtualBox
sudo nano /etc/apt/sources.list.d/virtualbox.list deb http://download.virtualbox.org/virtualbox/debian maverick non-free
wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add - sudo apt-get update sudo apt-get install virtualbox-3.2
3. Start VirtualBox
4. Create new virtual environment for Fedora
5. Attach download image to IDE secondary master
6. Change network adapter to use Bridged Adapter
7. Start virtual machine and install Fedora
8. Install mic2 tools (original instruction http://wiki.meego.com/Image_Creation )
su yum install wget cd /etc/yum.repos.d/ wget http://repo.meego.com/MeeGo/tools/repos/fedora/13/meego-tools.repo yum --nogpgcheck install mic2 rpm -Uvh http://download.meego.com/live/Tools:/Building/openSUSE_11.3/i586/qemu-arm-static-0.12.3-1.2.i586.rpm
9. Download .ks file for N900
cd /root/ wget http://repo.meego.com/MeeGo/builds/1.1.80/184.108.40.206.20101116.1/core/images/meego-core-armv7l-n900/meego-core-armv7l-n900-220.127.116.11.20101116.1.ks
10. Modify .ks file
11. Create image for N900
mic-image-creator --config=meego-core-armv7l-n900-18.104.22.168.20101116.1.ks --arch=armv7l --format=raw --cache=meego-core-armv7l --save-kernel --compress-disk-image=bz2
12. Install apache
yum install httpd
13. Copy image to /var/www/html 14. Open firewall ports
14. Test that webserver is visible from other machine
OTS 0.8.1 release status
|13196||[META] OTS 0.8.1 release|
⇐ 9035 [FEA] Support for test package custom distribution models - RELEASED
⇐ 13135 Change OTS python namespace to match packaging name - VERIFIED
⇐ 13200 [FEA] input plugin support for ots 0.8 - RESOLVED
⇐ 13141 Support for device groups and parallel testruns - VERIFIED