Views
User:Esmietti
Contents |
Proposals for test definition (tests.xml) and for result definition (results.xml) !DRAFT!
1. Not designed to be a test plan
"As a tester, I want to create a test plan which includes test cases from several test definitions".
This can be done by copying parts from the several test definitions and pasting them as a new tests.xml file, but this can cause error situations when the original test cases are changed.
To solve this problem, we can introduce a new tag element link to the test definition: <link testpackage="" filter="" />
One extra option is needed when using XML as test plan, property parameter is for OTS server. Property parameter can be defined for suite, set, case and link elements.
Another option could be to introduce a separated test plan format, but this option didn't get many votes. We want to have only one test definition format.
Example for <link> -tag and for property parameter:
<testdefinition version="1.1">
<suite name="MeeGo_Core_Set" domain="Core" device_properties="devicegroup:common_n900">
<set name="Connectivity" description="Connectivity test cases for Core" feature="Connectivity">
<link testpackage="mcts-connman" filter="testset='Core'" />
<link testpackage="mcts-bluetooth" filter="testcase='NFT-BT-Obex_file_transfer-NFT'"/>
</set>
<set name="Multimedia" description="Multimedia test cases for Core" feature="Multmedia" device_properties="devicegroup:multimedia_n900">
<link testpackage="mcts-gstreamer" filter="testset='Core_audio_pack_1'" />
<link testpackage="mcts-gstreamer" filter="testset='Core_audio_pack_2'" />
<link testpackage="mcts-gstreamer" filter="testset='Core_audio_pack_3'" />
<link testpackage="mcts-qtm-multimedia" filter="testset='Audio_recording'" />
<link testpackage="mcts-qtm-multimedia" filter="testset='Video_local_playback'" />
<link testpackage="mcts-qtm-multimedia" filter="testset='Video_stream_playback'" device_properties="devicegroup:multimedia_n900 devicename:stream"/>
</set>
<set name="System" description="System test cases for Core" feature="System">
<link testpackage="mwts-filesystem-scripts" filter="testcase='NFT-FS-Write_Data_TMP-THRO'" />
<link testpackage="mwts-filesystem-scripts" filter="testcase='NFT-FS-Read_Data_TMP-THRO'" />
<link testpackage="mwts-sensors" filter="testset='Core'" />
</set>
</suite>
</testdefinition>
This change will effect to the testplanner #9763, testrunner-lite and to OTS.
For OTS, the test definitions have been part of the test assets, but this change would require that the user can give his/her own tests.xml to the OTS.
2. Doesn't support host and device commands in the same tests.xml
Currently tests.xml steps are executed in the host or in the device, but XML doesn't support that there could be steps for host and steps for device.
Extending the current step-element to have parameter target could solve this problem. If the target is not defined the step is executed like before, so the backward compatibility will remain.
This would effect to the testrunner-lite mainly. Host-based commands can be used only then when testrunner-lite is executed as host based.
Example for host and device commands together:
<set name="Power_management" description="Measure power management during the features" feature="Power">
<case name="NFT-PWR-Power_concumption_during_MP3_playback" description="Measure power consumption during MP3 playback">
<step target="host">start_power_measuring audio_playback</step>
<step target="device">min -ct 'FUTE-Audio-Play_MP3'</step>
<step target="host">stop_power_measuring</step>
</case>
<case name="NFT-PWR-Power_concumption_during_voice_call" description="Measure power consumption during voice call">
<step target="host">start_power_measuring voice_call</step>
<step>min -ct 'FUTE-VoiceCall-Create_voice_call'</step>
<step target="host">stop_power_measuring</step>
</case>
</set>
3. Doesn't support device control commands, like reboot
There has been a request for feature that test case could say when to reboot. Device control commands are that kind of commands which testrunner-lite cannot handle, so the responsible component would be then OTS's conductor. This would require some kind of testrunner-lite python API for conductor, or creating python testrunner for conductor.
Anyway, device control commands would be:
- flash, if test case does something irreversible to the device, test case can reflash the device
- reboot, reboots the device and continues to next step
- install, install package from the repository or separated package
- uninstall, uninstall package from the device
- upload, upload files from the host pc to the device
- download, download files from device to the host pc
<!-- if necessary to reflash the device during the executions --> <flash /> <!-- reboot the device, sshsetup parameter for checking ssh connection --> <reboot sshsetup="false" /> <!-- install new package from the repository/local file system --> <install timeout="120">mcts-usb-massstorage</install> <!-- install new package from the repository/local file system --> <uninstall timeout="120">telepathy-ring</uninstall> <!-- upload file(s)/directory from the local host to the device --> <put timeout="120" dest="/usr/share/testdata/MP3/">/usr/testdata/MP3/test.mp3</upload> <!-- download file(s)/directory to the device --> <!-- if 'dest' not defined, downloads to working directory --> <get timeout="120" dest="/tmp/">/home/meego/Pictures/*.jpg</download>
4. Measurement values
Test case can measure FPS, CPU, MEM, throughput etc. during the executions. We should specify a measurement value format (XML or CSV) and way how to test case can report those values to testrunner/conductor.
Measurement value format can be simply:
name;value;unit;target;failure;
New tag can be used in the tests.xml to get there measurement values.
Examples
tests.xml
<case name="NFT-BT-Upload_10MB" description="Upload 10MB file and measure throughput">
<step>min -ct 'NFT-BT-Upload_10MB'</step>
<get measurement="true">/var/log/tests/NFT-BT-Upload_10MB.result</get>
</case>
bluetooth_measurement.txt
bt.upload;1.58;MBit/s;2.2;1.2;
results.xml
<case name="NFT-BT-Upload_10MB" description="Upload 10MB file and measure throughput" manual="false" insignificant="false" result="PASS">
<step command="min -ct 'NFT-BT-Upload_10MB'" result="PASS">
<expected_result>0</expected_result>
<return_code>0</return_code>
<start>2010-11-17 09:04:09</start>
<end>2010-11-17 09:04:09</end>
<stdout></stdout>
<stderr></stderr>
</step>
<measurement name="bt.upload" value="1.58" unit="MBit/s" target="2.2" fail="1.2" />
</case>
5. Plug-ins to conductor/testrunner-lite
TODO.
6. Coverage Analyze parameters
Mark Halmagiu requested two new optional parameters to case-element: watchmethod and watchexpression. These elements are used in CovAn tool.
<case name="NFT-BT-Upload_10MB" description="Upload 10MB file and measure throughput" watchmethod="ltrace" watchexpression="libluetooth3">
<step></step>
</case>
Architecture and effect to components
Current model
The figure above shows current flow model of test automation. All components on the left side are running on the host PC.
Tests.xml doesn't include any device controlling stuff, defines only test cases instructions.
- Pros
- Testrunner executes only tests.xmls
- Tests.xml is simple
- Cons
- Many different components handling the device
- No possibility to handle device controlling, like reflashing / rebooting from the testrunner
Option 1
Custom flasher is replaced with wider implementation, which is responsible of all hardware specified stuff like flashing, reboots etc.
Testrunner would have API for python and testrunner would inform conductor when reflashing or rebooting is required.
- Pros
- Testrunner can handle all other commands than reboot and flashing
- Only one testrunner implementation
- More detailed status info from the execution
- Cons
- Reasonable to write extra command handling with C/C++ instead of python
- Complex structure
- Tests.xml includes information that testrunner cannot handle as standalone
- Plugins to testrunner for handling external devices?
- Complex tests.xml
Option 2
Using same kind of device control structure than in option 1. Only change would be that testrunner would use directly device controller.
- Pros
- Only one testrunner implementation
- Only one component controls the device
- Python code for extra command handling
- Device control would be easy
- Cons
- How testrunner works standalone
- Complex structure
- Plugins to testrunner for handling external devices?
- Complex tests.xml
Option 3
In this option testrunner is replaced with python version. Python testrunner would be plug-in to the conductor and would be easy to replace with own executer.
The device control would remain the same, so it would handle all the HW commands.
- Pros
- All components are using same language
- Easy to maintain, replace and add components
- More detailed status info available
- Device control would be easy
- Cons
- Two testrunner
- Complex tests.xml
Option 4
Option 4 is pretty much similar than option 3, except tests.xml would stay as it is and using separated format for test plans.
Test plan includes information what to execute (links to tests.xmls) and host base functions. Test plan also includes information how to distribute test cases and how to handle error situations.
- Pros
- Simple structure, testrunner executes tests.xml and conductor the test plan
- Test plan could be executed as standalone in the host machine
- Cons
- Two xml formats
Responsibilities
N900 image creation
We used Fedora 13 in VirtualBox to create images for N900. Ubuntu 10.10 has a bug for building images, somehow Ubuntu rpm cannot build rpm database correctly.
You can install VirtualBox into the server or some another computer. Http server is used for sharing the image.
1. Download Fedora 13 i386 image
http://mirrors.fedoraproject.org/publiclist/Fedora/13/i386/
2. Install VirtualBox
- Add VirtualBox repositories
sudo nano /etc/apt/sources.list.d/virtualbox.list deb http://download.virtualbox.org/virtualbox/debian maverick non-free
- Get cerfitication key and install VirtualBox
wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add - sudo apt-get update sudo apt-get install virtualbox-3.2
3. Start VirtualBox
VirtualBox
4. Create new virtual environment for Fedora
5. Attach download image to IDE secondary master
6. Change network adapter to use Bridged Adapter
7. Start virtual machine and install Fedora
8. Install mic2 tools (original instruction http://wiki.meego.com/Image_Creation )
- Open terminal
su yum install wget cd /etc/yum.repos.d/ wget http://repo.meego.com/MeeGo/tools/repos/fedora/13/meego-tools.repo yum --nogpgcheck install mic2 rpm -Uvh http://download.meego.com/live/Tools:/Building/openSUSE_11.3/i586/qemu-arm-static-0.12.3-1.2.i586.rpm
9. Download .ks file for N900
cd /root/ wget http://repo.meego.com/MeeGo/builds/1.1.80/1.1.80.6.20101116.1/core/images/meego-core-armv7l-n900/meego-core-armv7l-n900-1.1.80.6.20101116.1.ks
10. Modify .ks file
- Add to the package list next packages
eat-device testrunner-lite-tests
11. Create image for N900
mic-image-creator --config=meego-core-armv7l-n900-1.1.80.6.20101116.1.ks --arch=armv7l --format=raw --cache=meego-core-armv7l --save-kernel --compress-disk-image=bz2
12. Install apache
yum install httpd
- Remove default view
rm /etc/httpd/conf.d/welcome.conf
13. Copy image to /var/www/html 14. Open firewall ports
- Use Firewall interface from System | Administrator | Firewall
- Enable WWW tcp 80
14. Test that webserver is visible from other machine
Bugzilla testing
OTS 0.8.1 release status
| 13196 | [META] OTS 0.8.1 release |
⇐ 9035 [FEA] Support for test package custom distribution models - RELEASED | |
⇐ 13135 Change OTS python namespace to match packaging name - VERIFIED | |
⇐ 13200 [FEA] input plugin support for ots 0.8 - RESOLVED | |
⇐ 13141 Support for device groups and parallel testruns - VERIFIED |