Latest revision as of 10:32, 12 April 2012

Introduction

This guide gives a walk through of the steps needed to automate testing in MeeGo. The guide is updated when automated testing solutions are updated or new parts for the system are taken into use so make sure to check the guide for updates every now and then.

The current automated testing solution offers two different ways to execute the tests, from host pc through ssh or on the device under test itself. OTS uses host based testing to execute the tests. On device testing is meant to be used with manual testing and in special cases where host based testing is not an feasible option.

For information about the tools used with test automation see QA-tools and maintainers.

Feedback

Feel free to contribute and improve this guide. For questions and/or feedback about it pop by #meego-qa-tools at freenode or use meego-qa mailing list.

Writing tests

Test plan

Test plan is a XML file that contains your test suites, cases and steps. The basic principle of test plan XML and tool support is that you can use 'any' executable for testing.

For more information on writing test plan XMLs see Test plan page in this wiki.

Test packaging

Test packaging is intended to provide flexible and consistent ways to run tests and get results. You may select testing tools of your choice.

Test packaging is set of simple rules to wrap tests with test plan inside rpm package. Tool support for validating test plan and automating test plan execution is provided with testrunner. Following describes rpm-version of test packaging.

A source package with test cases must:

• build binary rpm package with name which ends with "-tests" (this is a test package)

A test package must:

• contain all tests, scripts and configuration files required to run tests
• define dependencies - the ones it tests and the test tools it depends on (if any)
• contain a test plan located at /usr/share/<packagename>-tests/tests.xml [1]

For more information on creating test packages, see Test package page in this wiki.

Creating test images

To perform automated testing you need to create images that contain test automation enablers and the tests. You can create two types of test images depending on how tests are planned to be executed. Instructions for image creation are here and the beginner instructions are here.

For reference kickstart files for test images, see: http://gitorious.org/qa-tools/eat/commits/ks

Images for host based testing

For host based testing the image needs to have the eat-device package installed and optionally eat-syslog-device in it. In addition to these configuration packages you need in to install test packages to the image since OTS fetches the tests from the image.

Images for on device testing

For on device testing install eat-selftest package along with your test packages to the image. This package finds and executes all tests found in the image automatically after boot and writes test results to $HOME/testresults. The tests are executed as the user meego.

If you don't want the tests to be executed automatically and handle test execution yourself install testrunner-lite package to the image along with your tests. In this case the test execution is started like this

testrunner-lite -f /path/to/tests.xml -v -o /path/to/results.xml

Automating image creation

Solution for automated image creation is not yet available.

One option is to use crontab and mic-image-creator.

Running tests

Automated test execution in MeeGo can be done by using generic test execution tool called testrunner-lite. Testrunner-lite reads planned tests from test plan xml file which is validated against test-definition schema. The tool can be run on the device under test or from a test control pc.

To enable automatically setting up the test environment we have created `eat` (Enable(s) Automated Testing) package. Eat package splits into different sub-packages listed below:

• eat-device - Configures the device under test for host based testing
• eat-selftest - Configures the device run tests automatically on boot
• eat-host - Configures test control pc for host based testing
• eat-syslog-device - Configures device's syslog to be sent to the host machince
• eat-syslog-host - Configures host's syslog to receive syslogs from the device under test

Automatically running test packages on image

Tests can be run automatically on a MeeGo image by adding package eat-selftest to the image.

eat-selftest package pulls testrunner-lite and test-definition as dependencies to the image. The package itself installs an init script that finds all installed test packages and calls testrunner-lite to execute them. Test packages need to be installed in /usr/share/<packagename>-tests/tests.xml, otherwise the script doesn't find them.

Results from the test and logs from the test runs are written to $HOME/testresults whese $HOME is the path to the user's home who is executing the tests.

Running test from test control pc

If testing is performed by OTS the test packages are fetched from the device and you don't need to manually start the test execution. To manually start test execution from the test control pc (host machine) you need to have the test plan xml files available on the host. To start executing tests in the device under test you need to tell testrunner-lite the device's IP address. For example

testrunner-lite -f tests.xml -v -o results.xml -t root@192.168.2.15

Setting up test automation environment

This chapter describes how to set up you own test automation environment. The environment includes host machine configuration, setting up OTS server and configuring the device under test.

Setting up the host machine

RPM packages produced by obs are only tested to work with MeeGo and Fedora 13. If you're using Ubuntu as your host see the Ubuntu 10.04 subsection.

Install the package eat-host to you host machine. This package will install ssh-keys needed for passwordless authentication and testrunner-lite and test-definition as dependencies. If you need also to receive syslogs from the device under test, also install the eat-syslog-host/device packages.

After installing you need to configure the host machine to use usb networking with the device under test. With Fedora you can do this by right-clicking the network managet icon and selecting edit connections. Edit the "Auto usb0" -connections IPv4 Settings and set Method to Manual and define the following Address to it:

Address: 192.168.2.14
Netmask: 255.255.255.0
Gateway: 0.0.0.0

or if you don't use network manager to manage your networking do the following

# ifconfig usb0 192.168.2.14 up

After this your host machine should be ready to act as a test control pc. Have fun testing with it.

Note! If the usb connection is unstable or breaks time to time, one reason for this can be the network-manager (Ubuntu). You can remove the network-manager and configure network settings manually. This is recommended to do for OTS workers and maybe to your desktop, not for laptops (WLAN connections are hard to create after removing the network-manager).

sudo apt-get purge network-manager

Edit /etc/network/interfaces and add next lines (this assumes that your wired connection is eth0 and it uses dhcp):

auto eth0
iface eth0 inet dhcp

auto usb0
iface usb0 inet static
   address 192.168.2.14
   netmask 255.255.255.0
   network 192.168.2.0
   broadcast 192.168.2.255
   up iptables -A POSTROUTING -t nat -j MASQUERADE
   up echo 1 >/proc/sys/net/ipv4/ip_forward
   down echo 0 >/proc/sys/net/ipv4/ip_forward
   down iptables -t nat -F POSTROUTING

Ubuntu 10.04

If you're using Ubuntu 10.04 or 10.10 as your host you can get the host side configuration packages and tools from Tools:Testing debian repository by doing the following:

1. Set up Tools:Testing repository

2. Install the host configuration package

sudo apt-get install eat-host-ubuntu

The package 'eat-host-ubuntu' will install ssh-keys used to access the device under test for the root user and configure the host to receive syslogs from the device under test. In addition to the configuration the package will install testrunner-lite, testrunner-lite-hwinfo-meego and test-definition as it's dependencies.

Fedora 13

Host side tools are also available from a Fedora 13 repository. Set up Fedora 13 as the host by doing the following:

1. Set up the repository

2. Install the following packages from the repository mentioned above

# yum install eat-host eat-syslog-host

Setting up the device under test

Nokia N900

To enable host based test execution with N900 MeeGo image you need to install eat-device package to the image. The package will install ssh keys to enable passwordless logins from the host machine. Package eat-syslog-device can also be used to configure the images syslog settings so that syslogs get sent to the host machine.

MeeGo images for the N900 have usb networking enabled by default. The devices IP address is 192.168.2.15

Netbooks

Current assumption is that we will use ethernet connection to netbook. The netbook should have ip 192.168.2.15 and eat-device package installed. Configuring DHCP for netbook is explained in PXE_boot_server.

Other devices

If the device under test uses IP 192.168.2.15 and the test control PC uses IP 192.168.2.14 then you can make the set up by using the eat-device and eat-host packages. The default IP addresses need to be those to support OTS. If you need to use some other IP addresses you have to do some manual work to get host based test execution working.

To enable host based test execution do the following:

Add the test control PC's public ssh-key to the devices authorized_keys by

cat ~/.ssh/id_rsa.pub | ssh root@[devices's IP address] "mkdir -p ~/.ssh ; cat >> ~/.ssh/authorized_keys"

you may also need to increase the device's sshd startups by

echo "MaxStartUps 1024" >> /etc/ssh/sshd_config

after this you should be able to run testrunner-lite from the host machine. If you need to get the device's syslog sent to the test control PC you have to edit the device's /etc/rsyslog.conf or /etc/syslog.conf depending if the image is using rsyslog or sysklogd. Newer MeeGo images are using sysklogd. For sysklogd do the following

cp /etc/syslog.conf /etc/syslog.conf.back
echo "*.*;auth,authpriv.none                            @[control PC's IP]"\
>> /etc/syslog.conf
cp /etc/sysconfig/sysklogd /etc/sysconfig/sysklogd.back
sed -e 's/SYSLOGD_OPTIONS.*/SYSLOGD_OPTIONS=\"-m 0 -r\"/' \
/etc/sysconfig/sysklogd.back \
> /etc/sysconfig/sysklogd

After that setup the test control PC to receive the syslogs

cp /etc/sysconfig/sysklogd /etc/sysconfig/sysklogd.back
cp /etc/syslog.conf /etc/syslog.conf.back
echo "SYSLOGD_OPTIONS=\"-m 0 -r\"" >> /etc/sysconfig/sysklogd
echo ":fromhost-ip, isequal, "[device's IP]"                     /var/log/eat.log"\
>> /etc/syslog.conf

Automatic image installations

N900

MeeGo images can be installed to the N900 automatically by using autoinstaller-n900.sh script. The script is included in File:Autoinstaller-n900.tar.gz. The package also contains a custom initrd image and kernel that are used in the installation process. The package however does not include the Nokia flasher utility required by the process. The utility is available for N900 owners from Nokia

Autoinstaller for N900 images is available here: File:Autoinstaller-n900.tar.gz

Pre-requirements

• Flasher utility is in the same folder as the script
• Device has an external microSD card
• Device is connected with USB cable to the host machine
• The device is powered off

Usage After all the pre-requirements are met do the following

cd /path/to/directory/containing/script/
sudo ./autoinstaller-n900.sh /path/to/raw-image /path/to/kernel

Note that the kernel given to the script is the kernel that comes with the raw image.

Netbook

The automated install of netbooks is still work in progress. Current idea is that the test control PC acts as a PXE boot server for the SUT (=netbook). We boot into initrd image over ethernet and from there on do pretty much the same as with the N900. The following has been tested with Samsung NC 10 laptop.

Autoinstaller for netbooks: File:Autoinstaller-netbook.tar.gz

Pre-requirements

• Laptop with PXE support
• PXE boot server
• Laptop connected to PXE server over ethernet and powered off

Usage

• Power on the laptop and press F12 (Samsung NC 10 has "PXE always on" BIOS option but it does not seem to have the desired effect).
• If all goes well you should see the following line on the laptop screen:

Starting netcat for initial connection check

• Do the following

cd /path/to/directory/containing/script/
sudo ./autoinstaller-netbook.sh /path/to/raw-image

Other targets

No solution for other targets currently available

Troubleshooting

N900 auto installation never finishes

• Check if the copying is being done by

sudo kill -USR1 `pidof dd`

• Check if usb0 interface is configured

ifconfig

bring the interface up by

ifconfig usb0 192.168.2.14 up

The installation tries to bring the interface up automatically if it can't get a response from the device by doing the above command

Note that USB hubs, etc. can slow down the installation significantly.

Freezing ssh sessions with host based execution

Some users have a use case where they want to start a server process in a test step to be used by later test steps. If you fork a new process it is possible that the ssh session running freezes during logout until given timeout or when the process terminates. This blocks further test steps. Reason for this is that each test step is currently in its own ssh session and the forked process holds on to its standard pipes and working directory. This is discussed in more technical detail in bug 5718.

One solution to this issue is daemonizing the new process. Then it redirects it's input, output and error pipes to "/dev/null" and changes the working directory to root. Here's a short code snippet how to daemonize a C program.

#include <stdlib.h>

int main( int argc, char *argv[] ) {
  daemon(0, 0);
  /* actual code */
  return 0;
}

The server process must be terminated by a later test step so that it won't stay running in the device after test case.