Meego Wiki
Views

Quality/QA-tools/OTS/OptimisingThroughput

From MeeGo wiki
Jump to: navigation, search

Contents

Optimising Throughput

The ability to run tests in parallel is one of the key features of using OTS.

Test Managers in production environments are keen to ensure that the throughput of the Testruns is optimal.


Time and Motion Study of the Workers

The Workers perform three primary functions

  1. Flashing
  2. Booting
  3. Testing Packages

So time spent on a Testrun is: 

dT = Flashing_t + Booting_t + SUM(Testing_t)

Running in Testruns in Parallel

When OTS receives a Testrun request it distributes it to a Worker. The Worker runs through the sequence above on all Test Packages.

This is the simplest and in many respects the most efficient way of running the Testrun.

Splitting the Packages

If we have a Testrun with several packages and we wanted it processed as quickly as possible we can exploit the parallelism in a different way. By running the Packages on different Workers this Testrun *might* be processed more quickly.

It *might* because there are other factors at play:-

1. Increased number of Flashes

If more than one Testrun takes the same approach there is a good chance it will slow things down. Flashing is time consuming so the maximising the TestPackages that get run per flash is a good policy if all the Testruns are to be treated fairly.

So what works quickest is dependent on many factors. For example:

If a Testrun with many packages is known to take a long time and a single instance of this is run alongside a few Testruns that take a short amount of time then the total time for all the Testruns *might* be reduced by splitting the packages (see next section).

So this approach scales badly, splitting the packages for all Testruns will reduce throughput on a busy Worker farm.

2. Workers already have Tasks Running

Consider a Testrun made of a n of Packages:

dT_total_testrun = testrun_t1 + testrun_t2 + ..... + testrun_tn

We might consider dividing up the Testrun up into a number of Tasks of even length so that the no_of_tasks = no_of_workers

So the Task time is now approximately:

dT_task = flash_time + boot_time + (dT_total_testrun / no_of_workers)

There is no guarantee that this will give the speed-up that we are looking for. In many cases when a Testrun is added to the Queue there will already be Tasks in process on some of the Workers.

If the time remaining for the Tasks on the queues is significant (i.e. > dT_task ), then a proportion of the split tasks will run in serial rather than parallel. Running in serial means that flash_time and boot_time must now be repeated! Potentially we could be slowing down the very run that we want to speed up by taking this approach.

Parameters

The parameters that affect throughput are:

  • flashing_time
  • booting_time
  • no_of_packages (in the Testrun)
  • no_of_workers
  • no_of_testruns
  • testrun_importance

Looking at the parameters...

  • no_of_workers is generally static.
  • flashing_time, booting_time, no_of_packages can be estimated by characterising typical Testruns currently used in the System.
  • The no_of_testruns will generally vary in a Test System and the number of Testruns with a given testrun_importance will also vary.

Considering the examples of the different scenarios that might arise as sketched out above it is evident that the optimal system will depend on the usage model and the needs of the Test Manager.

At this point OTS will provide an API that allows Plugins for alternative Distribution Systems for your particular needs based on these Parameters

Retrieved from "http://wiki.meego.com/Quality/QA-tools/OTS/OptimisingThroughput"
Personal tools