Quality/QA-tools/OTS/DeveloperDocs/TypicalUsage

Latest revision as of 09:59, 29 December 2010

Typical OTS Usage

This page tries to describe how end users typically use OTS. The motivation for this page is to gain common understanding about the problem domain.

Users

Most of the users can be grouped to one of these 3 groups.

Basic User

Typically a test engineer or mobile SW developer.

• "I want to execute my tests on real target device."

Expected Knowledge

• Knows test definition and test packaging
• Knows his/her particular application/test area in detailed level
• Knows what SW product he/she is developing
• Understands OTS testrun in a rough level. (Download and flash the image, execute tests over ssh.)
• Knows some of the OTS input parameters. For example email address, timeout, SW Product

• Does not know python
• Does not know details about the particular OTS system setup (devicegroups or workers available)

OTS Usage

In most cases does not directly interact with OTS. Uses OTS service provided by others.

• Testrun from build machine
  • User makes changes to Testpackage or target SW package.
  • User creates image with build machine.
  • Build machine triggers automatic testrun for the new image containing new version of user's software or tests.
  • OTS uses default parameters for the particular SW Product. (User does not define devicegroup or other parameters. User knows only the SW Product.)
  • User follows testrun process from html log page.
  • User receives test results as an email. They are also available in a reporting tool.
  • If there was errors in the testrun, the testrun log gives clear explanation what went wrong. (Flashing failed, Testpackage missing from the image, Unknown SW Product etc.)

• Nightly testruns
  • User makes changes to Testpackage or target SW package.
  • Build machine automatically builds a nightly image with new changes included.
  • Build machine triggers automatic testrun for the new version of user's packages.
  • OTS parameters for nightly build are defined by an advanced user responsible for the nightly testruns. (Basic user does not define devicegroup or other parameters)
  • User receives test results as an email. They are also available in a reporting tool.
  • If there was errors in the testrun, the testrun log gives clear explanation what went wrong. (Flashing failed, Testpackage missing from the image, Unknown SW Product etc.)

Advanced User / Worker Maintainer

Typically a test engineer or mobile SW developer. Has own device and worker pc connected to the OTS system. Motivation for own worker might be that the test cases require a specific environment or the user just wants to see what's happening while tests are executed.

• "I want to execute my tests in my own environment."

Expected Knowledge

In addition to the basic user knowledge:

• Knows how to set up and configure OTS worker.
• Knows the device properties and how to use the own device for a testrun

OTS Usage

Uses OTS service provided by others.

In addition to the basic user use cases, defines the device group (or device parameters) in the test run parameters so that testrun is routed to the specific worker.

• Local conductor run
  • Executes conductor from command line with custom built image.
  • Follows testrun from conductor output & log.
  • Result files are available in file system. (~/conductor/)
  • If errors or failures happen, the logs/output gives enough data for debugging

OTS System Maintainer

A system administrator is responsible for the OTS server and the whole testing system instance.

• "I want to provide a common test environment for my peers."

Expected Knowledge

In addition to the Advanced User / Worker Maintainer:

• Knows how to set up and configure OTS server
• Knows all OTS input parameters and how to trigger runs from command line
• Knows the common device pool setup
• Knows the default settings for each SW Product
• Knows testrun process in detailed level. Is able to debug problems in the system.
• Some python knowledge. Can read tracebacks and understands some of the common exceptions (import errors etc.)
• Is able to make detailed bug reports

OTS Usage

Provides OTS service for others.

• Setup
  • Installs ots server and common workers
  • Defines supported SW products for the particular OTS instance.
  • Defines device parameters for the common devices.
  • Defines default values for supported SW products so that the basic user does not need to know any details about the system.
    • Basic users can just trigger a run and everything goes right thanks to the default values.
    • For example the test run will be routed to a suitable device because default devicegroup is defined in the config by the system maintainer.
    • The testrun result is reported correctly in the reporting tool because the system maintainer has defined correct reporting category in the config file and that will be automatically used when communicating with reporting tool.

• Debug a testrun
  • A testrun triggered by a user ends with errors. User is not able to solve the problem so system maintainer is asked for help.
  • System maintainer studies the logs and debugs the problem
    • System maintainer can re-trigger the testrun for debugging purposes by checking the exact input parameters from the log.
  • System maintainer files bugs against OTS SW component if needed
  • System maintainer fixes problems in the OTS setup if needed
  • System maintainer instructs the user in OTS usage if needed