Quality/Plans/Testability-checklist

Revision as of 13:55, 7 October 2010

For a previous version of the Testability Checklist, see: http://wiki.meego.com/Quality/Testability_checklist_MeeGo_1.1

Introduction

Describing user interaction is a simple way to describe features, thus, they should be simple to write and to understand. By describing user interaction with the features in Bugzilla, features can contain sufficient information to make test development possible without increasing the workload of the feature owner (as much as the "MeeGo 1.1 Testability Checklist"). The testability checklist used for MeeGo 1.1 contained items that were related to the process of feature creation which should be self evident and not be included in testability checklist.

Describing the Feature

The most effective way to create testable features from QA point of view is to describe how the user interacts with the feature. The objective of writing such descriptions is to explain the feature to the extent that a test to verify the feature can be written. To that end, after writing a description, ask yourself this question:

Can someone write a test using the information in this description?

If the answer is 'yes' then the description is complete, if the answer is 'no', then it may be worth it to consider rewriting the it or (if this seems like it will not help) add extra information (see the extra information section).

Some of the use case may not be fully applicable for Core OS QA but in most of the use cases can be divided to smaller particles and can be considered as enablers for the main use case. E.g. Browsing use case can be verified using these smaller particles (enablers) as follows:

• Scan and connect to WLAN Access Point and conduct data transfer
• Download specific file over WLAN with performance measurement (Throughput)

Structuring the Description

There is a good wikipedia article on the concept of the describing a feature from a user's point of view here. While there are some differences from the article because MeeGo is not a small project that is being implemented by one team.

Three basic forms for a description are:

• I as a <user type/role> want to <perform action> and <result/goal>
• "As a <role>, I want <goal/desire> so that <benefit>" (from wikipedia)
• "As a <role>, I want <goal/desire>" (shortened version from wikipedia)

It is important to define the goal clearly. This allows the success criteria of the test to be clearly defined.

It is also important to define the set of actions leading up to the goal clearly. For example, if the user needs to tilt the device from horizontal to vertical for an action to occur, this needs to be described.

Dissected Example

The following feature can be found here and can be considered a good example of a feature description:

User is browsing multimedia content in browser and a system event triggers a sound, e.g. battery low or alarm. System sound is mixed in with the browser audio.

We can see in this example:

• The user action (browsing multimedia content) is defined clearly.
• There is system action (system event triggering a sound).
• The goal is clearly defined ("System sound is mixed in with the browser audio").

From the above description, a test can be written.

Extra Information

The basic description from a user's point of view does not usually include the provision for extra information (things that the user would not see). This is usually not a problem for a single, coherent, project. In a project with as many components (from as many sources) as MeeGo, extra information may be required.

NOTE: Adding extra information is not a requirement, if your feature is adequately described by the description itself, then, stop there.

Some suggestions on what extra information to add:

• Requirements: e.g. binutils needs to be installed for this feature to work.
• Time Limitations: The operation needs to be performed in less than 5 seconds.
• Specific API that should be used (if this is not obvious).

Testability Checklist

Even though Testability Checklist is tool for QA teams, persons creating features in Bugzilla should have a thorough look to this in order to make feature flow fluently. Testability Checklist contains mandatory parts (to fulfill the criteria to have Testability marked as YES) and optional part which is recommended for one creating features to consider seriously if there is such information available that will help QA teams to conduct their work at appropriate level, leading well tested MeeGo features and Releases.

Mandatory:

• Can I write a test based on the information in this description?
  
  • Is the goal defined clearly?
    
  • Is the procedure required to reach the goal clearly defined?
  • Is the description short enough?
  • Is there any further information that is easy to put down and may help a test developer?
• Are relevant technology parameters available (Sample rates, Bit Rates, supported picture sizes, etc.) described?
• For Core OS: Is API Under test defined clearly?
  

Optional:

• Is there be reference to API documentation of API(s) implementing features? Location where more information is available?
  
• Is there Performance aspects or requirements (response time, latency, raw throughput) that should be verified concerning this feature?
  
• Is there Reliability aspects or requirements (expected use rate per day, how long feature should survive in stress situation) that should be verified?
• Is there Power Management aspects or requirements (E.g. local music playback use case should not consume more than XX,XX mW) that should be verified?
• Are there clear targets in terms of performance or reliability concerning this feature?
• Is there significant interactions (e.g. local music playback while browsing) with other features which should be taken into account when designing test cases concerning this feature?

Conclusion

By creating descriptions at user level for each feature, the feature owner can avoid having to write a "Testability Checklist" as was required for MeeGo 1.1. Adequately written descriptions should make life easier for both the product owner and the test developer.