Quality/Plans/Testability-checklist

For a previous version of the Testability Checklist, see: http://wiki.meego.com/Quality/Testability_checklist_MeeGo_1.1

Contents 1 Introduction 2 Describing the Feature 3 The Story 3.1 Dissected Example 4 Extra Information 5 Testability Checklist 6 Conclusion

Introduction

User stories are a simple method to describe features, thus, they should be simple to write and to understand. By writing user stories to the features in Bugzilla, features can contain sufficient information to make test development possible without increasing the workload of the feature owner (as much as the "MeeGo 1.1 Testability Checklist"). The testability checklist used for MeeGo 1.1 contained items that were related to the process of feature creation which should be self evident and not be included in testability checklist.

Describing the Feature

The most effective way to create testable features from QA point of view is to write or user stories. The objective of writing user stories is to explain the feature to the extent that a test to verify the feature can be written. To that end, after writing a user story, ask yourself this question:

Can someone write a test using the information in this user story?

If the answer is 'yes' then the user story is complete, if the answer is 'no', then it may be worth it to consider rewriting the user story or (if this seems like it will not help) add extra information (see the extra information section).

Some of the use case may not be fully applicable for Core OS QA but in most of the cases use cases can be divided to smaller particles and can be considered as enablers for use case. E.g. Browsing use case can be verified using these smaller particles (enablers) as follows:

• Scan and connect to WLAN Access Point and conduct data transfer
• Download specific file over WLAN with performance measurement (Throughput)

The Story

There is a good wikipedia article on the concept of the user story here. While there are some differences from the article because MeeGo is not a small project that is being implemented by one team.

Three basic forms for a User Story are:

• I as a <user type/role> want to <perform action> and <result/goal>
• "As a <role>, I want <goal/desire> so that <benefit>" (from wikipedia)
• "As a <role>, I want <goal/desire>" (shortened version from wikipedia)

It is important to define the goal clearly. This allows the success criteria of the test to be clearly defined.

It is also important to define the set of actions leading up to the goal clearly. For example, if the user needs to tilt the device from horizontal to vertical for an action to occur, this needs to be described.

Dissected Example

The following feature can be found here and can be considered a good example of a user story:

User is browsing multimedia content in browser and a system event triggers a sound, e.g. battery low or alarm. System sound is mixed in with the browser audio.

We can see in this example:

• The user action (browsing multimedia content) is defined clearly.
• There is system action (system event triggering a sound).
• The goal is clearly defined ("System sound is mixed in with the browser audio").

From the above description, a test can be written.

Extra Information

The basic concept of user stories does not usually include the provision for extra information. This is usually not a problem for a single, coherent, project. In a project with as many components (from as many sources) as MeeGo, extra information may be required.

NOTE: Adding extra information is not a requirement, if your feature is adequately described by the story itself, then, stop there.

Some suggestions on what extra information to add:

• Requirements: e.g. binutils needs to be installed for this feature to work.
• Time Limitations: The operation needs to be performed in less than 5 seconds.
• Specific API that should be used (if this is not obvious).

Testability Checklist

Even though Testability Checklist is tool for QA teams, persons creating features in Bugzilla should have a thorough look to this in order to make feature flow fluently. Testability Checklist contains mandatory parts (to fulfill the criteria to have Testability marked as YES) and optional part which is recommended for one creating features to consider seriously if there is such information available that will help QA teams to conduct their work at appropriate level, leading well tested MeeGo features and Releases.

Mandatory:

• Can I write a test based on the information in this user story?
  
  • Is the goal defined clearly?
    
  • Is the procedure required to reach the goal clearly defined?
  • Is the user story short enough?
  • Is there any further information that is easy to put down and may help a test developer?
• Are relevant technology parameters available (Sample rates, Bit Rates, supported picture sizes, etc.) described?
• For Core OS: Is API Under test defined clearly?
  

Optional:

• Is there be reference to API documentation of API(s) implementing features? Location where more information is available?
  
• Is there Performance aspects or requirements (response time, latency, raw throughput) that should be verified concerning this feature?
  
• Is there Reliability aspects or requirements (expected use rate per day, how long feature should survive in stress situation) that should be verified?
• Is there Power Management aspects or requirements (E.g. local music playback use case should not consume more than XX,XX mW) that should be verified?
• Are there clear targets in terms of performance or reliability concerning this feature?
• Is there significant interactions (e.g. local music playback while browsing) with other features which should be taken into account when designing test cases concerning this feature?

Conclusion

By creating user stories for each feature, the feature owner can avoid having to write a "Testability Checklist" as was required for MeeGo 1.1. Adequately written user stories should make life easier for both the product owner and the test developer.

-- Main.MarkHalmagiu - 28 Sep 2010