Revision as of 07:03, 8 March 2011

QA tools development

This page provides information on the development activities and practices of QA tools. The focus is on presenting things that are relevant for people interested in developing the tools. The main page for the end users is here.

Anyone is welcome to contribute and non-member contributions will be treated with same process and review as member contributions. We follow MeeGo contribution guidelines. In addition, you may take personal clone from our git repositories and create merge request. Tool maintainers in our projects will review your contributions and decide on merge.

Targets:

• Improve MeeGo test reporting tools - target from MeeGo Quality Assurance
• Improve MeeGo test automation (execution and software installation) - target from MeeGo release engineering

"As Core OS release release manager I want to verify trunk:testing packages frequently so that I know the quality of nightly/weekly releases."

Tools and Maintainers

Tool maintainers are selected based on developer experience with particular tool/package or seniority. Tool maintainers have been agreed in the QA-tools meeting Tuesday September 7th 2010. Changes, if needed, are discussed also there.

In practice only tool maintainers will have commit and review right to particular repository - later several people may have rights to repository based on merit (as proposed by tool maintainer). Others must follow MeeGo contribution guidelines to submit patches or personal clone + merge request approach.

The maintainer of the tree shall update the changelog.

If you are interested in the user experience work regarding these tools, you can find more information here.

Overview

The figure below tries to summarize the relations and tasks of the tools when used in test automation context.

xfig file:File:Qatools.fig

Release Practices

Here is the workflow for QA tools release practices.

Kivio file: File:Release.flw

1. Developer creates merge request(s) in gitorious.
2. VCS Maintainer tests and accepts merge requests.
3. VCS Maintainer checks/updates change logs.
4. VCS Maintainer tags a version.
5. VCS maintainer sends email to meego-qa mailing list based on the following template

Topic: Integration request: package-name version

PACKAGE: package-name
TAG: tag name
URL: link to sources
CHANGES: short description of changes containing bugs.meego.com bug numbers of fixed bugs

1. Package maintainer updates the OBS package.
2. Package maintainer tests the OBS package.
3. If the package belongs to tools:testing and passes testing, Package Maintainer may accept it. If the package belongs to trunk:testing, Package Maintainer creates a promotional request to Release Management. (If the package belongs to both repositories, we let the Release Management set bugs fixed by the package to RELEASED state).
4. Host side tools are updated to tools:testing after verifying functionality
5. Package maintainer replies to meego-qa list about the actions done with the updated package. e.g. 'Updated in devel:quality and sent promotion request to testing'
6. Release Management accepts the package. Or not. (Follow meego-packaging and meego-commits.)

YouTube videos

YouTube is a good way to communicate new features. You can find existing demo videos on meegoqatools channel on Youtube.

If you shoot a video to YouTube, promote it on meego-qa mailing list!

You can find some hints how to shoot, edit, and upload a video here: YouTube_Hints

Release checklist

To make sure fixes are released without delay, check that the following conditions are met

1. Change logs are updated and contain relevant references to MeeGo bugzilla
2. Created obs request include fixes bug numbers from MeeGo bugzilla
3. Bugzilla items listed in changes are set as resolved
4. Spec file matches MeeGo packaging guidelines
5. Rpmlint warnings are either fixed or explained by comments in the spec file. e.g. eat packages install files into root's home and the reasoning for it needs to be explained
6. Host side tool packages use the same source tar ball to produce debian and rpm packages

Roadmap

(We need still rough estimates for releases -timakima)

We'd like to have the best guess of the moment about the delivery of features. Remember to set the target build of the bug you're working with according to: Release Engineering/Plans/1.1 and Release Engineering/Plans/1.2

When you add a new bug, add correct dependencies to the corresponding roadmap meta bug.

The release dates defined in MeeGo Release plans 1.1 and 1.2. These dates are the latest estimation. They will be updated as work progresses.

The features in the roadmap are followed with META FEA bugs. The features are split to small tool specific FEA:s that block the feature META bug. The META bug is then the last bug to be closed when the feature is finished. You can also follow the tool bug progression from the dependency trees of the META bugs.

Adding a new feature to the roadmap:

1. Add a feature bug describing the main purpose of the feature with META and FEA tags.
2. Add separate bugs for each tool specific change
3. Add correct dependencies between tool specific bugs (e.g. testrunner bug depends on testrunner-lite bug that depends on test-definition bug...)
4. Make all of the bugs block the META bug

Design/ Planning

• MCTS Test Automation (QA Tools support for MCTS)

Documentation

Please make sure that all the relevant documentation regarding the tools is kept up-to-date whenever you update any of the tools. Please consider the end users of the tool as the primary readers of the documentation and make the documentation task-oriented for end user needs. Separate developer documentation can exist as well.

Wiki guidelines

Every tool should have a wiki page that contains at least 1) Overview of the tool and its benefits 2) Installation instructions 3) Basic use tutorial 4) Contact information.

Make sure that you are familiar with MeeGo wiki contribution guidelines before creating new content. Currently, there are quite a lot problems relating to e.g. page names, wiki links, deep hierarchies and obsolete pages in our wiki section.

Documentation checklist

The following table lists the tool-specific documentation that should exist (marked with 'x'). These will be checked more frequently and more in depth in the future and bugs will be filed for the missing or inaccurate documentation (marked as 'NOK' i.e. "Not OK").

Other documentation

• Autotest-Guide
• How to set up the repositories that are needed to install QA tools

Meetings

All meetings will be held in #meego-meeting on irc.freenode.net.

• Team meetings
  • QA tools team meetings will be held on need basis for specific topics. This was agreed in the last weekly meeting held on December 7th 2010
  • Meeting agendas and minutes

• Architecture meetings
  • Architecture meetings will be held on-demand basis.
  • Meeting agendas and minutes

• Workshops
  • Face-2-face meetings within the team.
  • Meeting agendas and minutes

Collaboration spaces

Team communication is in English. Our collaboration spaces are:

• meego-qa@lists.meego.com mailing list
• meego-dev@meego.com mailing list, please prefix with 'QA-tools' for team related topics.
  • Please also poke team members or Ville Ilvonen either by email or on IRC because of high traffic @ meego-dev
• #meego-qa-tools IRC channel on irc.freenode.net
  • #meego-qa-tools statistics
  • #meego-qa-tools irc logs
• Gitorious team, http://meego.gitorious.org/meego-quality-assurance/
• Youtube channel for demo videos
• QA-tools team blog
• MeeGo OBS - devel:quality
• This wiki area
• ServiceOS
• PXEInstallation

Team members

The current team members are (in no particular order):