Quality/Glossary

Latest revision as of 07:37, 31 March 2011

Glossary

Smoke Test (a.k.a Sanity Test)

Smoke test ~ intake+smoke test from ISTQB. smoke test (set): A subset of all defined/planned test cases that cover the main functionality of a daily build for MeeGo distribution, to ascertaining that the most crucial features of a distribution work, but not bothering with finer details. Smoke test is carried out at the start of test execution phase to decide if the distribution is ready for detailed and further testing.

The purpose of Sanity Test for MeeGo:

• Provide basic health status of the whole system on daily basis, so people have basic understanding where we are in terms of quality
• Report out outstanding issues and regressions and track them fixed quickly
• Sanity test results are used to measure if the MeeGo repositories are in good shape so that decisions can be made if the software is ready for further testing or release. Of course, the release decision cannot be made only based on sanity test results, further testing results will also be referred to.

Test Collection maintainer (a.k.a Test Suite maintainer)

Test collection maintainer ~ package maintainer (archlinux) + technical test analyst (ISTQB):

• The role of the test collection maintainer is to update tests as new versions become available upstream and to field support questions relating to bugs in said tests. The term may be applied to any of the following:
  • A test analyst who maintains a test collection in one of the official test repositories (tests.meego.com). Test analyst shall be capable to:
    • Analyze tests and the internal structure of the system in sufficient detail to meet the expected quality level
    • Evaluate tests in terms of technical quality attributes as performance, security, etc.
  • A trusted tester of the community who maintains test collections in the unsupported/unofficial community test repository.
• The maintainer of a test collection is the person currently responsible for the test collection. Previous maintainers should be listed as contributors along with others who have contributed to the test collection.

Test collection ~ tests testing on specific areas of the distribution (e.g. Application Group, Application, Domain, Component). You can replace the test collection maintainer with test package maintainer – if you wish.

Test case

• (Low level) Test case: A test case with concrete (implementation level) values for input data and expected results.
• High level test case (a.k.a test idea): A test case without concrete (implementation level) values for input data and expected results. Logical operators are used; instances of the actual values are not yet defined and/or available.

Test case verdict

QA verdict definition:

• Pass: A test is deemed to pass if its actual result matches its expected result.
• Fail: A test is deemed to fail if its actual result does not match its expected result.
• N/A - Not Applicable - can be seen as a initial value which is left as a verdict for a test case when no other verdict cannot be given due to some reason (e.g. feature is not implemented yet, test case could not be executed due to failure in test infra, etc.).

For all Fail verdicts it would be preferable that bug ID is given.

For all N/A verdicts it would be preferable that reason why pass/fail verdict could not be given is documented either by comment “Test case X verdict not given due to reason Y" or bug ID.

For the case which functionality is implemented but case is missing, it is recommended to find case substitute to cover the check point. If no substitute is available to cover, mark the case as N/A.

If a testrun has one or more N/A verdicts it will be rendered as Fail.

References

• ISTQB Glossary