Quality/Glossary

Revision as of 05:56, 1 December 2010

Glossary

Smoke Test (a.k.a Sanity Test)

Smoke test ~ intake+smoke test from ISTQB. smoke test (set): A subset of all defined/planned test cases that cover the main functionality of a daily build for MeeGo distribution, to ascertaining that the most crucial features of a distribution work, but not bothering with finer details. Smoke test is carried out at the start of test execution phase to decide if the distribution is ready for detailed and further testing.

The purpose of Sanity Test for MeeGo:

• Provide basic health status of the whole system on daily basis, so people have basic understanding where we are in terms of quality
• Report out outstanding issues and regressions and track them fixed quickly
• Sanity test results are used to measure if the MeeGo repositories are in good shape so that decisions can be made if the software is ready for further testing or release. Of course, the release decision cannot be made only based on sanity test results, further testing results will also be referred to.

Test Collection maintainer (a.k.a Test Suite maintainer)

Test collection maintainer ~ package maintainer (archlinux) + technical test analyst (ISTQB):

• The role of the test collection maintainer is to update tests as new versions become available upstream and to field support questions relating to bugs in said tests. The term may be applied to any of the following:
  • A test analyst who maintains a test collection in one of the official test repositories (tests.meego.com). Test analyst shall be capable to:
    • Analyze tests and the internal structure of the system in sufficient detail to meet the expected quality level
    • Evaluate tests in terms of technical quality attributes as performance, security, etc.
  • A trusted tester of the community who maintains test collections in the unsupported/unofficial community test repository.
• The maintainer of a test collection is the person currently responsible for the test collection. Previous maintainers should be listed as contributors along with others who have contributed to the test collection.

Test collection ~ tests testing on specific areas of the distribution (e.g. Application Group, Application, Domain, Component). You can replace the test collection maintainer with test package maintainer – if you wish.

Test case

• (Low level) Test case: A test case with concrete (implementation level) values for input data and expected results.
• High level test case (a.k.a test idea): A test case without concrete (implementation level) values for input data and expected results. Logical operators are used; instances of the actual values are not yet defined and/or available.

Test case verdict

• Pass: A test is deemed to pass if its actual result matches its expected result.
• Fail: A test is deemed to fail if its actual result does not match its expected result.
• N/A - Not Applicable - can be seen as a initial value which is left as a verdict for a test case when no other verdict cannot be given due to some reason (e.g. test case could not be executed due to failure in test infra, etc.).
• Blocked: This is special value required for the situations where verdict cannot be given due to failure in preconditions.
• Measured: Non-functional cases whose data has been collected but verdict cannot be given due to missing pass/fail criteria

For all Blocked and N/A verdicts it would be preferable that reason why pass/fail verdict could not be given is documented either by comment “Test case X verdict not given due to reason Y" or bug ID.

References

• [ISTQB Glossary]