Quality/Test areas and types

Revision as of 05:55, 18 April 2011

Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Note that this includes also description for test types used in Test plan xml files visible through http://testplans.meego.com (e.g. MCTS - Meego Core Test Suite - through http://testplans.meego.com/mcts/#matrix).

Functionality quality characteristics

• Functional (accuracy) testing: testing to determine the capability of the software product to provide the right or agreed results or effects with the needed degree of precision (ISO 9126).
• Functional (suitability) testing: testing to determine the capability of the software product to provide an appropriate set of functions for specified tasks and user objectives (ISO 9126). This testing can be based on use cases or procedures. Testing shall also evaluate the application's or component's tolerance to faults in terms of handling unexpected input values, data, messages or triggers (so-called negative tests).
  • Test type Functional positive: tests aim at showing that a component or a system works as expected in terms of accuracy or suitability when handling valid inputs or expected user behaviors.
  • Test type Functional negative: tests aim at ensuring that a component or a system can gracefully handle invalid inputs or exceptions.
• Technical Security Testing: testing to determine the ability of the software product to prevent unauthorized access, whether accidental or deliberate, to programs and data (ISO 9126).
  • Test type Security : tests aim at attempting to compromise a system's security policy by assessing a system's vulnerability to threats, this domain is mainly covered with a dedicated test plan, see Meego Security Test Plan

Efficiency quality characteristics

• Performance Testing : Focuses on the ability of a application or component to respond to user or system inputs within a specified time and under specified conditions (e.g. normal load, low resource and stress situations). Performance might be measured according to CPU cycles, response or latency times, frame rates, etc.
  • Test type performance : The process of testing to determine the performance (or efficiency) of a software product. It can serve to validate and verify different aspects of the system, such as
    • Framerate : Measuring the frequency rate at which an imaging device produces unique consecutive images.
    • Latency : Measures time delay between the moment something is initiated, and the moment one of its effects begins or becomes detectable.
    • Response : Measures the time a system takes to react to given input by end-user; the total time it takes from a user request until response is received. When talking about response time one usually means completion time.
    • Benchmark : A test that is be used to compare components or systems to each other or to a standard.
    • Throughput : The (average) rate of successful message delivery over a communication channel in a given period of time.
    • Memory : Amount of (RAM) memory used during a use case or over a period of time
    • Load : Average amount of work that system is doing over a period of time.

Reliability quality characteristics

• Reliability Testing : Tests to discover potential problems with the design as early as possible and, ultimately, provide confidence that the system meets its reliability requirements.
  • Test Type Reliability : Testing to discover potential problems with the design as early as possible and, ultimately, provide confidence that the system meets its reliability requirements. It can serve to validate and verify different aspects of the system, such as
    • Robustness Testing : Evaluate a application's or component's tolerance to faults which occur externally to the system under the test (e.g. memory full, process or service not available, file not found).
      • Low-resource : Testing how system behaves (functional vice) under low system resources. For example, low disk space, available RAM etc.
    • Recoverability Testing : Evaluate the software application's or component's ability to recover from hardware or software failures in a predetermined manner which subsequently allows normal operations to be resumed.
      • Recovery : Testing aimed at verifying the system's ability to recover from varying degrees of failures, for example, reaction to unplanned breakdowns of service. Recovery testing ensures that the integrity of data won’t break when something unexpected happens.
    • Iterative : Performing multiple iterations of the same test case in order to expose possible problems in sequential runs.
    • Long-lasting : Executing a test case for a long time in order to find errors, which occur during a longer usage.

Usability quality characteristics

• Usability Testing : Testing to determine the extent to which the application or feature is understood, easy to learn, easy to operate and attractive to the users under specified conditions.
  • Test type User experience : The tests focusing on users' feelings stemming both from pragmatic and hedonic aspects of the system. The focus is on pleasure and value rather than on performance (getting task done), which is aim for usability testing.

Maintainability quality characteristics

• Dynamic maintainability testing : Evaluating the documented procedures developed for maintaining a particular application (e.g. for performing software upgrades).
• Changeability testing : Evaluate the effort required to make changes to a system (e.g. code changes). Both testability (effort required to test the changes made) and stability (system’s response to change) aspect should be taken into account here.
• Proposal: A test type ‘Maintainability’ needs to be added to cover following items: analyzability, modifiability, stability and testability
• Answer: ‘Maintainability’ test type and sub-test types to be added and defined

Portability quality characteristics

• Installability testing : Testing conducted on the software used to install other software on its target environment.
• Compatibility testing : Evaluation of possible adverse impact on functionality when applications are loaded on the same environment (e.g. conflict resource usage when a server runs multiple applications).
• Adaptability testing : Evaluating whether a give application can function correctly in all intended target environments (hardware, software, middleware, operating system, etc.). This testing can be done by using a selection of functional test cases which exercise the various components present in the environment.
• Replaceability testing : Evaluating the ability of software components within the system to be exchanged for others.
• Proposal: A test type ‘Portability’ needs to be added to cover following items: installability, compatibility, adaptability and replaceability
• Answer: ‘Portability’ test type and sub-test types to be added and defined