Quality/Test areas and types

Revision as of 10:13, 16 February 2011

Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Note that this includes also description for test types used in Test plan xml files visible through http://testplans.meego.com (e.g. MCTS - Meego Core Test Suite - through http://testplans.meego.com/mcts/#matrix).

Functionality quality characteristics

• Functional (accuracy) testing : Testing the application's or component's adherence to the specified or implied requirements.
  • Test type Certification : Testing the SW component’s adherence to the specified or implied certification requirements.
• Functional (suitability) testing : Evaluating and validating the appropriateness of application or a set of features for its intended specified tasks. This testing can be based on use cases or procedures. Testing shall also evaluate the application's or component's tolerance to faults in terms of handling unexpected input values, data, messages or triggers (so-called negative tests).
  • Test type Functional : See description above.
• SW Interoperability Testing (a.k.a. System Integration Testing): Tests whether a given group components and applications can function correctly in intended target configuration. Testing for interoperability may, for example, focus on ability of the software to automatically detect the communications needs of the systems it interacts with and switch accordingly.
• Technical Security Testing : Technical security testing attempts to compromise a system's security policy by assessing a system's vulnerability to threats.
• Accessibility Testing : Evaluating and validating the accessibility of software to those with particular requirements or restrictions in its use. This includes those with disabilities.

Efficiency quality characteristics

• Performance Testing : Focuses on the ability of a application or component to respond to user or system inputs within a specified time and under specified conditions (e.g. normal load, low resource and stress situations). Performance might be measured according to CPU cycles, response or latency times, frame rates, etc.
  • Test type Framerate : Measuring the frequency rate at which an imaging device produces unique consecutive images.
  • Test type Latency : Measures time delay between the moment something is initiated, and the moment one of its effects begins or becomes detectable.
  • Test type Response : Measures the time a system takes to react to given input by end-user; the total time it takes from a user request until response is received. When talking about response time one usually means completion time.
• Benchmark : (1) A standard against which measurements or comparisons can be made. (2) A test that is be used to compare components or systems to each other or to a standard as in (1).
  • Test type Benchmark : See definition (2) above
• Resource Utilisation Testing : Evaluate the usage of system resources (e.g. CPU, memory, energy, bandwidth of internal buses or external interfaces, etc.) either with a single measurement/snapshot or sequence of measurements over the time (e.g. in order to detect memory leaks).
  • Test type Throughput : The (average) rate of successful message delivery over a communication channel in a given period of time.
  • Test type Memory : Amount of (RAM) memory used during a use case or over a period of time
• Scalability testing : Evaluate the ability of a system to meet future efficiency requirements, which may be beyond those currently required.

Reliability quality characteristics

• Robustness Testing : Evaluate a application's or component's tolerance to faults which occur externally to the system under the test (e.g. memory full, process or service not available, file not found).
  • Test type Low-resource : Testing how system behaves (functional vice) under low system resources. For example, low disk space, available RAM etc.
• Recoverability Testing : Evaluate the software application's or component's ability to recover from hardware or software failures in a predetermined manner which subsequently allows normal operations to be resumed.
• Test type Iterative : Performing multiple iterations of the same test case in order to expose possible problems in sequential runs.
• Test type Long-lasting : Executing a test case for a long time in order to find errors, which occur during a longer usage.

Usability quality characteristics

• Usability Testing : Testing to determine the extent to which the application or feature is understood, easy to learn, easy to operate and attractive to the users under specified conditions.

Maintainability quality characteristics

• Dynamic maintainability testing : Evaluating the documented procedures developed for maintaining a particular application (e.g. for performing software upgrades).
• Changeability testing : Evaluate the effort required to make changes to a system (e.g. code changes). Both testability (effort required to test the changes made) and stability (system’s response to change) aspect should be taken into account here.

Portability quality characteristics

• Installability testing : Testing conducted on the software used to install other software on its target environment.
• Compatibility testing : Evaluation of possible adverse impact on functionality when applications are loaded on the same environment (e.g. conflict resource usage when a server runs multiple applications).
• Adaptability testing : Evaluating whether a give application can function correctly in all intended target environments (hardware, software, middleware, operating system, etc.). This testing can be done by using a selection of functional test cases which exercise the various components present in the environment.
• Replaceability testing : Evaluating the ability of software components within the system to be exchanged for others.