Quality/Test areas and types

Revision as of 08:06, 11 April 2011

Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Note that this includes also description for test types used in Test plan xml files visible through http://testplans.meego.com (e.g. MCTS - Meego Core Test Suite - through http://testplans.meego.com/mcts/#matrix).

Functionality quality characteristics

• Functional (accuracy) testing : Testing the application's or component's adherence to the specified or implied requirements.
  • Test type Certification : Testing the SW component’s adherence to the specified or implied certification requirements.
    • Proposed to be moved under Compliance: Test type Certification has to be moved to a missing attribute: ‘Compliance’ that is one of the functionality quality characteristics defined by the ISO 9126. Here is the ISTQB definition of certification: the process of confirming that a component, system or person complies with its specified requirements, e.g. by passing an exam. So, Compliance Testing : has to be added here. Compliance is the capability of the software product to adhere to standards, conventions or regulations in laws and similar prescription (according to ISO9126). The test type ‘Certification’ could be moved functional (accuracy) testing to this new characteristic. Certification as such is not done in meego.com.
    • Answer: Just remove Certification and add Compliance testing with description given in proposal.
• Functional (suitability) testing : Evaluating and validating the appropriateness of application or a set of features for its intended specified tasks. This testing can be based on use cases or procedures. Testing shall also evaluate the application's or component's tolerance to faults in terms of handling unexpected input values, data, messages or triggers (so-called negative tests).
  • Test type Functional : See description above.
    • Proposed to be replaced by negative and positive types: This is too generic and can be decomposed into 2 test types: ‘Functional positive’ and ‘Functional negative’ that cover both ‘accuracy’ and ‘suitability’ attributes.
    • Answer: Functional can be removed and these two types added over here. One needs definition to be written for both.
• SW Interoperability Testing (a.k.a. System Integration Testing): Tests whether a given group components and applications can function correctly in intended target configuration. Testing for interoperability may, for example, focus on ability of the software to automatically detect the communications needs of the systems it interacts with and switch accordingly.
  • Proposed to be removed.
  • Answer: Could be removed. This is more like a test set than test type.
• Technical Security Testing : Technical security testing attempts to compromise a system's security policy by assessing a system's vulnerability to threats.
  • Proposed to have Security testing as a new test type: A ‘Security’ test type should be added as a dedicated Meego test plan already covers this area, http://wiki.meego.com/Quality/Plans/MeeGo_Security_Test_Plan.
  • Answer: Security to be added as test type with description given in proposal and/or test plan.
• Accessibility Testing : Evaluating and validating the accessibility of software to those with particular requirements or restrictions in its use. This includes those with disabilities.
  • Proposed to be move under Usability: Accessibility is not listed in the functional quality characteristics defined by the ISO9126 standard. Moreover, this domain is related to the user experience, here is the definition provided by the ISTQB: Testing to determine the ease by which users with disabilities can use a component or system. So, it can be moved to ‘usability quality characteristics’ section below.
  • Answer: Lets remove this totally. Usability as such might have similar checks included but today Accessibility as such is not in a scope of meego.com.

Efficiency quality characteristics

• Performance Testing : Focuses on the ability of a application or component to respond to user or system inputs within a specified time and under specified conditions (e.g. normal load, low resource and stress situations). Performance might be measured according to CPU cycles, response or latency times, frame rates, etc.
  • Test type Framerate : Measuring the frequency rate at which an imaging device produces unique consecutive images.
  • Test type Latency : Measures time delay between the moment something is initiated, and the moment one of its effects begins or becomes detectable.
  • Test type Response : Measures the time a system takes to react to given input by end-user; the total time it takes from a user request until response is received. When talking about response time one usually means completion time.
• Benchmark : (1) A standard against which measurements or comparisons can be made. (2) A test that is be used to compare components or systems to each other or to a standard as in (1).
  • Test type Benchmark : See definition (2) above
    • Proposed to be removed and to be included to compliance
    • Answer: Let's keep this here since compliance (has acceptance aspect) is different than benchmarking with compare (no acceptance aspect)
• Resource Utilisation Testing : Evaluate the usage of system resources (e.g. CPU, memory, energy, bandwidth of internal buses or external interfaces, etc.) either with a single measurement/snapshot or sequence of measurements over the time (e.g. in order to detect memory leaks).
  • Test type Throughput : The (average) rate of successful message delivery over a communication channel in a given period of time.
  • Test type Memory : Amount of (RAM) memory used during a use case or over a period of time
  • Test type Load : Average amount of work that system is doing over a period of time.
• Scalability testing : Evaluate the ability of a system to meet future efficiency requirements, which may be beyond those currently required.

• Proposed to drop these (sub)test types and use only performance
• Answer: We could allow the performance as main test type here, but let's allow the usage of these (sub)test types too.

Reliability quality characteristics

• Robustness Testing : Evaluate a application's or component's tolerance to faults which occur externally to the system under the test (e.g. memory full, process or service not available, file not found).
  • Test type Low-resource : Testing how system behaves (functional vice) under low system resources. For example, low disk space, available RAM etc.
• Recoverability Testing : Evaluate the software application's or component's ability to recover from hardware or software failures in a predetermined manner which subsequently allows normal operations to be resumed.
  • Test type Recovery : Testing aimed at verifying the system's ability to recover from varying degrees of failures, for example, reaction to unplanned breakdowns of service. Recovery testing ensures that the integrity of data won’t break when something unexpected happens.
• Test type Iterative : Performing multiple iterations of the same test case in order to expose possible problems in sequential runs.
• Test type Long-lasting : Executing a test case for a long time in order to find errors, which occur during a longer usage.

Usability quality characteristics

• Usability Testing : Testing to determine the extent to which the application or feature is understood, easy to learn, easy to operate and attractive to the users under specified conditions.
• Proposed that ‘User experience’ (already created in testlink) needs to be described here
• Answer: ‘User experience’ to be added and described

Maintainability quality characteristics

• Dynamic maintainability testing : Evaluating the documented procedures developed for maintaining a particular application (e.g. for performing software upgrades).
• Changeability testing : Evaluate the effort required to make changes to a system (e.g. code changes). Both testability (effort required to test the changes made) and stability (system’s response to change) aspect should be taken into account here.
• Proposal: A test type ‘Maintainability’ needs to be added to cover following items: analyzability, modifiability, stability and testability
• Answer: ‘Maintainability’ test type and sub-test types to be added and defined

Portability quality characteristics

• Installability testing : Testing conducted on the software used to install other software on its target environment.
• Compatibility testing : Evaluation of possible adverse impact on functionality when applications are loaded on the same environment (e.g. conflict resource usage when a server runs multiple applications).
• Adaptability testing : Evaluating whether a give application can function correctly in all intended target environments (hardware, software, middleware, operating system, etc.). This testing can be done by using a selection of functional test cases which exercise the various components present in the environment.
• Replaceability testing : Evaluating the ability of software components within the system to be exchanged for others.