Views
Quality/TestSuite/MCTS/MCTS API analysis
Contents |
API analysis for functional test case design
Motivation
One of the major quality metrics for test cases is coverage. We'll informally define test coverage — as exercised by our test suite — as the portion of the total fixed set of functionality for which our tests can reliably tell whether it works as intended. In other words, the percentage of functionality for which we can provide evidence that it works.
Unit tests are often tracked with code coverage tools like gcov. With these, we can compare lines of code (LOC) executed during tests against total LOC, and decide that the unit test coverage is good when the number approaches 100%.
When testing the functionality of a component in a system, the picture is less clear. We can rarely call API methods one by one and claim full coverage when done. Instead, we need to identify situations where each function is applicable, enumerate different parameters for each possible state, and design a test case using this information. This may lead to situations where important functionality is only tested lightly at best, missing corner cases or risk areas.
For example, to test the cellular voice call functionality using oFono, we need to make sure that the cellular modem is powered and registered to the network (which might also involve handling SIM PINs, for example). As test parameters, we decide which numbers to call (What numbers are valid? What happens if we call 911? Do we have a test network available?) — and only then call the method to start the cellular call. Afterwards, we need to bring the system to a known state, regardless of what happened during the test, so that later tests are not interfered with.
We generally want to separate and fix setup and cleanup parts of tests in order to limit the variation in test cases to manageable levels. In the example, we don't want to deal with registration details or modem states, but instead focus on details of the cellular call — addressing, what happens during the call, how it is disconnected, and so on. Of course, the setup and cleanup parts should also be tested, but this is best done separately if at all possible.
The techniques presented here are intended to help designing functional tests, tracking coverage over the tested API, discovering gaps in current test coverage, and presenting the total status of the test suite to an audience.
Identifying the API
Deciding what parts of the API should be tracked for test coverage can be tricky. Components with good testability will have documentation available for this work. The situation is not so easy with all components, though, especially when they are under heavy development. Here are some places to look for information about a component's interfaces:
- It is usually simple to list functions in a single library. For example, look through the .h files, or the standard followed (where applicable). Published API documentation helps, when it is available, but should be compared against development headers or code when possible.
- Some components have an API used by passing messages with its users. Listing each kind of message available is a natural way to document this case. When a single message type can be used for different purposes, they should be separated. For example, the Netlink API to mac80211-based WLAN drivers uses certain messages for both commands (telling the driver to do a thing) and events (where the driver signals that a thing happened).
- Related APIs can be combined for coverage purposes when it is only possible to test different parts of a subsystem together. For example, a network socket is used with well-documented standard functions, but also accepts a number of ioctl calls for special situations, many of which are quite platform-specific. It makes sense to list each of these separately, even if they are technically accessed through a single function.
- Subsystems following some wider standard are sometimes best served by listing separate functions mentioned in the standard. For example, using low-level Bluetooth controller functions is done by issuing HCI commands and responding to HCI events it sends. These are listed in detail in the standard. Even though individual hardware components implement subsets of the functionality, using the whole set as a guide can make it easier to design test cases around how different functionality is defined in the standard.
Note that none of these methods are enough by themselves, without a good understanding how the tested system works.
The situation which the above techniques deal with has some similarities to difficulties encountered during black-box testing. We should consider it a separate problem, though. The techniques presented here are mostly developed for analyzing open but often not completely documented components.
Building a coverage matrix
Once a listing of the API functionality is available, we can build a matrix with test cases on one axis and API functions on the other, a simple task on a spreadsheet. For the following techniques, we keep API functions on top row and test cases in leftmost column.
For each test case, we mark each function covered. This way the total API coverage we're actually testing can be measured by looking at sums of the columns under each API function. We'll call this sort of table a coverage matrix.
It's recommended to mention the version numbers of the tested component and separate test suites used, and keep the whole document in a version control system.
Function and parameter coverage
One thing that is missing from the above picture are the parameters each tested function takes. This data should be tracked together with the test cases and APIs, so that changes in the component or the test content can be followed in one place. Here are some general guidelines for how to track test parameters:
- Make a list of parameters each API function can have, and combine ones that mean the same thing (this limits work done in the following parts).
- For each test case, consider each parameter in the tested functions. Describe all valid combinations of these parameters (most likely in terms of sets or ranges), and dependencies between them. Make note on what sorts of parameters the function gets in real use cases.
- Use common techniques including boundary value analysis and equivalence partitioning to select which sets of parameters to use in each test cases.
In practise, we often create test cases that take a set of values for each parameter as input, and generate sub-cases by permuting the variables in the test software (obviously some care is needed to avoid combinatoric issues here). This is not the only way to count test cases — some test designers prefer to explicitly list each tested parameter set as a test case — but it appears to work for us.
Identifying test cases
The above discussion often refers to test cases, but does not really describe what sort of set makes sense. Designing test cases for a component can be viewed as an iterative process, where test cases are added, combined, removed or refined, repeating for the life of the component. The following lists some inputs to this process:
- Each separate functionality that can be identified in the tested API should have its own test case. Example from MeeGo Feature 2988: "Graphics Enhancements - X Server to support window rotation" - we see (from XRandR documentation ) that we can change the display orientation with the "rotation" parameter to function "RRSetScreenConfig", so a test case going through the set of valid rotation states makes sense. (Obviously, this is a simplification of the actual code needed).
- Often a component is stateful; the invoked functions cause transition from one state to another. The testing of API functionality alone does not guarantee all states are covered. The coverage of states may be obtained by invoking the functions in particular order and with particular parameters.
- Components usually have requirements, or at least play a role in meeting them. There should be a set of test cases such that when each of them passes, the requirement can be thought as fulfilled (as far as the tested component is concerned). Continuing the above example, we could go through different rotation settings using the XRandR ("resize and rotate") API, and find a way to check the results.
- Risk areas that can be identified are an important source of test cases. This means functionality that is critical and/or easy to get subtly wrong, areas with regulatory or certification issues, or areas connected to user experience. For example, a buggy cellular stack could fail when calling some rare but valid range of numbers, get shut down for running low on memory due to a video player during an emergency call, or simply provide poor-quality audio.
- After designing a set of test cases for the component there are often coverage gaps left. These are easily visible in the coverage matrix, so identifying scenarios where the missing functionality would be applicable should not be too difficult.
Further iterations over the test set should be done to identify redundant tests (but take care to remove only tests where all parts of functionality under test are exercised elsewhere).
A large quantity of material has been written about test techniques, but a wider literature overview is beyond the scope of this document.
Further development
Using a test coverage matrix like the above suggests several possible enhancements. Some ideas for future development are listed below.
- The test software could use various techniques to combine a machine-readable description of the API with tracing data from test cases during the test runs, resulting in automatic gathering of coverage data. Care should be taken to isolate the functionality that is actually under test from incidental calls to setup functions and so on. Some components have specialized tracing support; as an example, the Linux kernel has trace point patches available, a set of which could be enabled during a test, with the trace output converted afterward to test coverage data.
- The coverage matrix could track test intensity or "thoroughness", for example by giving each function tested by a test case a score from, say, 1 to 5. These could be combined to form a metric for test suite state.
Examples
(There will be links here to real coverage analysis documentation, once they are up on the wiki.)