Quality/TestSuite/MCTS/MCTS Development Guideline

Revision as of 06:59, 28 July 2010

MeeGo Core Test Suites Development Guideline (Proposal)

Test Definition

• XML format :http://gitorious.org/qa-tools/test-definition
• One test package corresponds one test case definition file

Test Case Naming

• Show the purpose clearly, avoid simply use number.
• Name length should be less than 30
• Use '_' to connect words

File header

• Copyright
• License
• Author
• Changelog
• Template

Source Checklist (recommended)

• README ( template )(template2)
• src/ : contain source files
• utils/: contain utility and tools if any
• Makefile(or Makefile.am, configure.ac, autogen.sh)
• tests.xml
• data/: contain small data files (For big data such as media content, need separate package)
• pack.sh: the script to generate rpm package
• test.spec: spec files to create rpm package

Makefile (or autoconf/automake)

• make : build test suite.
• make install: install whole test suite (binary, script, utils, helper, data) into /opt/{PACKAGE}, and tests.xml into /usr/share/<testpackagename>/tests.xml
• make clean: remove all object file and tempory files
• make uninstall: remove test binary, scripts, and tests.xml out of target installation directory.

Coding Style

• C/C++ : http://www.chris-lott.org/resources/cstyle/indhill-cstyle.html
• Python: http://www.python.org/dev/peps/pep-0008/
• Shell : http://hub.opensolaris.org/bin/view/Community+Group+on/shellstyle

Test Case Design

• Common Rule:
  • Short: Test case should test one and only one test point
  • Clear result: Test case should return clear Pass/Fail result
  • Self explanatory: Test case should be easy to understand
  • Avoid duplicated code: Use common functions or classes as much as possible
  • Automate test under condition of stability
  • Main function should keep simple and structure keep neat.
  • For across-test common function, create separated lib file.
  • No dependency on other test case.
  • No dependency on UX or vertical specific apps.
  • No dependency on test framework.
  • Use relative path to access dependency files inside test package such as helper, util and data
  • Platform irrelevant
  • Clean environment after test complete.
  • If parameters number greater than 3, use getopts
  • Enough comments
  • Using common utilities: For example, switch user
  • Avoid complicated code logic
  • Remove useless code

• Test Coverage Category Rules:
  • Thorough
    • Cover all APIs.
    • Test case targets for usage (API combination) instead of single API.
    • With Parm-pairs of equivalence partitioning and boundary condition (All-pairs optional)
  • Average
    • High risk APIs(e.g. new functionality relevant APIs) should be covered with thorough test rules
    • Legacy or low risk APIs are covered only with basic tests
  • Lightweight
    • Test most critical functionality

  Test package list see http://wiki.meego.com/MeeGo_Core_Test_Suites_Packaging_draft