Release Infrastructure/BOSS Testing/BTF

Contents 1 BATS "BOSS Automatic Testing System" 2 Covering Packages 3 Target Users 4 Use Cases 5 Design 6 Implementation 7 Test Techniques in BATS 8 Current Test Status 9 Use Guide

BATS "BOSS Automatic Testing System"

• BATS is a system which is used to ensure the quality of BOSS. This system is developed to make the testing easely. It will manage all the precess of auto-testing: environemt deployment, test exectuion and result management. This system covers different levels of tests for BOSS:
  • Unit Test
  • Functional Test
  • Integration Test
  • Performance Test
• Test cases shall be easy to extend and customize with simple interface.
• Anything is automatic include real test environment deployment. Only one command is needed to triger the testing.

Covering Packages

• The target of BATS is to cover all components of BOSS and the dependencies, which can be found at: https://build.opensuse.org/project/packages?project=Maemo:MeeGo-Infra
• At the beginning, we're focusing more on packages BOSS and various participants.
• More test cases are expecting coming from BOSS developers/integrators or from upstream directly.

Target Users

• BOSS developers
• BOSS integrators
• BOSS QA

Use Cases

• For BOSS developer:
  • Modify participant A
  • Create or modify test cases upon his updating to A
  • Packaging work
  • Deploy BATS then deploy updated A, or update A in existing BATS
  • Trigger unit/functional testing
  • Trigger regression testing
  • If no error, publish A
• For BOSS integrator:
  • New BOSS package coming or new BOSS usage scenario coming
  • Change BATS configuration to include new package or update test case upon new scenario
  • Trigger integration testing and regression testing
  • If no error, add new package or scenario
• For QA:
  • Run various test cases upon BOSS updating

Design

• BATS is based on OTS. See our previous work: http://wiki.meego.com/Release_Infrastructure/BOSS_Testing/BOSS_Tesing_OTS
• The existed test cases from upstream are integrated into this system with the used test tools.
• BATS deploys the environment automatically by some scripts and configuration files
• Test cases are easy to add/modify based on OTS xml format
• A new test tool is implemented to covers the functionality test of participant and integration test of BOSS.
• Test result is base on OTS xml format. And it is easy to integrate existed report management system.
• More design detail is under discussion, update later...

Implementation

• Source code can be found at gitorious: (TBD)

Test Techniques in BATS

• For Unit/Functional test, following table lists existing packags/projects status:

Test Framework	Targets	Package/Project
RSepc	performance testing, functionality testing	BOSS, ruote-amqp, AIR ...
Test::Unit	unit test	ruote
py.test	unit test, functionality testing	MeeGo-IMG, AIR, ruote-amqp-pyclient, boss-participant-*, boss-launcher-*, ...
unittest	unit test	python-amqlib
participanttester.py	integration testing, functionality testing	boss-participant-*, BOSS

Current Test Status

Test Level	Test Coverage	Details
Unit	10%	http://wiki.meego.com/Release_Infrastructure/BOSS_Testing/BTF/Test_Status#Unit_Test
Functionality test	43%	http://wiki.meego.com/Release_Infrastructure/BOSS_Testing/BTF/Test_Status#Functionality_Test
Integration test	0	http://wiki.meego.com/Release_Infrastructure/BOSS_Testing/BTF/Test_Status#Integration_Test
Performance test	50%	http://wiki.meego.com/Release_Infrastructure/BOSS_Testing/BTF/Test_Status#Performance_Test

Use Guide

About how to get the engine scripts and start to triger the test framework, please refer to http://wiki.meego.com/Release_Infrastructure/BOSS_Testing/BTF/Usage