Views
Release Infrastructure/BOSS Testing/BTF/Usage
<Under editing>
Contents |
User Guide
This presentation is to describe the details about how to use this test framework to test BOSS and participants. BATS system support openSUSE 11.2 (verified).
Step1: Deploy Environment
All the codes of BATS have been packaged and uploaded to relevant repositories. To deployment the test environemt, please follow the steps:
1. Add repositories url(openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo zypper ar http://maemo-repo.europe.nokia.com/Tools:/MeeGoInfra/openSUSE_11.2/Tools:MeeGoInfra.repo zypper ar http://download.opensuse.org/repositories/devel:/languages:/ruby:/extensions/openSUSE_11.2/devel:languages:ruby:extensions.repo zypper ar http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_11.2/devel:languages:python.repo zypper ar http://download.opensuse.org/repositories/openSUSE:/Tools/openSUSE_11.2/openSUSE:Tools.repo zypper ar http://download.meego.com/live/Tools:/Building/openSUSE_11.2/Tools:Building.repo zypper ar http://download.opensuse.org/repositories/home:/victorliu:/bats/openSUSE_11.2/home:victorliu:bats.repo zypper ref
2. Install BATS package
zypper in bats
- After bats package installed, BOSS, participants, BATS scripts, BATS QA Reports system, Testparticipant and all their dependencies shall be installed
3. Configure and start BATS QA Reports system
bats-qa-reports setup bats-qa-reports start
- This step may take a while since we need to solve the dependency issue and configure QA Report system. After it finished, QA Report system shall startup, you can see it by "http://localhost:3000"
- Try "bats-qa-reports -h" to get more usage
4. Download sample test suites, then have a try
git clone http://git.gitorious.org/bats/bats-test-suites.git cd bats-test-suites bats-testrunner -c bats-test-ruote.conf # check results in your BATS QA reports system
- After testing finished, you will get one mail per configuration in /etc/bats.conf(please change to your mail address)
Then, the whole test environment is ready include BOSS, participants (not all participants are depoyed now, such as imager), BATS and all existed test cases has been integrated into BATS.
Step2: Launch BOSS and Participants
In this step, you need to launch the BOSS and the participants you want to test.
1. start boss
boss
2. config and start need participants (repeat)
configure participants in /etc/boss service <participant_name> start
Step3: Config BATS
Before running the test script, you have a chance to choose executing testing with default setting or customize the testing. There are 2 config files inside BATS to specify the test sets, the QA report system account, etc... Follows are the sample of global config file (default located in /etc/bats.conf):
# user name in the BATS QA report system USER_NAME=boss # user email , user can login BATS QA report system by this email to operate on test result. # user also gets test result notification mail through this address USER_EMAIL=zhi.dou@nokia.com # user password is used by user to login BATS QA report system USER_PASSWORD=88888888
The other one is to be specified the test sets which will be executed:
# root directory used to run these test cases TEST_PLAN_DIR=/home/boss/Downloads/meego-infrastructure-tools-boss/ruote/ruote/test # test plan xml file TEST_PLAN_XML=/home/boss/work/bats-test-suites/ruote-2.1-func-test.xml # 1st test category, test target can be "ruote", "participant" etc... TEST_TARGET=ruote # 2st test category, test version must only contains letter, number and "_" TEST_VERSION=ruote_2_1 # 3st test category, test type can be "functional", "unit" and "performance" etc... TEST_TYPE=functional # repeat above to specify another test set # ...
or as a XML file:
<?xml version="1.0" encoding="UTF-8"?>
<test_plans>
<test_plan>
<test_plan_dir>/usr/lib/ruby/gems/1.8/gems/ruote-2.1.11/test</test_plan_dir>
<test_plan_xml>/home/boss/work/bats-test-suites/ruote-2.1-func-test.xml</test_plan_xml>
<test_target>engine</test_target>
<test_type>functional</test_type>
<test_version>ruote_2_1</test_version>
</test_plan>
<test_plan>
...
</test_plan>
...
</test_plans>
Deploy a BATS QA Reports System on remote machine
Defaultly each BATS environment includes one BATS QA Reports system, you can run test and check results on same host. If you want a remote or centralized BATS QA Reports system to hold all test reports from multiple BATS environment on different hosts(clients), you need to do following steps on remote host:
1. Deploy BATS QA Reports system and mysql
zypper ar http://download.opensuse.org/repositories/home:/victorliu:/bats/openSUSE_11.2/home:victorliu:bats.repo zypper in bats-qa-reports mysql bats-qa-reports setup
2. Configure the BATS QA Reports system
change /srv/www/bats-qa-reports/config/database.yml as following example:
development:
adapter: mysql
host: localhost
database: db_test
username: tester
password: 12345
encoding: utf8
timeout: 5000
3. Create and migrate database with following commands:
cd /srv/www/bats-qa-reports bin/rake db:create bin/rake db:migrate
4. Configure database to grant rights for client accessing(take "db_test" as example database and "tester" as user)
mysql -h localhost -u root -p GRANT ALL ON db_test.* TO 'tester'@'%' IDENTIFIED BY '12345'; FLUSH PRIVILEGES;
5. Start rails server
bats-qa-reports start
After this, you can access your QA reports system in browser from anywhere by "http://<remote IP>:3000"
Connect to a remote QA Reports System
On client side, connect a remote QA report system to upload the test results to this server.
1. Deploy BATS following introduction in "Step 1"
2. Configure the BATS QA Reports system
change /srv/www/bats-qa-reports/config/database.yml as following example:
delopment:
adapter: mysql
host: <server name/server IP points to remote host>
database: db_test
username: tester
password: <password>
encoding: utf8
timeout: 5000
3. Execute test and upload test result, refer to Step4.
Add a Test Set
All the test cases are organized as XML file. A XML file means a test set(plan). You can config this testing with several test sets. To add a new test set, just need append an additional section in test.cfg file.
About the details of the test plan xml file, please refer to http://wiki.meego.com/Quality/QA-tools/Test_plan
Add new Test Case
BTAS invokes different test tools/framworks to do the real testing. Follow table is the list of test tools used by BATS and relevant guide. You can refer the sample test cases also.
| Test Framework | Test Level | Guide |
|---|---|---|
| RSepc | performance, functionality | http://rspec.info |
| Test::Unit | unit | http://test-unit.rubyforge.org/test-unit/ |
| py.test | unit, functionality | http://en.wikipedia.org/wiki/Py.test |
| unittest | unit | http://docs.python.org/library/unittest.html |
| participanttester.py | integration, functionality (participant) | http://wiki.meego.com/Release_Infrastructure/BOSS_Testing/Participants_Testing |
You also can refer to the existed test cases we have implemented (mentioned in Step1).
If the new test cases are ready, you need change the exist test plan (http://wiki.meego.com/Quality/QA-tools/Test_plan ) or create a new one ("Add a Test Set")
Measure Code Coverage
Two code coverage tools are installed in BATS environment - "rcov" for Ruby and "coverage" for Python. To use them getting code coverage for ruby code in BOSS engine and Python code in various participants, you need to start up boss and participants with these tools. Then you can get code coverage reports after executing your test cases.
- User guide for each tool:
- rcov - http://eigenclass.org/hiki/rcov
- coverage - http://nedbatchelder.com/code/coverage/
Step4: Execute Test
bast-testrunner is used to run testsuite and import test result to bats-qa-reports.
Usage
-----
bats-testrunner [Options]
For help use: bats-testrunner -h
Options
-------
-h, --help Displays help message
-v, --version Display the version, then exit
-V, --verbose Verbose output
-c <bats_cnf>, --config <bats_cnf> set <bats_cnf> as BATS test config
During the execution, bast-testrunner will generate the test result as a xml file for per test plan. Then upload the results to QA report system. After all the activities are done, you will get an email with the link of test result. Such as:
Step5: Check Test Result
1. startup the bats-qa-reports under bats-qa-reports root directory:
rails server
2. check http://localhost:3000/boss to access local QA report
