Release Infrastructure/BOSS/Performance/Results

Revision as of 13:40, 28 November 2010

This page is to talk about the way of BOSS performance testing.

Concepts

• "Rate" is to measure the throughput of BOSS - how many workflows can be handled in one second. With the "Rate" of different workflow request scales, we will have a view of the BOSS capability for different service levels.
• "Load": how many requests(workflows) sending to engine at same time
• "Iteration": one iteration begins at sending specific number of load workflows to engine, ends at engine finishing all received workflows

Test Method

• The basic method in this test project is to simulate boss using in real world - running as a service for long time and dealing with multiple requests from multiple users continually; Then observe the performance data to get the evaluation
• The way to get "Rate" is straightforward. In each test case, sending specified number of workflows(such 3000, 20000...) for on shot or iteratively to BOSS then record the corresponding start and end time. Then it's easy to get the "Rate" for that test case.

Test Environment

• Hardware: CPU E5520(x16 cores), RAM 16G, openSUSE 11.2
• BOSS packages and all its dependency taken from OBS(https://build.opensuse.org/project/show?project=Maemo%3AMeeGo-Infra%3ABOSS)
• A patch to fix crash issue caused by AMQP channel creating bug(https://github.com/kennethkalmer/ruote-amqp/issues/issue/3/)

Test Scripts

• Test suite code can be found from project "boss-performance-test"(http://meego.gitorious.org/meego-infrastructure-tools/boss-performance-test), which including:
  • Config files to help set up testing environment
  • Scripts to simulate BOSS using in real world
  • Utilities to help analyzing various test results
• A "lite" version is also available on the branch "New":
  • It uses your current BOSS environment directly rather than starting another new BOSS instance

Test Cases And Test Results

* Test case: Compare performance using single worker and multiple workers

• • Multiple workers run on the same host
  • Get rate by testing following config then calculate the average rate
    • FS storage
    • load: 1k
    • iteration: 2
  • Test results:
    • 
  • Conclusion
    • Running multiple workers on same host will not increase the performance

* Test case: Compare performance using different load

• • Get rate by testing following config then calculate the average rate
    • FS storage
    • 1 worker
    • load: 300, 500, 1k, 3k, 5k, 8k, 10k, 20k, 30k, 50k
    • iteration: 1 for each load
  • Test results:
    • 
  • Conclusion
    • Performance is decreasing while load increasing

* Test case: Observe the performance for long time running

• • Get rate by testing following config then calculate the average rate
    • FS storage
    • 1 worker
    • load: 1k
    • iteration: 3200(running for two days, about 3.2 million workflows totally))
  • Test results:
    • 
  • Conclusion
    • Performance decreased 25% during beginning 200 iterations then keep stable(<10%) for rest 3000 iterations - much better than before
    • CPU and DISK is almost occupied all the time - it's normal as expectation
    • Memory is increasing continuously - it could be an issue
    • No crash and No workflow lose - good!