This page is to talk about the way of BOSS performance testing.
We created the test suit(http://gitorious.org/boss-performance-test) including:
In this case set, we test the rates for launching different workflow scales(for N=[several hundreds, ..., several thousands]). Here we also compared the situation of different number of workers. And the CPU/MEM/DISK load is record.
* Raw data and graph
* CPU/MEM/DISK load for 10k and 20k cases
From above results, we can find some interesting things:
* Rates are decreasing while more number of workflows launched once time; that means the performance is decreasing when more request coming. * CPU/DISK is at high rate during testing * Memory is still taken after workflow finished. It looks like "memory leak" but it may not be if considering the "warm cache(or GC)" in Ruby/Python like language * Even multiple workers, if they all run in one machine, performance can not be boost. This is as excepted considering the CPU/DISK load * Some workflows lost in some test cases. Need further investigation * Observed engine crashed some times. Need further investigation
Here we got the "reponse time" testing results. The way is to repeat to launch 1k workflows each time after previous 1k workflows finished. Observing the durations for executing each 1k workflows we can get the response time trend. Following are results on VM and HW
From above results, reponse time is increasing while handling more and more workflows; that means the performance(or rates) is decreasing. So BOSS could be very low capacity after servicing for long time. This is a pontential issue and need further investigation.