Quality/QA-tools/OTS/DeveloperDocs/Experimental

Contents 1 Experimental Area 1.1 0.1 1.2 0.8.2 1.3 How to monitor ots? 1.3.1 Who receives monitor dto:s? 1.3.2 WebUI performance 1.3.3 TODO 1.4 Bug 9036 - [FEA] Test package distribution based on history (last execution time of a package)

Experimental Area

Documentation for experimental branches, spikes, WIPs etc. You can also add documentation related to your merge requests or patches here.

The docs should be moved to proper place in OTS wiki after the feature gets integrated to master branch.

This area is relevant for OTS developers only! Information under this area does not apply to master or released OTS versions.

0.1

• Advanced OTS server using django (Released in 0.1.1 version)

0.8.2

How to monitor ots?

We have monitor DTO:s. Probably some kind of hierarchy of dto monitor objects.

• Testrun Events for all testrun states (flashing, bootup, testing, test result processing...):
  • Test pkg execution event (start/stop) (minimum requirement for [FEA] 9036)
  • Flashing Event (start/stop) (n:th try)
  • Bootup event (n:th try)
  • Testsuite/set/case/step event? (What level of detail we want to monitor?) Remember that logging still exists...
  • Testrun overall result

• Worker events:
  • Worker state (Start/stop/busy(testrun id)/free) (information about queues)
  • Heartbeat?

• Failure event dto? (Flashing failed, device failure...)?

What about statistics monitoring? What do we need?

• Number of testruns?
• Number of workers?
• Active testruns?
• PASS/FAIL/ERROR ratios?
• Error statistics?
  • Most common errors
  • Some kind of grouping of errors and statistics for them (user fault vs catastrophic ots failure vs device/image error vs...)

Notes:

• One host can have multiple worker instances running. Dto:s need to have information about the instance, not just hostname.

Who receives monitor dto:s?

Current implementation uses the same testrun response queue and process for monitor dto:s. This is simple to implement for testrun related events but very limited for generic monitoring of the whole system. For example most of the worker events cannot be sent to testrun queue because they are not related to a specific testrun and we don't know if there even is any testrun response queues around.

Having a separate permanent queue and a separate listener process for monitoring would allow monitoring also outside a testrun.

Pros:

• Can be deployed as a completely separate component
  • even into a different machine
  • Maintenance, update etc. does not affect test execution in any way

Cons:

• One more process/daemon to maintain
  • More complicated install
• Might cause errors.
  • Not running => Lots of messages pile up in the queue
• Dependencies
  • What if for example the distribution model relies on monitor component to provide up to date data?
• How to provide testrun specific monitor info to Publishers?

WebUI performance

Tested with sqlite and MySQL It looks like that MySQL does some caching because when a page is loaded for the first time it takes much longer to render the data than on following loads. This means that on fixed time frame the pages can be loaded quite quickly, but when the time frame changes then the page loading takes more time.
200000 testruns and 1160443 events within fixed time frame average page loading time:

page	sqlite	MySQL
main	4.5322	0.0378
group details	61.3120	28.2552
requestor details	0.5142	0.0653
testrun details	0.0033	0.0128
testrun list	1.0347	0.0730

200000 testruns and 1160443 events within dynamic time frame:

page	MySQL
main	6.4605
group details	43.5432
requestor details	0.2901
testrun details	0.0957
testrun list	0.5161

1000 testruns and 5802 events average page loading time:

page	sqlite
main	0.0619
group details	0.3595
requestor details	0.0111
testrun details	0.0030
testrun list	0.0173

TODO

• Current server side implementation does not send monitor dtos to Publishers in real time. They are sent only after the testrun is done.

Bug 9036 - [FEA] Test package distribution based on history (last execution time of a package)

Proposal:

• Data provided by monitor dto:s
• Data stored to a DB
  • A separate DB for this plugin only or dependency to a more common monitor DB?
• A custom distribution model that reads data from DB and creates tasks based on that

Notes:

• We cannot limit to testpackage level. Implementation for set level distribution is already ongoing.