Contents |
The status of 0.8 wrt ots.system_tests (Updated 2010-01-06)
| Name | Status |
|---|---|
| test_testrun_with_test_definition_tests | See "Image Problems" below |
| test_host_based_testrun_with_test_definition_test | See "Image Problems" below |
| test_hw_and_host_based_testrun_with_test_definition_tests | |
| test_bad_image_url | Works after minor fixes* |
| test_timeout | Works after minor fixes* |
| test_non_existing_devicegroup | Works after minor fixes* |
| test_non_existing_sw_product | Works after minor fixes* |
| test_bad_testpackage_names | Works after minor fixes* |
| test_no_image_url | Works after minor fixes* |
| test_bad_distribution_model | Distribution model needs fixing first |
| test_perpackage_distribution_no_packages | Distribution model needs fixing first. |
| test_multiple_devicegroups | Not done yet |
| test_one_devicegroup_multiple_devicenames | Not done yet |
| test_one_devicegroup_one_devicename_multiple_device_ids | Not done yet |
I'm ashamed to say that I don't understand the return values CITA giving the overall status of the Testrun.
This behaviour of the Python unittest makes sense to me and it seems that in some respects the logic in CITA is the same e.g.the example above. But the current "ERROR" behaviour is unclear.
| Passes | Errors | Success |
|---|---|---|
| False | False | False |
| True | False | False |
| False | True | False |
| True | True | True |
I'm having problems running the tests with the image provided; ots_system_test_image.tar.gz. Being able to identify and communicate the images we are working with would seem necessary - do they have a hash?
On the image provided I get this error when running the generated command through the conductor direct:
File "/home/tom/projects/ots/ots.worker/ots/worker/command.py", line 118, in execute_with_retries raise FailedAfterRetries
FailedAfterRetries 2011-01-05 09:09:09,015 INFO Testing in Hardware ended with error
Running this hangs...
0:06:00.086145 2011-01-05 19:23:14.224707 tom-desktop executor WARNING Retrying (1) to fetch file after (5) seconds sleep... 0:06:05.772793 2011-01-05 19:23:19.911355 tom-desktop executor WARNING Failed to fetch file /var/log/syslog (command scp root@192.168.2.15:/var /log/syslog /home/tom/conductor/a37fe3b418ef11e0b2dd0025648621f1/Host_Hardware/post returned 1) 0:06:05.946484 2011-01-05 19:23:20.085046 tom-desktop executor WARNING Retrying (2) to fetch file after (5) seconds sleep... 0:06:11.633024 2011-01-05 19:23:25.771586 tom-desktop executor WARNING Failed to fetch file /var/log/syslog (command scp root@192.168.2.15:/var/log/syslog /home/tom/conductor/a37fe3b418ef11e0b2dd0025648621f1/Host_Hardware/post returned 1) 0:06:11.782244 2011-01-05 19:23:25.920806 tom-desktop executor INFO Storing testrun log file /var/log/testrun.log
A similar thing happened using the generated command and running on the Conductor:-
2011-01-05 18:07:04,394 WARNING Retrying (1) to fetch file after (5) seconds sleep... 2011-01-05 18:07:19,550 WARNING Failed to fetch file /var/log/syslog (command scp root@192.168.2.15:/var/log/syslog /home/tom/conductor/a37fe3b418ef11e0b2dd0025648621f1/Host_Hardware/post timed out) 2011-01-05 18:07:19,702 WARNING Retrying (2) to fetch file after (5) seconds sleep... 2011-01-05 18:07:34,855 WARNING Failed to fetch file /var/log/syslog (command scp root@192.168.2.15:/var/log/syslog /home/tom/conductor/a37fe3b418ef11e0b2dd0025648621f1/Host_Hardware/post timed out) 2011-01-05 18:07:35,020 INFO Storing testrun log file /var/log/testrun.log
| Problem | Reporter | Exposed by | Description | Short Term Resolution | Long Term Recommendation | New unittest |
|---|---|---|---|---|---|---|
| File recognition | TG | soak_test.py | Results XML recognition by magic name "tatam_xml_testrunner" | Patch with 0.1 implementation | Formalised file transfer mechanism (e.g Mime Types and xsi instance in XML) | |
| Response Queue | TG | soak_test.py | Worker tries to post to response queue that isn't there | Non-assertive posting for logging. Merged back in 0.1 | Tighten protocol for Queue shut down. Robust behaviour for orphaned workers | test_no_response_queue |
| Conductor Commands | TV | ots.system_tests | Regression on command construction | Code fixes | Conductor will have an API in 0.9 | test_conductor_commands_regression |
| Publisher Bootstrapping | TV | ots.system_tests | http logs not seen when Value Errors on input parameters before Publisher boot | Sandboxing and forced adaptation of Publisher parameters | Is this a reasonable requirement. Can we refine the Publisher API to avoid this kludge | test_hub.TestHubFailSafePublishing |
| Orphaned Messages | E-P | soak_tests.py | Timed out messages picked up by new workers leading to Orphaned worker | Not addressed! (Not arising from changes and no reported cases in Production) | e.g. System synchronised clocks and "best before" stamp on Task Message | |
| Missing Logs | tv | ots.system tests | Essential log messages missing | Reinstated | N/A | django_ots.TestView |
| Result return Value | tv | ots.system tests | WIP |