Views
ARM/N900/QA
(Fix some stuff, see http://lists.meego.com/pipermail/meego-qa/2011-September/002229.html) |
(Transfer useless image that only includes text into text) |
||
| Line 278: | Line 278: | ||
====Reporting bugs==== | ====Reporting bugs==== | ||
| - | *Bugs are reported in [https://bugs.meego.com/ | + | *Bugs are reported in [https://bugs.meego.com/ Bugzilla]. |
*[[Quality/How_To_Report_Bugs|How to report bugs]] | *[[Quality/How_To_Report_Bugs|How to report bugs]] | ||
| - | *File a new bug | + | *File a new bug: |
| - | [ | + | ** Search for an already existing report in [https://bugs.meego.com/query.cgi Bugzilla]. |
| - | + | ** If a similar report is found, check the Hardware Platform. If it is the same there is no need to file a new report. | |
| + | ** If the bug is in CLOSED or VERIFIED status and this status was set more than four weeks ago (you can check via the "History" link), file a new report (and mention the bug number of the old report) instead of reopening the existing report. | ||
| + | ** If no similar report is found, [https://bugs.meego.com/enter_bug.cgi file a new bug report]. | ||
====Questions or Doubts==== | ====Questions or Doubts==== | ||
Revision as of 15:18, 19 September 2011
Contents |
Meego Community Edition Quality Assurance
Quality Assurance for Meego Community Edition. Community Edition QA uses many same components as in core Meego, therefore remember to look Meego core quality page. Monitoring the Community Edition maturity is one of the main tasks of Community Edition QA. The current maturity status can be found from the Status page.
Organization
- Error Management
- Error Manager: Iekku Pylkkä
- QA Tools
- Core Testing
- User Experience Testing (UX)
Reports
Meetings
QA IRC meeting every Tuesday:
N900 CE (Blocker) Bug Triage meeting minutes:
- Meeting minutes 15-09-2011
- Meeting minutes 08-09-2011
- Meeting minutes 01-09-2011
- Meeting minutes 25-08-2011
- Meeting minutes 11-08-2011
- Meeting minutes 23-06-2011
- Meeting minutes 19-05-2011
- Meeting minutes 12-05-2011
- Meeting minutes 05-05-2011
- Meeting minutes 21-04-2011
- Meeting minutes 14-04-2011
- Meeting minutes 07-04-2011
- Meeting minutes 31-03-2011
- Meeting minutes 24-03-2011
Contact
- QA-mailing list
- join #meego-qa at freenode
QA Tools
Commumity Edition uses same QA Tools as in core Meego. For more information please refer to the Quality/QA-tools.
QA-Tools Task List
List of tasks the QA-Tools are doing for Meego Developer Edition. If you need something from QA please tell it to us :)
OTS setup
(Open Testing System)
- OTS server DONE
- OTS worker(s) for core tests- DONE
- Reporting of hourly tests to QA-reports - DONE
- Minimize automatic installation time to 10mins - DONE
- OTS worker for UX tests - NOTSTARTED
- OTS Worker(s) for DE tests - ONGOING (timakima)
- Power consumption measurements - NOTSTARTED
Test automation images
- Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
- We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
- Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED
Applications for testing
- qt-demos already available from the repos
- Small applications that use Qt mobility APIs to access things like sensors to help manual testing
Test asset
We are using mainly MCTS test assets, so please refer to the MCTS page. You can find list of open bugs also from there.
- List of open bugs for MCTS
- see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
- WLAN cases
- Call/SMS cases
- Audio policy framework cases (lower priority)
- Camera cases (lower priority)
- Sensor data cases (Qt Mobility, lower priority)
Crash analysis support for ARM core dumps
Core dump processing and backtraces from crashing ARM processes (click the image on right).
- Rich Core dumping
- Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
- Changes in MeeGo Gitorious (meego-n900de branch).
- Dumps are generated in /home/meego/core-dumps
- In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
- Get latest packages here
- Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
- Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
- Oopslog (and lifelog) functionality - NOT STARTED
- Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
- Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)
- Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)
- Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)
- testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)
- Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)
- After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
- testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
<crashes> <crash-id>1234567890ABCDEF</crash-id> <crash-id>1234567890ABCDEF</crash-id> <crash-id>1234567890ABCDEF</crash-id> </crashes>
- The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)
- The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)
- Upload processed crash data using Crash Reports API - DONE (rikhalon)
Boot time measurement
- Measure and optimize N900 boot time (timakima, ONGOING)
CPU load measurement during audio/video playback
Application Manager
- Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).
UX testing
Test execution schedule
- UX testing schedule: DE / Meego.com testing
| Day | Test set (status) | Release | Priority |
|---|---|---|---|
| Monday | Dataflow | DE Weekly | P1 |
| Monday | Use cases | DE Weekly | P2 |
| Monday | Key feature | DE Weekly | P3 |
| Monday | Performance | DE Weekly | P5 |
| Monday | Reliability / Iterative | DE Weekly | P6 |
| Monday | Dataflow | DE Trunk testing | P4 |
| Tuesday | Dataflow | DE Trunk testing | P1 |
| Tuesday | Dataflow | DE Trunk | P2 |
| Tuesday | Acceptance | Meego Trunk testing | P3 |
| Tuesday | Key feature | DE Tablet (N900) | P4 |
| Wednesday | Dataflow | DE Trunk testing | P1 |
| Wednesday | Dataflow | DE Trunk | P2 |
| Wednesday | Key feature | Meego.com weekly | P3 |
| Wednesday | Sanity | Meego.com weekly | P4 |
| Thursday | Dataflow | DE Trunk testing | P1 |
| Thursday | Dataflow | DE Trunk | P2 |
| Thursday | Acceptance | Meego.com Trunk testing | P3 |
| Friday | Dataflow | DE Trunk Testing | P1 |
| Friday | Dataflow | DE Trunk | P2 |
| Friday | Acceptance | Meego.com Trunk testing | P3 |
Performance Testing
Performance testing results done from UI can be found here
Functional Testing
Daily testing
Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.
- Acceptance (Trunk:Testing)
- Sanity (Trunk:Daily)
Weekly testing
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).
- Key Basic Testing
- Data Flow Testing
- Basic Feature Testing
Execution Process
Testing images Community Edition image
- Daily Testing image called a Acceptance image
- Daily Stable image called a Sanity image
- Every Monday stable image is called as Weekly Image
Execution
- Download Test Sets from gitorious Test Sets by using git
- List all test cases in a a spreadsheet application (LibreOffice/OpenOffice.org Calc, Microsoft Excel, etc.) and create a .csv file (Example csv file)
- Test cases verdict
Pass = If all the steps and Functionality is as per expected Fail = If any of the one step or Functionality is not as per expected N/A = If the test cases are not executable due to the absence of related package in the image, the test results should be N/A Blocked = This is a special value required in the situations where verdict cannot be given due to failure in preconditions
- Upload Test report to with respective Test cycle in QA reporting tool QA Reporting tool
Reporting bugs
- Bugs are reported in Bugzilla.
- How to report bugs
- File a new bug:
- Search for an already existing report in Bugzilla.
- If a similar report is found, check the Hardware Platform. If it is the same there is no need to file a new report.
- If the bug is in CLOSED or VERIFIED status and this status was set more than four weeks ago (you can check via the "History" link), file a new report (and mention the bug number of the old report) instead of reopening the existing report.
- If no similar report is found, file a new bug report.
Questions or Doubts
Any question please leave your query in #meego-arm (irc.freenode.net) or you can meego-qa mailing list
Suggestion about updating of this page Srikanth Yarlagadda
Questions about execution and reporting please ping me in IRC srikanth_rst
Core Testing
QA Tasks For Developer Edition
QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in Developer Edition Targets. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.
Test Sets
Sanity Test Set
The sanity set should be run automatically on every image. As such it must meet the following requirements:
- 100% automated
- Testing only basic features
Feature Test Set
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.
Suggestions are welcome.
Testing schedule
| Day | Test set (status) | Release |
|---|---|---|
| Monday | Feature | DE Weekly |
| Tuesday | Sanity | DE Trunk testing |
| Wednesday | Sanity | DE Trunk testing |
| Thursday | Sanity | DE Trunk testing |
| Friday | Sanity | DE Trunk Testing |
Core QA Team Backlog
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:
Continuous tasks
- DE testing
- Bug verification
- Wiki clean/update
Backlog
- Reliability testset planning
- 1.3 Vanilla vs 1.2 DE current measurements
- DE Filesystem benchmark
In progress
- DE Hourly Automation improvement
Done
- Qt API test investigation
- Wiki clean/update (waiting for comments)
- Testcase automation list
- Plan Feature testset (.xml updated)
- Create weekly schedule for MRT (currently in draft form)
- Week 15 DE Sanity Testing
- Maturity statement of Alpha RELEASE (result in QA-report)
- Alpha RELEASE testing
QA Tasks for MeeGo.com N900
Test execution schedule
- MeeGo.com N900 Core weekly test schedule for MeeGo1.3
| Day | Test set (status) | Release | Priority |
|---|---|---|---|
| Monday | Acceptance test (OK)& test for changes | MeeGo.com trunk testing | P1 |
| Monday | Basic feature test (OK) | MeeGo.com pre-weekly | P2 |
| Tuesday | Acceptance test (OK)& test for changes | MeeGo.com trunk testing | P1 |
| Tuesday | Sanity test (OK) | MeeGo.com trunk | P2 |
| Wednesday | Dataflow (OK) | MeeGo.com weekly | P1 |
| Wednesday | Acceptance test (OK)& test for changes | MeeGo.com trunk testing | P2 |
| Thursday | Acceptance test (OK)& test for changes | MeeGo.com trunk testing | P1 |
| Thursday | Sanity test (OK) | MeeGo.com trunk | P2 |
| Friday | Acceptance test (OK)& test for changes | MeeGo.com trunk testing | P1 |
| Friday | Sanity test (OK) | MeeGo.com trunk | P2 |
META team backlog for MeeGo1.2 N900
Continuous tasks
- Bug verification
- MeeGo1.3 N900 daily validation
Backlog
- Optimize acceptance automation script
- Testability for MeeGo1.3 features.
In progress
- Automation testing for trunk:test and trunk image
Done
- Publish automation test result
- MeeGo1.2 feature verification
Usefull links
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/