Meego Wiki
Views

ARM/N900/QA

From MeeGo wiki
(Difference between revisions)
Jump to: navigation, search
(→Crash analysis support for ARM core dumps)
(→Crash analysis support for ARM core dumps)
Line 78: Line 78:
** Oopslog (and lifelog) functionality - NOT STARTED
** Oopslog (and lifelog) functionality - NOT STARTED
-
* Back-end server
+
* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)
-
** Set up back-end server for core processing - ONGOING (rikhalon, sampos)
+
-
* Script to produce backtraces locally
+
* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)
-
** Extend rich-core-extract to proceduce backtrace from rich-core file - NOT STARTED
+
 
-
** Write howto guide in wiki - NOT STARTED
+
* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- ONGOING (rikhalon)
 +
 
 +
* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)
 +
 
 +
* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)
 +
 
 +
* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - NOT STARTED
 +
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
 +
 
 +
<crashes>
 +
  <crash-id>1234567890ABCDEF</crash-id>
 +
  <crash-id>1234567890ABCDEF</crash-id>
 +
  <crash-id>1234567890ABCDEF</crash-id>
 +
</crashes>
 +
 
 +
 
 +
* The core processing backend extracts the rich core and looks for proper target (if not available waits...) -  PofC DONE (minus the waiting part) (sampos)
 +
 
 +
* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - Tested manually, untested python script exists (sampos)
 +
 
 +
* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - NOT STARTED
== Boot time measurement ==
== Boot time measurement ==

Revision as of 11:19, 10 May 2011

Contents

General info

Quality Assurance for Meego Developers Edition.

Organization

  • Error Management
    • Error Manager: Iekku Huttunen
  • QA Tools
  • Core Testing
  • User Experience Testing (UX)

Reports

Meetings

QA IRC meeting every Tuesday:

N900 DE Blocker Bug Triage meeting minutes:

Tools

OTS setup and automated hourly testing

OTS setup

  • OTS server - Ville Ilvonen/Riku Halonen, DONE
  • OTS worker(s) for core tests- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
  • Reporting of hourly tests to QA-reports - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
  • Minimize automatic installation time to 10mins - Timo Makimattila, DONE
  • OTS worker for UX tests - NOTSTARTED
  • OTS Worker(s) for DE tests - ONGOING (timakima)
  • Power consumption measurements - NOTSTARTED

Test automation images

  • Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
  • Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

Applications for testing

  • qt-demos already available from the repos
  • Small applications that use Qt mobility APIs to access things like sensors to help manual testing

Test asset

  1. WLAN cases
  2. Call/SMS cases
  3. Audio policy framework cases (lower priority)
  4. Camera cases (lower priority)
  5. Sensor data cases (Qt Mobility, lower priority)

Crash analysis support for ARM core dumps

Automated testing and crash reporting

Core dump processing and backtraces from crashing ARM processes (click the image on right).

  • Rich Core dumping
    • Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
      • Changes in MeeGo Gitorious (meego-n900de branch).
      • Dumps are generated in /home/meego/core-dumps
      • In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
      • Get latest packages here
      • Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
    • Fix core-reducer (Something goes wrong, when reducer processes coredump.) - DONE (sampos)
    • Oopslog (and lifelog) functionality - NOT STARTED
  • Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)
  • Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)
  • Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- ONGOING (rikhalon)
  • testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)
  • Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)
  • After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - NOT STARTED
    • testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core. 
<crashes>
 <crash-id>1234567890ABCDEF</crash-id> 
 <crash-id>1234567890ABCDEF</crash-id>
 <crash-id>1234567890ABCDEF</crash-id>
</crashes>


  • The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (minus the waiting part) (sampos)
  • The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - Tested manually, untested python script exists (sampos)

Boot time measurement

  • Measure and optimize N900 boot time (timakima, ONGOING)

CPU load measurement during audio/video playback

Application Manager

  • Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

UX testing

Test execution schedule

  • UX testing schedule: DE / Meego.com testing
Day Test set (status) Release Priority
Monday Dataflow DE Weekly P1
Monday Use cases DE Weekly P2
Monday Key feature DE Weekly P3
Monday Performance DE Weekly P5
Monday Reliability / Iterative DE Weekly P6
Monday Dataflow DE Trunk testing P4
Tuesday Dataflow DE Trunk testing P1
Tuesday Dataflow DE Trunk P2
Tuesday Acceptance Meego Trunk testing P3
Tuesday Key feature DE Tablet (N900) P4
Wednesday Dataflow DE Trunk testing P1
Wednesday Dataflow DE Trunk P2
Wednesday Key feature Meego.com weekly P3
Wednesday Sanity Meego.com weekly P4
Thursday Dataflow DE Trunk testing P1
Thursday Dataflow DE Trunk P2
Thursday Acceptance Meego.com Trunk testing P3
Friday Dataflow DE Trunk Testing P1
Friday Dataflow DE Trunk P2
Friday Acceptance Meego.com Trunk testing P3

Performance Testing

Performance testing results done from UI can be found here

Core Testing

QA Tasks For Developer Edition

There is a wiki article about the Developer Edition.

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

Test Sets

Sanity Test Set

The sanity set should be run automatically on every image. As such it must meet the following requirements:

  • 100% automated
  • Testing only basic features

Feature Test Set

The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

Testing schedule

Day Test set (status) Release
Monday Feature DE Weekly
Tuesday Sanity DE Trunk testing
Wednesday Sanity DE Trunk testing
Thursday Sanity DE Trunk testing
Friday Sanity DE Trunk Testing

Core QA Team Backlog

The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

Continuous tasks

  • Bug verification

Backlog

In progress

  • Plan Feature testset
  • Create weekly schedule for MRT (currently in draft form)
  • DE Hourly Automation improvement
  • Wiki clean/update

Done

  • Week 15 DE Sanity Testing
  • Maturity statement of Alpha RELEASE (result in QA-report)
  • Alpha RELEASE testing

QA Tasks for MeeGo.com N900

Test execution schedule

  • MeeGo.com N900 Core weekly test schedule for MeeGo1.2(Tu Qingqing)
Day Test set (status) Release Priority
Monday Acceptance test (OK)& test for changes MeeGo.com trunk testing P1
Monday Sanity test (OK) MeeGo.com trunk P3
Monday, Tuesday Basic feature test (OK) MeeGo.com pre-weekly P2
Tuesday Acceptance test (OK)& test for changes MeeGo.com trunk testing P1
Tuesday Sanity test (OK) MeeGo.com trunk P2
Wednesday Acceptance test (OK)& test for changes MeeGo.com trunk testing P2
Wednesday Sanity test (OK) MeeGo.com trunk P3
Wednesday, Thursday Dataflow (OK) MeeGo.com weekly P1
Thursday Acceptance test (OK)& test for changes MeeGo.com trunk testing P1
Thursday Sanity test (OK) MeeGo.com trunk P2
Friday Acceptance test (OK)& test for changes MeeGo.com trunk testing P1
Friday Sanity test (OK) MeeGo.com trunk P2

META team backlog for MeeGo1.2 N900

Continuous tasks

  • Bug verification
  • MeeGo1.2 N900 daily validation

Backlog

  • Optimize acceptance automation script

In progress

  • MeeGo1.2 feature verification
  • Automation testing for trunk:test and trunk image

Done

  • Publish automation test result

Usefull links

Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

Retrieved from "http://wiki.meego.com/ARM/N900/QA"
Personal tools