MeeGo wiki - User contributions [en]

HandsetBugTriageMinutesArchive

2011-08-24T07:54:15Z

Srikanthyarlagadda:

*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-08-24-07.00.html MeeGo Handset UX Bug Triage 20110824]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-08-17-07.00.html MeeGo Handset UX Bug Triage 20110817]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-08-03-06.57.html MeeGo Handset UX Bug Triage 20110803]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-07-20-06.58.html MeeGo Handset UX Bug Triage 20110720]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-07-13-06.57.html MeeGo Handset UX Bug Triage 20110713]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-07-06-07.12.html MeeGo Handset UX Bug Triage 20110706]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-29-06.57.html MeeGo Handset UX Bug Triage 20110629]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-22-06.56.html MeeGo Handset UX Bug Triage 20110622]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-15-06.57.html MeeGo Handset UX Bug Triage 20110615]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-08-06.57.html MeeGo Handset UX Bug Triage 20110608]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-18-06.59.html MeeGo Handset UX Bug Triage 20110518]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-11-06.58.html MeeGo Handset UX Bug Triage 20110511]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-04-06.59.html MeeGo Handset UX Bug Triage 20110504]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-20-07.00.html MeeGo Handset UX Bug Triage 20110420]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-13-07.00.html MeeGo Handset UX Bug Triage 20110413]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-06-07.00.html MeeGo Handset UX Bug Triage 20110406]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-30-06.58.html MeeGo Handset UX Bug Triage 20110330]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-23-06.58.html MeeGo Handset UX Bug Triage 20110323]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-16-07.04.html MeeGo Handset UX Bug Triage 20110316]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-09-07.01.html MeeGo Handset UX Bug Triage 20110309]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-02-06.59.html MeeGo Handset UX Bug Triage 20110302]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-23-06.59.html MeeGo Handset UX Bug Triage 20110223]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-16-06.59.html MeeGo Handset UX Bug Triage 20110216]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-09-06.59.html MeeGo Handset UX Bug Triage 20110209]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-02-06.59.html MeeGo Handset UX Bug Triage 20110202]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-26-06.59.html MeeGo Handset UX Bug Triage 20110126]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-19-07.18.html MeeGo Handset UX Bug Triage 20110119]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-12-07.00.html MeeGo Handset UX Bug Triage 20110112]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-05-07.01.html MeeGo Handset UX Bug Triage 20110105]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-22-07.00.html MeeGo Handset UX Bug Triage 20101222]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-15-07.00.html MeeGo Handset UX Bug Triage 20101215]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-08-07.00.html MeeGo Handset UX Bug Triage 20101208]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-01-07.01.html MeeGo Handset UX Bug Triage 20101201]

HandsetBugTriageMinutesArchive

2011-08-17T08:16:15Z

Srikanthyarlagadda:

*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-08-17-07.00.html MeeGo Handset UX Bug Triage 20110817]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-08-03-06.57.html MeeGo Handset UX Bug Triage 20110803]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-07-20-06.58.html MeeGo Handset UX Bug Triage 20110720]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-07-13-06.57.html MeeGo Handset UX Bug Triage 20110713]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-07-06-07.12.html MeeGo Handset UX Bug Triage 20110706]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-29-06.57.html MeeGo Handset UX Bug Triage 20110629]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-22-06.56.html MeeGo Handset UX Bug Triage 20110622]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-15-06.57.html MeeGo Handset UX Bug Triage 20110615]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-08-06.57.html MeeGo Handset UX Bug Triage 20110608]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-18-06.59.html MeeGo Handset UX Bug Triage 20110518]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-11-06.58.html MeeGo Handset UX Bug Triage 20110511]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-04-06.59.html MeeGo Handset UX Bug Triage 20110504]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-20-07.00.html MeeGo Handset UX Bug Triage 20110420]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-13-07.00.html MeeGo Handset UX Bug Triage 20110413]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-06-07.00.html MeeGo Handset UX Bug Triage 20110406]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-30-06.58.html MeeGo Handset UX Bug Triage 20110330]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-23-06.58.html MeeGo Handset UX Bug Triage 20110323]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-16-07.04.html MeeGo Handset UX Bug Triage 20110316]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-09-07.01.html MeeGo Handset UX Bug Triage 20110309]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-02-06.59.html MeeGo Handset UX Bug Triage 20110302]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-23-06.59.html MeeGo Handset UX Bug Triage 20110223]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-16-06.59.html MeeGo Handset UX Bug Triage 20110216]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-09-06.59.html MeeGo Handset UX Bug Triage 20110209]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-02-06.59.html MeeGo Handset UX Bug Triage 20110202]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-26-06.59.html MeeGo Handset UX Bug Triage 20110126]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-19-07.18.html MeeGo Handset UX Bug Triage 20110119]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-12-07.00.html MeeGo Handset UX Bug Triage 20110112]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-05-07.01.html MeeGo Handset UX Bug Triage 20110105]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-22-07.00.html MeeGo Handset UX Bug Triage 20101222]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-15-07.00.html MeeGo Handset UX Bug Triage 20101215]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-08-07.00.html MeeGo Handset UX Bug Triage 20101208]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-01-07.01.html MeeGo Handset UX Bug Triage 20101201]

HandsetBugTriageMinutesArchive

2011-08-17T08:15:58Z

Srikanthyarlagadda:

*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-08-17-07.00.html MeeGo Handset UX Bug Triage 201108017]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-08-03-06.57.html MeeGo Handset UX Bug Triage 20110803]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-07-20-06.58.html MeeGo Handset UX Bug Triage 20110720]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-07-13-06.57.html MeeGo Handset UX Bug Triage 20110713]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-07-06-07.12.html MeeGo Handset UX Bug Triage 20110706]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-29-06.57.html MeeGo Handset UX Bug Triage 20110629]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-22-06.56.html MeeGo Handset UX Bug Triage 20110622]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-15-06.57.html MeeGo Handset UX Bug Triage 20110615]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-08-06.57.html MeeGo Handset UX Bug Triage 20110608]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-18-06.59.html MeeGo Handset UX Bug Triage 20110518]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-11-06.58.html MeeGo Handset UX Bug Triage 20110511]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-04-06.59.html MeeGo Handset UX Bug Triage 20110504]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-20-07.00.html MeeGo Handset UX Bug Triage 20110420]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-13-07.00.html MeeGo Handset UX Bug Triage 20110413]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-06-07.00.html MeeGo Handset UX Bug Triage 20110406]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-30-06.58.html MeeGo Handset UX Bug Triage 20110330]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-23-06.58.html MeeGo Handset UX Bug Triage 20110323]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-16-07.04.html MeeGo Handset UX Bug Triage 20110316]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-09-07.01.html MeeGo Handset UX Bug Triage 20110309]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-02-06.59.html MeeGo Handset UX Bug Triage 20110302]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-23-06.59.html MeeGo Handset UX Bug Triage 20110223]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-16-06.59.html MeeGo Handset UX Bug Triage 20110216]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-09-06.59.html MeeGo Handset UX Bug Triage 20110209]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-02-06.59.html MeeGo Handset UX Bug Triage 20110202]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-26-06.59.html MeeGo Handset UX Bug Triage 20110126]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-19-07.18.html MeeGo Handset UX Bug Triage 20110119]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-12-07.00.html MeeGo Handset UX Bug Triage 20110112]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-05-07.01.html MeeGo Handset UX Bug Triage 20110105]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-22-07.00.html MeeGo Handset UX Bug Triage 20101222]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-15-07.00.html MeeGo Handset UX Bug Triage 20101215]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-08-07.00.html MeeGo Handset UX Bug Triage 20101208]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-01-07.01.html MeeGo Handset UX Bug Triage 20101201]

HandsetBugTriageMinutesArchive

2011-08-03T07:47:33Z

Srikanthyarlagadda:

*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-08-03-06.57.html MeeGo Handset UX Bug Triage 20110803]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-07-20-06.58.html MeeGo Handset UX Bug Triage 20110720]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-07-13-06.57.html MeeGo Handset UX Bug Triage 20110713]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-07-06-07.12.html MeeGo Handset UX Bug Triage 20110706]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-29-06.57.html MeeGo Handset UX Bug Triage 20110629]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-22-06.56.html MeeGo Handset UX Bug Triage 20110622]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-15-06.57.html MeeGo Handset UX Bug Triage 20110615]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-08-06.57.html MeeGo Handset UX Bug Triage 20110608]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-18-06.59.html MeeGo Handset UX Bug Triage 20110518]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-11-06.58.html MeeGo Handset UX Bug Triage 20110511]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-04-06.59.html MeeGo Handset UX Bug Triage 20110504]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-20-07.00.html MeeGo Handset UX Bug Triage 20110420]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-13-07.00.html MeeGo Handset UX Bug Triage 20110413]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-06-07.00.html MeeGo Handset UX Bug Triage 20110406]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-30-06.58.html MeeGo Handset UX Bug Triage 20110330]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-23-06.58.html MeeGo Handset UX Bug Triage 20110323]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-16-07.04.html MeeGo Handset UX Bug Triage 20110316]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-09-07.01.html MeeGo Handset UX Bug Triage 20110309]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-02-06.59.html MeeGo Handset UX Bug Triage 20110302]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-23-06.59.html MeeGo Handset UX Bug Triage 20110223]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-16-06.59.html MeeGo Handset UX Bug Triage 20110216]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-09-06.59.html MeeGo Handset UX Bug Triage 20110209]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-02-06.59.html MeeGo Handset UX Bug Triage 20110202]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-26-06.59.html MeeGo Handset UX Bug Triage 20110126]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-19-07.18.html MeeGo Handset UX Bug Triage 20110119]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-12-07.00.html MeeGo Handset UX Bug Triage 20110112]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-05-07.01.html MeeGo Handset UX Bug Triage 20110105]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-22-07.00.html MeeGo Handset UX Bug Triage 20101222]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-15-07.00.html MeeGo Handset UX Bug Triage 20101215]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-08-07.00.html MeeGo Handset UX Bug Triage 20101208]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-01-07.01.html MeeGo Handset UX Bug Triage 20101201]

ARM/N900/Status

2011-07-27T09:26:01Z

Srikanthyarlagadda: /* N900 Community Edition current status */

== Status ==

Status page is meant to be, a frequently updated, status report of the progress of Meego Community Editions features. The status list itself is divided into two categories, Core and UX. Core level testing is all testing between UI and HW adaptation. UX (User Experience) is testing features from the UI level. The latest weekly image is marked in the list header (eg. wk17 against week 17 CE image).

=== Links ===
* '''Open [https://bugs.meego.com/buglist.cgi?priority=High&keywords=N900CE&query_format=advanced&keywords_type=allwords&field0-0-0=flagtypes.name&bug_severity=critical&bug_status=NEW&bug_status=NEEDINFO&bug_status=INDEFINITION&bug_status=ASSIGNED&bug_status=ACCEPTED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=WAITING&bug_status=REOPENED&type0-0-0=anyexact&value0-0-0=MeeGo_N900CE_Release_Blocker%2B&known_name=N900DE%20TOP%2010 TOP 10 Release Blocker Bugs]'''
* '''Open [https://bugs.meego.com/buglist.cgi?query_format=advanced&field0-0-0=flagtypes.name&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&type0-0-0=anyexact&value0-0-0=MeeGo_N900CE_Release_Blocker%2B DevEdition blocker bugs]''' ([https://bugs.meego.com/buglist.cgi?query_format=advanced&field0-0-0=flagtypes.name&type0-0-0=equals&value0-0-0=MeeGo_N900CE_Release_Blocker%3F proposed bugs])
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature/N900ce Use case reports], [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900ce Data flow reports], [http://qa-reports.meego.com/1.2/Handset/Basic%20Feature/N900ce Basic feature reports]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [https://bugs.meego.com/metrics/?stats=N900_DE_-_Release_Blocker Bug metrics]
* [https://bugs.meego.com/buglist.cgi?keywords=N900CE&query_format=advanced&keywords_type=allwords&resolution=--- All current open N900 specific bugs], based on 'N900' keyword.

=== N900 Community Edition current status ===
* Core = Core level testing. Testing from just under UI, down to hardware adaptation level. (eg. Qt-APIs, ofono)
* UX = User Experience level testing. Testing the product from users point of view. (eg. UI level features)

{| cellpadding="5" cellspacing="0" border="1"
! Feature !! Subfeature !! Core [http://qa-reports.meego.com/1.2/Core/Key%20Feature/N900CE/3137 Stable-CE.2011-07-18.2] !! UX [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE/3217 Stable-CE.2011-07-25.1]
|-
| Cellular voice call
| Make a call directly
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Initiate call from contacts
! style="background:green" | OK
! style="background:green" | OK (1)
|-
|
| Initiate call from history
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Receive Voice call
! style="background:green" | OK (1)
! style="background:green" | OK
|-
|
| Default ringing tone plays
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Volume control
! style="background:green" | OK
! style="background:green" | OK
|-
|
| SIM PIN code entry support
! style="background:green" | OK
! style="background:green" | OK
|-
| SMS
| Send SMS, Direct number input
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Send SMS, select contact
! style="background:green" | OK
! style="background:green" | OK
|-
|
| SMS, Reply to sender (Core: receive)
! style="background:green" | OK
! style="background:green" | OK (2)
|-
| Multimedia
| Audio Playback
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Video Playback
! style="background:red" | NOK (2)
! style="background:red" | NOK (3)
|-
|
| Image Viewing
! style="background:green" | OK
! style="background:green" | OK
|-
| Sensors
| Sensors return data
! style="background:green" | OK
! style="background:grey" | N/A
|-
| Browser over WLAN
| Able to connect to WLAN AP (with security etc.)
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Open a complex modern website (Core: Transfer data)
! style="background:green" | OK
! style="background:grey" | N/A (4)
|-
| Camera (meegocamera)
| Still image capture
! style="background:red" | NOK(3)
! style="background:green" | OK
|-
|
| Support for N900 keys (zoom, capture)
! style="background:green" | OK
! style="background:green" | OK
|-
| System UI
| Smooth and fast basic functionality
! style="background:grey" | N/A
! style="background:red" | NOK
|-
| Homescreen
| Smooth and fast basic functionality
! style="background:grey" | N/A
! style="background:green" | OK
|-
| Settings
| Clear and correct functionality
! style="background:grey" | N/A
! style="background:orange" | OK (3)
|-
| Performance
| Response times
! style="background:yellow" | [http://wiki.meego.com/ARM/N900/QA/Performance MEASURED]
! style="background:yellow" | [http://wiki.meego.com/ARM/N900/QA/Performance MEASURED]
|-
|
| Power management
! style="background:yellow" | [http://wiki.meego.com/ARM/N900/QA/Performance MEASURED]
! style="background:gray" | N/A
|-
|}

* Note number referring to that issue.
==== Core Notes ====
# Rings (very softly, even at max volume) and *sometimes* dialer does open, but does not meet the requirements for a primary device, cannot be heard/felt, in pocket.
# Video playback skewed (bug #19832)
# Top half of image is just noise when flash is used (bug #18061)

==== UX Notes ====
# Call UI is not launched while making outgoing call from Contacts (bug #19815). Dialers People page doesn't show contacts (bug #19586)
# SMS notification does not play an alert sound (bug #14922)
# The contact matching does not work correctly in sms application (bug #19283)
# Video files are not playing properly (bug #19832 #17105 #16434)
# Browser cannot be restarted (bug #17274)
# Systesm UI bugs can be found [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE/3217#test-set-41289]

[[Category:N900]]

File:Rotation time.PNG

2011-07-27T06:08:44Z

Srikanthyarlagadda: uploaded a new version of "File:Rotation time.PNG"

File:Incoming call.PNG

2011-07-27T06:07:59Z

Srikanthyarlagadda: uploaded a new version of "File:Incoming call.PNG"

File:Application startup.PNG

2011-07-27T06:07:12Z

Srikanthyarlagadda: uploaded a new version of "File:Application startup.PNG"

File:Handset shutdown.PNG

2011-07-27T06:06:29Z

Srikanthyarlagadda: uploaded a new version of "File:Handset shutdown.PNG"

File:Handset startup.PNG

2011-07-27T06:05:42Z

Srikanthyarlagadda: uploaded a new version of "File:Handset startup.PNG"

ARM/N900/Status

2011-07-20T08:22:37Z

Srikanthyarlagadda: /* UX Notes */

== Status ==

Status page is meant to be, a frequently updated, status report of the progress of Meego Community Editions features. The status list itself is divided into two categories, Core and UX. Core level testing is all testing between UI and HW adaptation. UX (User Experience) is testing features from the UI level. The latest weekly image is marked in the list header (eg. wk17 against week 17 CE image).

=== Links ===
* '''Open [https://bugs.meego.com/buglist.cgi?priority=High&keywords=N900CE&query_format=advanced&keywords_type=allwords&field0-0-0=flagtypes.name&bug_severity=critical&bug_status=NEW&bug_status=NEEDINFO&bug_status=INDEFINITION&bug_status=ASSIGNED&bug_status=ACCEPTED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=WAITING&bug_status=REOPENED&type0-0-0=anyexact&value0-0-0=MeeGo_N900CE_Release_Blocker%2B&known_name=N900DE%20TOP%2010 TOP 10 Release Blocker Bugs]'''
* '''Open [https://bugs.meego.com/buglist.cgi?query_format=advanced&field0-0-0=flagtypes.name&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&type0-0-0=anyexact&value0-0-0=MeeGo_N900CE_Release_Blocker%2B DevEdition blocker bugs]''' ([https://bugs.meego.com/buglist.cgi?query_format=advanced&field0-0-0=flagtypes.name&type0-0-0=equals&value0-0-0=MeeGo_N900CE_Release_Blocker%3F proposed bugs])
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature/N900ce Use case reports], [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900ce Data flow reports], [http://qa-reports.meego.com/1.2/Handset/Basic%20Feature/N900ce Basic feature reports]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [https://bugs.meego.com/metrics/?stats=N900_DE_-_Release_Blocker Bug metrics]
* [https://bugs.meego.com/buglist.cgi?keywords=N900CE&query_format=advanced&keywords_type=allwords&resolution=--- All current open N900 specific bugs], based on 'N900' keyword.

=== N900 Community Edition current status ===
* Core = Core level testing. Testing from just under UI, down to hardware adaptation level. (eg. Qt-APIs, ofono)
* UX = User Experience level testing. Testing the product from users point of view. (eg. UI level features)

{| cellpadding="5" cellspacing="0" border="1"
! Feature !! Subfeature !! Core [http://qa-reports.meego.com/1.2/Core/Key%20Feature/N900CE/3137 Stable-CE.2011-07-18.2] !! UX [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE/3181 Stable-CE.2011-07-18.2]
|-
| Cellular voice call
| Make a call directly
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Initiate call from contacts
! style="background:green" | OK
! style="background:green" | OK (1)
|-
|
| Initiate call from history
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Receive Voice call
! style="background:green" | OK (1)
! style="background:green" | OK
|-
|
| Default ringing tone plays
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Volume control
! style="background:green" | OK
! style="background:green" | OK
|-
|
| SIM PIN code entry support
! style="background:green" | OK
! style="background:green" | OK
|-
| SMS
| Send SMS, Direct number input
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Send SMS, select contact
! style="background:green" | OK
! style="background:green" | OK
|-
|
| SMS, Reply to sender (Core: receive)
! style="background:green" | OK
! style="background:green" | OK (2)
|-
| Multimedia
| Audio Playback
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Video Playback
! style="background:red" | NOK (2)
! style="background:red" | NOK (3)
|-
|
| Image Viewing
! style="background:green" | OK
! style="background:green" | OK
|-
| Sensors
| Sensors return data
! style="background:green" | OK
! style="background:grey" | N/A
|-
| Browser over WLAN
| Able to connect to WLAN AP (with security etc.)
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Open a complex modern website (Core: Transfer data)
! style="background:green" | OK
! style="background:grey" | N/A (4)
|-
| Camera (meegocamera)
| Still image capture
! style="background:red" | NOK(3)
! style="background:green" | OK
|-
|
| Support for N900 keys (zoom, capture)
! style="background:green" | OK
! style="background:green" | OK
|-
| System UI
| Smooth and fast basic functionality
! style="background:grey" | N/A
! style="background:red" | NOK
|-
| Homescreen
| Smooth and fast basic functionality
! style="background:grey" | N/A
! style="background:green" | OK
|-
| Settings
| Clear and correct functionality
! style="background:grey" | N/A
! style="background:orange" | OK (3)
|-
| Performance
| Response times
! style="background:yellow" | [http://wiki.meego.com/ARM/N900/QA/Performance MEASURED]
! style="background:yellow" | [http://wiki.meego.com/ARM/N900/QA/Performance MEASURED]
|-
|
| Power management
! style="background:yellow" | [http://wiki.meego.com/ARM/N900/QA/Performance MEASURED]
! style="background:gray" | N/A
|-
|}

* Note number referring to that issue.
==== Core Notes ====
# Rings (very softly, even at max volume) and *sometimes* dialer does open, but does not meet the requirements for a primary device, cannot be heard/felt, in pocket.
# Video playback skewed (bug #19832)
# Top half of image is just noise when flash is used (bug #18061)

==== UX Notes ====
# Call UI is not launched while making outgoing call from Contacts (bug #19815). Dialers People page doesn't show contacts (bug #19586)
# SMS notification does not play an alert sound (bug #14922)
# The contact matching does not work correctly in sms application (bug #19283)
# Video files are not playing properly (bug #19832 #17105 #16434)
# Browser cannot be restarted (bug #17274)
# Systesm UI bugs can be found [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE/3181#test-set-40803]

[[Category:N900]]

ARM/N900/Status

2011-07-20T08:16:54Z

Srikanthyarlagadda: /* UX Notes */

== Status ==

Status page is meant to be, a frequently updated, status report of the progress of Meego Community Editions features. The status list itself is divided into two categories, Core and UX. Core level testing is all testing between UI and HW adaptation. UX (User Experience) is testing features from the UI level. The latest weekly image is marked in the list header (eg. wk17 against week 17 CE image).

=== Links ===
* '''Open [https://bugs.meego.com/buglist.cgi?priority=High&keywords=N900CE&query_format=advanced&keywords_type=allwords&field0-0-0=flagtypes.name&bug_severity=critical&bug_status=NEW&bug_status=NEEDINFO&bug_status=INDEFINITION&bug_status=ASSIGNED&bug_status=ACCEPTED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=WAITING&bug_status=REOPENED&type0-0-0=anyexact&value0-0-0=MeeGo_N900CE_Release_Blocker%2B&known_name=N900DE%20TOP%2010 TOP 10 Release Blocker Bugs]'''
* '''Open [https://bugs.meego.com/buglist.cgi?query_format=advanced&field0-0-0=flagtypes.name&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&type0-0-0=anyexact&value0-0-0=MeeGo_N900CE_Release_Blocker%2B DevEdition blocker bugs]''' ([https://bugs.meego.com/buglist.cgi?query_format=advanced&field0-0-0=flagtypes.name&type0-0-0=equals&value0-0-0=MeeGo_N900CE_Release_Blocker%3F proposed bugs])
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature/N900ce Use case reports], [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900ce Data flow reports], [http://qa-reports.meego.com/1.2/Handset/Basic%20Feature/N900ce Basic feature reports]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [https://bugs.meego.com/metrics/?stats=N900_DE_-_Release_Blocker Bug metrics]
* [https://bugs.meego.com/buglist.cgi?keywords=N900CE&query_format=advanced&keywords_type=allwords&resolution=--- All current open N900 specific bugs], based on 'N900' keyword.

=== N900 Community Edition current status ===
* Core = Core level testing. Testing from just under UI, down to hardware adaptation level. (eg. Qt-APIs, ofono)
* UX = User Experience level testing. Testing the product from users point of view. (eg. UI level features)

{| cellpadding="5" cellspacing="0" border="1"
! Feature !! Subfeature !! Core [http://qa-reports.meego.com/1.2/Core/Key%20Feature/N900CE/3137 Stable-CE.2011-07-18.2] !! UX [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE/3181 Stable-CE.2011-07-18.2]
|-
| Cellular voice call
| Make a call directly
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Initiate call from contacts
! style="background:green" | OK
! style="background:green" | OK (1)
|-
|
| Initiate call from history
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Receive Voice call
! style="background:green" | OK (1)
! style="background:green" | OK
|-
|
| Default ringing tone plays
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Volume control
! style="background:green" | OK
! style="background:green" | OK
|-
|
| SIM PIN code entry support
! style="background:green" | OK
! style="background:green" | OK
|-
| SMS
| Send SMS, Direct number input
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Send SMS, select contact
! style="background:green" | OK
! style="background:green" | OK
|-
|
| SMS, Reply to sender (Core: receive)
! style="background:green" | OK
! style="background:green" | OK (2)
|-
| Multimedia
| Audio Playback
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Video Playback
! style="background:red" | NOK (2)
! style="background:red" | NOK (3)
|-
|
| Image Viewing
! style="background:green" | OK
! style="background:green" | OK
|-
| Sensors
| Sensors return data
! style="background:green" | OK
! style="background:grey" | N/A
|-
| Browser over WLAN
| Able to connect to WLAN AP (with security etc.)
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Open a complex modern website (Core: Transfer data)
! style="background:green" | OK
! style="background:grey" | N/A (4)
|-
| Camera (meegocamera)
| Still image capture
! style="background:red" | NOK(3)
! style="background:green" | OK
|-
|
| Support for N900 keys (zoom, capture)
! style="background:green" | OK
! style="background:green" | OK
|-
| System UI
| Smooth and fast basic functionality
! style="background:grey" | N/A
! style="background:red" | NOK
|-
| Homescreen
| Smooth and fast basic functionality
! style="background:grey" | N/A
! style="background:green" | OK
|-
| Settings
| Clear and correct functionality
! style="background:grey" | N/A
! style="background:orange" | OK (3)
|-
| Performance
| Response times
! style="background:yellow" | [http://wiki.meego.com/ARM/N900/QA/Performance MEASURED]
! style="background:yellow" | [http://wiki.meego.com/ARM/N900/QA/Performance MEASURED]
|-
|
| Power management
! style="background:yellow" | [http://wiki.meego.com/ARM/N900/QA/Performance MEASURED]
! style="background:gray" | N/A
|-
|}

* Note number referring to that issue.
==== Core Notes ====
# Rings (very softly, even at max volume) and *sometimes* dialer does open, but does not meet the requirements for a primary device, cannot be heard/felt, in pocket.
# Video playback skewed (bug #19832)
# Top half of image is just noise when flash is used (bug #18061)

==== UX Notes ====
# Call UI is not launched while making outgoing call from Contacts (bug #19815). Dialers People page doesn't show contacts (bug #19586)
# SMS notification does not play an alert sound (bug #14922)
# The contact matching does not work correctly in sms application (bug #19283)
# Video files are not playing properly (bug #13856)
# Browser cannot be restarted (bug #17274)
# Systesm UI bugs can be found [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE/3181#test-set-40803]

[[Category:N900]]

ARM/N900/Status

2011-07-20T08:14:54Z

Srikanthyarlagadda: /* N900 Community Edition current status */

== Status ==

Status page is meant to be, a frequently updated, status report of the progress of Meego Community Editions features. The status list itself is divided into two categories, Core and UX. Core level testing is all testing between UI and HW adaptation. UX (User Experience) is testing features from the UI level. The latest weekly image is marked in the list header (eg. wk17 against week 17 CE image).

=== Links ===
* '''Open [https://bugs.meego.com/buglist.cgi?priority=High&keywords=N900CE&query_format=advanced&keywords_type=allwords&field0-0-0=flagtypes.name&bug_severity=critical&bug_status=NEW&bug_status=NEEDINFO&bug_status=INDEFINITION&bug_status=ASSIGNED&bug_status=ACCEPTED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=WAITING&bug_status=REOPENED&type0-0-0=anyexact&value0-0-0=MeeGo_N900CE_Release_Blocker%2B&known_name=N900DE%20TOP%2010 TOP 10 Release Blocker Bugs]'''
* '''Open [https://bugs.meego.com/buglist.cgi?query_format=advanced&field0-0-0=flagtypes.name&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&type0-0-0=anyexact&value0-0-0=MeeGo_N900CE_Release_Blocker%2B DevEdition blocker bugs]''' ([https://bugs.meego.com/buglist.cgi?query_format=advanced&field0-0-0=flagtypes.name&type0-0-0=equals&value0-0-0=MeeGo_N900CE_Release_Blocker%3F proposed bugs])
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature/N900ce Use case reports], [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900ce Data flow reports], [http://qa-reports.meego.com/1.2/Handset/Basic%20Feature/N900ce Basic feature reports]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [https://bugs.meego.com/metrics/?stats=N900_DE_-_Release_Blocker Bug metrics]
* [https://bugs.meego.com/buglist.cgi?keywords=N900CE&query_format=advanced&keywords_type=allwords&resolution=--- All current open N900 specific bugs], based on 'N900' keyword.

=== N900 Community Edition current status ===
* Core = Core level testing. Testing from just under UI, down to hardware adaptation level. (eg. Qt-APIs, ofono)
* UX = User Experience level testing. Testing the product from users point of view. (eg. UI level features)

{| cellpadding="5" cellspacing="0" border="1"
! Feature !! Subfeature !! Core [http://qa-reports.meego.com/1.2/Core/Key%20Feature/N900CE/3137 Stable-CE.2011-07-18.2] !! UX [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE/3181 Stable-CE.2011-07-18.2]
|-
| Cellular voice call
| Make a call directly
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Initiate call from contacts
! style="background:green" | OK
! style="background:green" | OK (1)
|-
|
| Initiate call from history
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Receive Voice call
! style="background:green" | OK (1)
! style="background:green" | OK
|-
|
| Default ringing tone plays
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Volume control
! style="background:green" | OK
! style="background:green" | OK
|-
|
| SIM PIN code entry support
! style="background:green" | OK
! style="background:green" | OK
|-
| SMS
| Send SMS, Direct number input
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Send SMS, select contact
! style="background:green" | OK
! style="background:green" | OK
|-
|
| SMS, Reply to sender (Core: receive)
! style="background:green" | OK
! style="background:green" | OK (2)
|-
| Multimedia
| Audio Playback
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Video Playback
! style="background:red" | NOK (2)
! style="background:red" | NOK (3)
|-
|
| Image Viewing
! style="background:green" | OK
! style="background:green" | OK
|-
| Sensors
| Sensors return data
! style="background:green" | OK
! style="background:grey" | N/A
|-
| Browser over WLAN
| Able to connect to WLAN AP (with security etc.)
! style="background:green" | OK
! style="background:green" | OK
|-
|
| Open a complex modern website (Core: Transfer data)
! style="background:green" | OK
! style="background:grey" | N/A (4)
|-
| Camera (meegocamera)
| Still image capture
! style="background:red" | NOK(3)
! style="background:green" | OK
|-
|
| Support for N900 keys (zoom, capture)
! style="background:green" | OK
! style="background:green" | OK
|-
| System UI
| Smooth and fast basic functionality
! style="background:grey" | N/A
! style="background:red" | NOK
|-
| Homescreen
| Smooth and fast basic functionality
! style="background:grey" | N/A
! style="background:green" | OK
|-
| Settings
| Clear and correct functionality
! style="background:grey" | N/A
! style="background:orange" | OK (3)
|-
| Performance
| Response times
! style="background:yellow" | [http://wiki.meego.com/ARM/N900/QA/Performance MEASURED]
! style="background:yellow" | [http://wiki.meego.com/ARM/N900/QA/Performance MEASURED]
|-
|
| Power management
! style="background:yellow" | [http://wiki.meego.com/ARM/N900/QA/Performance MEASURED]
! style="background:gray" | N/A
|-
|}

* Note number referring to that issue.
==== Core Notes ====
# Rings (very softly, even at max volume) and *sometimes* dialer does open, but does not meet the requirements for a primary device, cannot be heard/felt, in pocket.
# Video playback skewed (bug #19832)
# Top half of image is just noise when flash is used (bug #18061)

==== UX Notes ====
# Call UI is not launched while making outgoing call from Contacts (bug #19815). Dialers People page doesn't show contacts (bug #19586)
# SMS notification does not play an alert sound (bug #14922)
# The contact matching does not work correctly in sms application (bug #19283)
# Video files are not playing properly (bug #13856)
# Browser cannot be restarted (bug #17274)
# Systesm UI bugs can be found [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE/3124#test-set-40095]

[[Category:N900]]

HandsetBugTriageMinutesArchive

2011-07-20T08:07:57Z

Srikanthyarlagadda:

*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-07-20-06.58.html MeeGo Handset UX Bug Triage 20110720]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-07-13-06.57.html MeeGo Handset UX Bug Triage 20110713]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-07-06-07.12.html MeeGo Handset UX Bug Triage 20110706]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-29-06.57.html MeeGo Handset UX Bug Triage 20110629]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-22-06.56.html MeeGo Handset UX Bug Triage 20110622]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-15-06.57.html MeeGo Handset UX Bug Triage 20110615]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-08-06.57.html MeeGo Handset UX Bug Triage 20110608]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-18-06.59.html MeeGo Handset UX Bug Triage 20110518]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-11-06.58.html MeeGo Handset UX Bug Triage 20110511]
*[http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-04-06.59.html MeeGo Handset UX Bug Triage 20110504]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-20-07.00.html MeeGo Handset UX Bug Triage 20110420]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-13-07.00.html MeeGo Handset UX Bug Triage 20110413]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-06-07.00.html MeeGo Handset UX Bug Triage 20110406]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-30-06.58.html MeeGo Handset UX Bug Triage 20110330]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-23-06.58.html MeeGo Handset UX Bug Triage 20110323]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-16-07.04.html MeeGo Handset UX Bug Triage 20110316]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-09-07.01.html MeeGo Handset UX Bug Triage 20110309]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-02-06.59.html MeeGo Handset UX Bug Triage 20110302]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-23-06.59.html MeeGo Handset UX Bug Triage 20110223]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-16-06.59.html MeeGo Handset UX Bug Triage 20110216]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-09-06.59.html MeeGo Handset UX Bug Triage 20110209]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-02-02-06.59.html MeeGo Handset UX Bug Triage 20110202]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-26-06.59.html MeeGo Handset UX Bug Triage 20110126]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-19-07.18.html MeeGo Handset UX Bug Triage 20110119]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-12-07.00.html MeeGo Handset UX Bug Triage 20110112]
*[http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-01-05-07.01.html MeeGo Handset UX Bug Triage 20110105]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-22-07.00.html MeeGo Handset UX Bug Triage 20101222]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-15-07.00.html MeeGo Handset UX Bug Triage 20101215]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-08-07.00.html MeeGo Handset UX Bug Triage 20101208]
*[http://trac.tspre.org/meetbot/meego-meeting/2010/meego-meeting.2010-12-01-07.01.html MeeGo Handset UX Bug Triage 20101201]

File:Rotation time.PNG

2011-07-19T08:45:42Z

Srikanthyarlagadda: uploaded a new version of "File:Rotation time.PNG"

File:Incoming call.PNG

2011-07-19T08:45:04Z

Srikanthyarlagadda: uploaded a new version of "File:Incoming call.PNG"

File:Application startup.PNG

2011-07-19T08:44:21Z

Srikanthyarlagadda: uploaded a new version of "File:Application startup.PNG"

File:Handset shutdown.PNG

2011-07-19T08:43:39Z

Srikanthyarlagadda: uploaded a new version of "File:Handset shutdown.PNG"

File:Handset startup.PNG

2011-07-19T08:42:45Z

Srikanthyarlagadda: uploaded a new version of "File:Handset startup.PNG"

ARM/N900/QA

2011-07-14T11:41:43Z

Srikanthyarlagadda: /* Questions or Doubts */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
====Daily testing====

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

====Weekly testing====
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

==== Execution Process ====

[[File:Process.PNG]]
 

==== Testing images Community Edition image ====
[http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]

[http://wiki.meego.com/ARM/N900/Install/MMC#Linux Flashing Image to MMC]
* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable image is called as '''Weekly Image'''

==== Execution ====

*Download Test Sets from gitorious [http://meego.gitorious.org/meego-quality-assurance/handset-ux-tests/trees/master Test Sets]
*List all test cases in Excel and create .csv file [http://qa-reports.meego.com/example.csv example csv]
*[http://wiki.meego.com/Quality/HandsetUXQAGettingStarted#Spirit_of_Testing Test cases verdict]
'''Pass''' = If all the steps and Functionality is as per expected
'''Fail''' = If any of the one step or Functionality is not as per expected
'''N/A''' = If the test cases are not executable due to the absence of related package in the image, the test results should be N/A
'''Blocked''' = This is a special value required in the situations where verdict cannot be given due to failure in preconditions
* Upload Test report to with respective Test cycle in QA reporting tool [http://qa-reports.meego.com/ QA Reporting tool]

====Reporting bugs====
*Bug reporting tool: http://bugs.meego.com/
*Bugs are reported in Bugzilla.
*http://wiki.meego.com/Quality/How_To_Report_Bugs
*Logging new bug
[[File:Newbug.PNG]]
 

====Questions or Doubts====
Any question please leave your query in #meego-arm (irc.freenode.net) or you can [mailto:meego-qa@lists.meego.com meego-qa mailing list]

Suggestion about updating of this page [mailto:srikanth.4.yarlagadda@nokia.com Srikanth Yarlagadda]

Questions about execution and reporting please ping me in IRC srikanth_rst

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T11:39:44Z

Srikanthyarlagadda: /* Questions or Doubts */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
====Daily testing====

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

====Weekly testing====
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

==== Execution Process ====

[[File:Process.PNG]]
 

==== Testing images Community Edition image ====
[http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]

[http://wiki.meego.com/ARM/N900/Install/MMC#Linux Flashing Image to MMC]
* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable image is called as '''Weekly Image'''

==== Execution ====

*Download Test Sets from gitorious [http://meego.gitorious.org/meego-quality-assurance/handset-ux-tests/trees/master Test Sets]
*List all test cases in Excel and create .csv file [http://qa-reports.meego.com/example.csv example csv]
*[http://wiki.meego.com/Quality/HandsetUXQAGettingStarted#Spirit_of_Testing Test cases verdict]
'''Pass''' = If all the steps and Functionality is as per expected
'''Fail''' = If any of the one step or Functionality is not as per expected
'''N/A''' = If the test cases are not executable due to the absence of related package in the image, the test results should be N/A
'''Blocked''' = This is a special value required in the situations where verdict cannot be given due to failure in preconditions
* Upload Test report to with respective Test cycle in QA reporting tool [http://qa-reports.meego.com/ QA Reporting tool]

====Reporting bugs====
*Bug reporting tool: http://bugs.meego.com/
*Bugs are reported in Bugzilla.
*http://wiki.meego.com/Quality/How_To_Report_Bugs
*Logging new bug
[[File:Newbug.PNG]]
 

====Questions or Doubts====
Any question please leave your query in #meego-arm (irc.freenode.net) or you can [mailto:meego-qa@lists.meego.com meego-qa mailing list]
Any suggestion about updating of this page [mailto:srikanth.4.yarlagadda@nokia.com Srikanth Yarlagadda]

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T11:37:40Z

Srikanthyarlagadda: /* =Questions or Doubts */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
====Daily testing====

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

====Weekly testing====
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

==== Execution Process ====

[[File:Process.PNG]]
 

==== Testing images Community Edition image ====
[http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]

[http://wiki.meego.com/ARM/N900/Install/MMC#Linux Flashing Image to MMC]
* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable image is called as '''Weekly Image'''

==== Execution ====

*Download Test Sets from gitorious [http://meego.gitorious.org/meego-quality-assurance/handset-ux-tests/trees/master Test Sets]
*List all test cases in Excel and create .csv file [http://qa-reports.meego.com/example.csv example csv]
*[http://wiki.meego.com/Quality/HandsetUXQAGettingStarted#Spirit_of_Testing Test cases verdict]
'''Pass''' = If all the steps and Functionality is as per expected
'''Fail''' = If any of the one step or Functionality is not as per expected
'''N/A''' = If the test cases are not executable due to the absence of related package in the image, the test results should be N/A
'''Blocked''' = This is a special value required in the situations where verdict cannot be given due to failure in preconditions
* Upload Test report to with respective Test cycle in QA reporting tool [http://qa-reports.meego.com/ QA Reporting tool]

====Reporting bugs====
*Bug reporting tool: http://bugs.meego.com/
*Bugs are reported in Bugzilla.
*http://wiki.meego.com/Quality/How_To_Report_Bugs
*Logging new bug
[[File:Newbug.PNG]]
 

====Questions or Doubts====
Any question please leave your query in #meego-arm (irc.freenode.net) or you can mail to meego-qa@lists.meego.com
Any suggestion about updating of this page srikanth.4.yarlagadda@nokia.com

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T11:35:11Z

Srikanthyarlagadda: /* Functional Testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
====Daily testing====

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

====Weekly testing====
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

==== Execution Process ====

[[File:Process.PNG]]
 

==== Testing images Community Edition image ====
[http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]

[http://wiki.meego.com/ARM/N900/Install/MMC#Linux Flashing Image to MMC]
* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable image is called as '''Weekly Image'''

==== Execution ====

*Download Test Sets from gitorious [http://meego.gitorious.org/meego-quality-assurance/handset-ux-tests/trees/master Test Sets]
*List all test cases in Excel and create .csv file [http://qa-reports.meego.com/example.csv example csv]
*[http://wiki.meego.com/Quality/HandsetUXQAGettingStarted#Spirit_of_Testing Test cases verdict]
'''Pass''' = If all the steps and Functionality is as per expected
'''Fail''' = If any of the one step or Functionality is not as per expected
'''N/A''' = If the test cases are not executable due to the absence of related package in the image, the test results should be N/A
'''Blocked''' = This is a special value required in the situations where verdict cannot be given due to failure in preconditions
* Upload Test report to with respective Test cycle in QA reporting tool [http://qa-reports.meego.com/ QA Reporting tool]

====Reporting bugs====
*Bug reporting tool: http://bugs.meego.com/
*Bugs are reported in Bugzilla.
*http://wiki.meego.com/Quality/How_To_Report_Bugs
*Logging new bug
[[File:Newbug.PNG]]
 

====Questions or Doubts===
Any question please leave your query in #meego-arm or you can mail to meego-qa@lists.meego.com
Any suggestion about updating of this page srikanth.4.yarlagadda@nokia.com

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T11:30:14Z

Srikanthyarlagadda: /* Testing images Community Edition image */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
====Daily testing====

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

====Weekly testing====
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

==== Execution Process ====

[[File:Process.PNG]]
 

==== Testing images Community Edition image ====
[http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]

[http://wiki.meego.com/ARM/N900/Install/MMC#Linux Flashing Image to MMC]
* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable image is called as '''Weekly Image'''

==== Execution ====

*Download Test Sets from gitorious [http://meego.gitorious.org/meego-quality-assurance/handset-ux-tests/trees/master Test Sets]
*List all test cases in Excel and create .csv file [http://qa-reports.meego.com/example.csv example csv]
*[http://wiki.meego.com/Quality/HandsetUXQAGettingStarted#Spirit_of_Testing Test cases verdict]
'''Pass''' = If all the steps and Functionality is as per expected
'''Fail''' = If any of the one step or Functionality is not as per expected
'''N/A''' = If the test cases are not executable due to the absence of related package in the image, the test results should be N/A
'''Blocked''' = This is a special value required in the situations where verdict cannot be given due to failure in preconditions
* Upload Test report to with respective Test cycle in QA reporting tool [http://qa-reports.meego.com/ QA Reporting tool]

====Reporting bugs====
*Bug reporting tool: http://bugs.meego.com/
*Bugs are reported in Bugzilla.
*http://wiki.meego.com/Quality/How_To_Report_Bugs
*Logging new bug
[[File:Newbug.PNG]]
 

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T11:29:54Z

Srikanthyarlagadda: /* Testing images Community Edition image */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
====Daily testing====

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

====Weekly testing====
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

==== Execution Process ====

[[File:Process.PNG]]
 

==== Testing images Community Edition image ====
[http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]
[http://wiki.meego.com/ARM/N900/Install/MMC#Linux Flashing Image to MMC]

* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable image is called as '''Weekly Image'''

==== Execution ====

*Download Test Sets from gitorious [http://meego.gitorious.org/meego-quality-assurance/handset-ux-tests/trees/master Test Sets]
*List all test cases in Excel and create .csv file [http://qa-reports.meego.com/example.csv example csv]
*[http://wiki.meego.com/Quality/HandsetUXQAGettingStarted#Spirit_of_Testing Test cases verdict]
'''Pass''' = If all the steps and Functionality is as per expected
'''Fail''' = If any of the one step or Functionality is not as per expected
'''N/A''' = If the test cases are not executable due to the absence of related package in the image, the test results should be N/A
'''Blocked''' = This is a special value required in the situations where verdict cannot be given due to failure in preconditions
* Upload Test report to with respective Test cycle in QA reporting tool [http://qa-reports.meego.com/ QA Reporting tool]

====Reporting bugs====
*Bug reporting tool: http://bugs.meego.com/
*Bugs are reported in Bugzilla.
*http://wiki.meego.com/Quality/How_To_Report_Bugs
*Logging new bug
[[File:Newbug.PNG]]
 

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T11:27:49Z

Srikanthyarlagadda: /* Functional Testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
====Daily testing====

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

====Weekly testing====
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

==== Execution Process ====

[[File:Process.PNG]]
 

==== Testing images Community Edition image ====
[http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]
* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable image is called as '''Weekly Image'''

==== Execution ====

*Download Test Sets from gitorious [http://meego.gitorious.org/meego-quality-assurance/handset-ux-tests/trees/master Test Sets]
*List all test cases in Excel and create .csv file [http://qa-reports.meego.com/example.csv example csv]
*[http://wiki.meego.com/Quality/HandsetUXQAGettingStarted#Spirit_of_Testing Test cases verdict]
'''Pass''' = If all the steps and Functionality is as per expected
'''Fail''' = If any of the one step or Functionality is not as per expected
'''N/A''' = If the test cases are not executable due to the absence of related package in the image, the test results should be N/A
'''Blocked''' = This is a special value required in the situations where verdict cannot be given due to failure in preconditions
* Upload Test report to with respective Test cycle in QA reporting tool [http://qa-reports.meego.com/ QA Reporting tool]

====Reporting bugs====
*Bug reporting tool: http://bugs.meego.com/
*Bugs are reported in Bugzilla.
*http://wiki.meego.com/Quality/How_To_Report_Bugs
*Logging new bug
[[File:Newbug.PNG]]
 

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T11:22:59Z

Srikanthyarlagadda: /* Execution */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
====Daily testing====

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

====Weekly testing====
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

==== Execution Process ====

[[File:Process.PNG]]
 

==== Testing images Community Edition image ====
[http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]
* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable image is called as '''Weekly Image'''

==== Execution ====

*Download Test Sets from gitorious [http://meego.gitorious.org/meego-quality-assurance/handset-ux-tests/trees/master Test Sets]
*List all test cases in Excel and create .csv file [http://qa-reports.meego.com/example.csv example csv]
*[http://wiki.meego.com/Quality/HandsetUXQAGettingStarted#Spirit_of_Testing Test cases verdict]
'''Pass''' = If all the steps and Functionality is as per expected
'''Fail''' = If any of the one step or Functionality is not as per expected
'''N/A''' = If the test cases are not executable due to the absence of related package in the image, the test results should be N/A
'''Blocked''' = This is a special value required in the situations where verdict cannot be given due to failure in preconditions
* Upload Test report to with respective Test cycle in QA reporting tool [http://qa-reports.meego.com/ QA Reporting tool]

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T11:20:06Z

Srikanthyarlagadda: /* Execution */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
====Daily testing====

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

====Weekly testing====
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

==== Execution Process ====

[[File:Process.PNG]]
 

==== Testing images Community Edition image ====
[http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]
* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable image is called as '''Weekly Image'''

==== Execution ====

*Download Test Sets from gitorious [http://meego.gitorious.org/meego-quality-assurance/handset-ux-tests/trees/master Test Sets]
*List all test cases in Excel and create .csv file [http://qa-reports.meego.com/example.csv example csv]
*[http://wiki.meego.com/Quality/HandsetUXQAGettingStarted#Spirit_of_Testing Test cases verdict]
'''Pass''' = If all the steps and Functionality is as per expected
'''Fail''' = If any of the one step or Functionality is not as per expected
'''N/A''' = If the test cases are not executable due to the absence of related package in the image, the test results should be N/A
'''Blocked''' = This is a special value required in the situations where verdict cannot be given due to failure in preconditions

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T11:19:36Z

Srikanthyarlagadda: /* Execution */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
====Daily testing====

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

====Weekly testing====
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

==== Execution Process ====

[[File:Process.PNG]]
 

==== Testing images Community Edition image ====
[http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]
* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable image is called as '''Weekly Image'''

==== Execution ====

*Download Test Sets from gitorious [http://meego.gitorious.org/meego-quality-assurance/handset-ux-tests/trees/master Test Sets]
*List all test cases in Excel and create .csv file [http://qa-reports.meego.com/example.csv example csv]
*[Test cases verdict http://wiki.meego.com/Quality/HandsetUXQAGettingStarted#Spirit_of_Testing]
'''Pass''' = If all the steps and Functionality is as per expected
'''Fail''' = If any of the one step or Functionality is not as per expected
'''N/A''' = If the test cases are not executable due to the absence of related package in the image, the test results should be N/A
'''Blocked''' = This is a special value required in the situations where verdict cannot be given due to failure in preconditions

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T11:12:56Z

Srikanthyarlagadda: /* Execution */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
====Daily testing====

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

====Weekly testing====
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

==== Execution Process ====

[[File:Process.PNG]]
 

==== Testing images Community Edition image ====
[http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]
* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable image is called as '''Weekly Image'''

==== Execution ====

Download Test Sets from gitorious [http://meego.gitorious.org/meego-quality-assurance/handset-ux-tests/trees/master Test Sets]

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T11:12:18Z

Srikanthyarlagadda: /* Functional Testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
====Daily testing====

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

====Weekly testing====
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

==== Execution Process ====

[[File:Process.PNG]]
 

==== Testing images Community Edition image ====
[http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]
* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable image is called as '''Weekly Image'''

==== Execution ====

Download Test Sets from [http://meego.gitorious.org/meego-quality-assurance/handset-ux-tests/trees/master]

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T10:14:39Z

Srikanthyarlagadda: /* Testing images Community Edition image */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

===Weekly testing===
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

=== Execution Process ===

[[File:Process.PNG]]
 

==== Testing images Community Edition image ====
[http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]
* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable image is called as '''Weekly Image'''

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T10:12:52Z

Srikanthyarlagadda: /* Testing images Community Edition image */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

===Weekly testing===
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

=== Execution Process ===

[[File:Process.PNG]]
 

=== Testing images Community Edition image ===
[http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]
* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable image is called as '''Weekly Image'''

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T10:12:29Z

Srikanthyarlagadda: /* Testing images Community Edition image */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

===Weekly testing===
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

=== Execution Process ===

[[File:Process.PNG]]
 

=== Testing images Community Edition image ===
http://repository.maemo.org/meego/n900-de/daily/ Community Edition images
* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable image is called as '''Weekly Image'''

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T10:11:10Z

Srikanthyarlagadda: /* Functional Testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

===Weekly testing===
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

=== Execution Process ===

[[File:Process.PNG]]
 

=== Testing images Community Edition image ===
http://repository.maemo.org/meego/n900-de/daily/ Community Edition images
* Daily '''Testing''' image called a Acceptance image
* Daily '''Stable''' image called a Sanity image
* Every Monday stable images is called as weekly

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T10:07:00Z

Srikanthyarlagadda: /* Execution Process */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

===Weekly testing===
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

*Every Monday stable images is called as weekly Community Edition image [http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]
http://wiki.meego.com/File:Process.PNG

=== Execution Process ===

[[File:Process.PNG]]
 

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T10:05:57Z

Srikanthyarlagadda: /* Functional Testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

===Weekly testing===
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

*Every Monday stable images is called as weekly Community Edition image [http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]
http://wiki.meego.com/File:Process.PNG

==== Execution Process ====

[[File:Process.PNG]]
 

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T10:04:45Z

Srikanthyarlagadda: /* Functional Testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

===Weekly testing===
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

*Every Monday stable images is called as weekly Community Edition image [http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]
http://wiki.meego.com/File:Process.PNG

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

File:Process.PNG

2011-07-14T10:03:34Z

Srikanthyarlagadda:

ARM/N900/QA

2011-07-14T09:04:03Z

Srikanthyarlagadda: /* Functional Testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

*Acceptance (Trunk:Testing)

*Sanity (Trunk:Daily)

===Weekly testing===
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

*Every Monday stable images is called as weekly Community Edition image [http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T09:02:58Z

Srikanthyarlagadda: /* Weekly testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

'''Acceptance (Trunk:Testing)'''

'''Sanity (Trunk:Daily)'''

===Weekly testing===
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

** Every Monday stable images is called as weekly Community Edition image [http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T09:02:42Z

Srikanthyarlagadda: /* Weekly testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

'''Acceptance (Trunk:Testing)'''

'''Sanity (Trunk:Daily)'''

===Weekly testing===
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

*Key Basic Testing

*Data Flow Testing

*Basic Feature Testing

* Every Monday stable images is called as weekly Community Edition image [http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T09:01:54Z

Srikanthyarlagadda: /* Weekly testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

'''Acceptance (Trunk:Testing)'''

'''Sanity (Trunk:Daily)'''

===Weekly testing===
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

'''Key Basic Testing'''

'''Data Flow Testing'''

'''Basic Feature Testing'''

* [http://repository.maemo.org/meego/n900-de/daily/ Community Edition images]Every Monday stable images is called as weekly Community Edition image]

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T08:59:09Z

Srikanthyarlagadda: /* Weekly testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

'''Acceptance (Trunk:Testing)'''

'''Sanity (Trunk:Daily)'''

===Weekly testing===
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

'''Key Basic Testing'''

'''Data Flow Testing'''

'''Basic Feature Testing'''

Every Monday stable images is called as weekly Community Edition image[[http://repository.maemo.org/meego/n900-de/daily/]]

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T08:56:56Z

Srikanthyarlagadda: /* Functional Testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.

'''Acceptance (Trunk:Testing)'''

'''Sanity (Trunk:Daily)'''

===Weekly testing===
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).

'''Key Basic Testing'''

'''Data Flow Testing'''

'''Basic Feature Testing

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T08:56:25Z

Srikanthyarlagadda: /* Functional Testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===

Acceptance/Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.
'''Acceptance (Trunk:Testing)'''
'''Sanity (Trunk:Daily)'''

===Weekly testing===
This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).
'''Key Basic Testing'''
'''Data Flow Testing'''
'''Basic Feature Testing

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T08:14:17Z

Srikanthyarlagadda: /* Weekly testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===
'''Acceptance (Trunk:Testing)'''
(Acceptance test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.)

'''Sanity (Trunk:Daily)'''
(Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in sanity test reflects the most important and basic functionality of the distribution.)

===Weekly testing===
'''Key Basic Testing'''
(This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).)

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T08:13:57Z

Srikanthyarlagadda: /* Weekly testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===
'''Acceptance (Trunk:Testing)'''
(Acceptance test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.)

'''Sanity (Trunk:Daily)'''
(Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in sanity test reflects the most important and basic functionality of the distribution.)

===Weekly testing===
Key Basic Testing
(This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).)

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T08:13:06Z

Srikanthyarlagadda: /* Key basic testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===
'''Acceptance (Trunk:Testing)'''
(Acceptance test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.)

'''Sanity (Trunk:Daily)'''
(Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in sanity test reflects the most important and basic functionality of the distribution.)

===Weekly testing===
= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T08:12:31Z

Srikanthyarlagadda: /* Daily testing */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===
'''Acceptance (Trunk:Testing)'''
(Acceptance test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.)

'''Sanity (Trunk:Daily)'''
(Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in sanity test reflects the most important and basic functionality of the distribution.)

===Weekly testing===
===Key basic testing ===
(This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).)

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-07-14T08:12:04Z

Srikanthyarlagadda: /* = Acceptance (Trunk:Testing) */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Dataflow/N900CE Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Key%20Feature Key Features reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-06-23-06.58.html Meeting minutes 23-06-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== Contact ==
* [http://lists.meego.com/listinfo/meego-qa QA-mailing list]
* join #meego-qa at freenode

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - DONE (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - DONE (sampos)
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - DONE (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]
== Functional Testing ==
===Daily testing===
'''Acceptance (Trunk:Testing)'''
(Acceptance test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in acceptance test reflects the most important and basic functionalities of the distribution. We do the acceptance test against Trunk:Testing repo.)
'''Sanity (Trunk:Daily)'''
(Sanity test is a very brief run-through of the functionality of the entire MeeGo distribution, to assure that the basic health of the distribution and report major regressions at the earliest time. All the checkpoints in sanity test reflects the most important and basic functionality of the distribution.)

===Weekly testing===
===Key basic testing ===
(This is QA weekly testing cycle for weekly images released by distribution team. Our testing focus would be *Key feature testing* to cover the key features and ensure MeeGo Handset UX delivered features are basically covered and major issues are exposed in timely fashion, as well as changes introduced from the last build do not break working features. The test covers UX and applications (MeeGo Audio player ,Video player, Dialer, SMS, fennec browser,VKB, Email, Photo Viewer, Calendar, IM and contacts, etc).)

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* DE testing
* Bug verification
* Wiki clean/update

==== Backlog ====
* Reliability testset planning
* 1.3 Vanilla vs 1.2 DE current measurements
* DE Filesystem benchmark

==== In progress ====
* DE Hourly Automation improvement

==== Done ====
* Qt API test investigation
* Wiki clean/update (waiting for comments)
* Testcase automation list
* Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]