MeeGo wiki - User contributions [en]

Quality/Plans/MeeGo Core Test Plan

2011-05-30T06:31:24Z

Cindy: /* Testability */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [[Release_Engineering/Process|Release Engineering’s Process]].

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [[Quality/TestSuite/MCTS/MCTS_Development_Guideline|MCTS Development Guidelines]].

For these activities MeeGo Core OS QA follows release cycle described in [[Release_Engineering/Release_Timeline|Release Engineering’s release timeline]].

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [[Quality/QA-tools|MeeGo QA Tools Team]].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [[Quality#MeeGo_Handset_Testing|Handset UX Test Plan]].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [[Quality/Test_areas_and_types|test areas]].

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [[Quality/TestSuite/MCTS/MCTS_Development_Guideline|MCTS Development Guidelines]].

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
* http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
* [[Quality/Plans/Testability-checklist|Testability Checklist]]

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [[Quality/TestDesignProcessAndGuideline|MeeGo QA Common Test Design Process and Guidelines]]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [[Quality/QA-tools/Test plan|test definition]] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[[Quality/Test_case_template|Test Case Template]] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[[Quality/TestSuite/MCTS/MCTS_API_analysis|MCTS API analysis]] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Each component will have detailed test plan for the specific component. Test Plans are available in [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details (each component name is link to test plan).

Test plan shall follow structure described below: :

* 1 Introduction
* 2 Test Coverage
** 2.1 Software Coverage
** 2.2 Hardware Coverage
** 2.3 Items not Covered
* 3 Running the tests
** 3.1 Executing tests
** 3.2 Test case arguments
** 3.3 Test cases
** 3.4 Configuration file
* 4 Module Design and Architecture
** 4.1 Architecture diagram
* 5 References
* 6 Change History

See [[Quality/TestSuite/Video_Playback_Driver_Test_Specification|Video playback driver test plan]] for reference.

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

[[File:Coreosqawkcycle.PNG]]

==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionality which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for release content developed under Bugzilla components
* Set of fully automated system level test cases verifying functionality from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, QA will start the full validation testing for it, following test will be involved:

* Component Functional Test – which will follow the Bugzilla component test plan to run all tests.
* System functional Test – which are defined in the individual Bugzilla component test plan.
* System Non-functional Test – which are defined in the individual Bugzilla component test plan.

Purpose of the Full Pass is to measure release maturity in detailed level. In Full Pass testing entire test asset is executed for all the features released and previously marked as verified. Thus visibility and detailed information about release maturity is gained. Target is to have two Full Pass testing cycles during release life cycle. First Pull Pass test round starts at feature complete and last round ends few days prior to release date.

Between these two rounds failing cases and related bugs are followed closely in weekly testing. If there are very good grounds Full Pass testing can be executed more than twice during release cycle life cycle.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [http://qa-reports.meego.com/ MeeGo QA Reports]

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== Risks ==

One main expectation from program management is that QA is able to pinpoint risks concerning Quality Risks. For MeeGo 1.2 QA sees following risks:

{|cellspacing="0" border="1"
!|Risk #
!|Description
!|impact
!|Severity
!|Mitigation actions

|-
|| 1 || Insufficient documentation and/or low quality feature descriptions not meeting that testability requirements || Bad quality test asset not verifying feature in detailed level. || '''High''' || Significant effort put to feature testability in early phase of release life cycle. Review test assets with feature owner and assignees.
|-
|| 2 || Insufficient amount of system level test cases to show functionality of the release || Unclear release quality statement. Community confused and developers hunting errors which do not exist|| '''High''' || Significant effort to system level test case development. Each functionality (main success scenario and it's variants tested at least with one set of parameters.
|-
|| 3 || Changed components are verified in weekly testing but if dependencies are not understood impact to indirectly affected component not verified. || Surprises in release quality || Medium || Developers to address dependencies in detailed level showing significant dependencies
|}

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* [[Quality/TestSuite/MCTS|MCTS docs]]
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://qa-reports.meego.com/
* MeeGo bugzilla: http://bugs.meego.com
* [[Core_OS_Program_Dashboard#1.2_Release|MeeGo 1.2 release planning]]
* [[Quality/Test_areas_and_types|MeeGo test areas and types]]
* [[Release_Engineering/Process|Release Engineering Process]]
* [[Release_Engineering/Release_Timeline|Release Engineering/Release Timeline]]
* [[Quality/TestSuite/MCTS/MCTS_Development_Guideline|MCTS Development Guideline]]
* [[Quality/QA-tools|QA tools team]]
* MeeGo Handset Testing: [[Quality]]
* [[Quality/Plans/Testability-checklist|Testability Checklist]]
* [[Quality/TestDesignProcessAndGuideline|Test Design Process And Guideline]]
* [[Quality/QA-tools/Test plan|Test Packaging]]
* [[Quality/Test case template|Common Test Plan Template]]
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* [[Quality/TestSuite/MCTS/MCTS_API_analysis|MCTS API analysis]]
* [[Quality/TestSuite/Video_Playback_Driver_Test_Specification|Video Playback Driver Test Specification]]

User:Andre/Bug Summary Prefixes

2011-05-26T05:55:19Z

Cindy: /* List */

= Bug summary prefixes in bugs.meego.com =

{|
| style="background: #ffcc00; color: black" | '''Note:''' First and rough list. Does not mean that the current practice necessarily makes sense, just trying to document it. -- [[User:Andre]] 2011-05-17
|}

== List ==

Feel free to edit, but please keep the alphabetical order.

{| border="1" + Items
! Prefix
! Description
|-
| [CONX]
|
|-
| [DE]
| Bug is reproducible only with [http://wiki.meego.com/ARM/N900/DeveloperEdition N900 Development Edition] image
|-
| [FEA]
| Feature request
|-
| GLS -
|
|-
| [Installer]
|
|-
| [IVI]
|
|-
| [MCTS-]
| For bug related to MCTS
|-
| [Meta] / [META] / [METABUG]
| For collecting bug reports from same area together (e.g. power management issues) by using 'Depends on'.
|-
| [MTF]
|
|-
| [MWTS-]
| For bug related to MWTS
|-
| [REG]
|
|-
| [Trunk:Daily] / [trunk: daily] / [Trunk : Daily]
|
|-
| [Trunk:Testing]
|
|}

[[Category:QA]]

ARM/N900/QA

2011-05-24T03:35:52Z

Cindy: /* QA Tasks for MeeGo.com N900 */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Data%20Flow%20For%20Developer%20Edition Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Use%20Case%20Testing%20For%20Developer%20Edition Use case reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - ONGOING (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - NOT STARTED
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - ONGOING (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* Bug verification

==== Backlog ====
* Testcase automation list

==== In progress ====
* DE Hourly Automation improvement
* Wiki clean/update (waiting for comments)

==== Done ====
* * Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.3 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script
* Testability for MeeGo1.3 features.

==== In progress ====
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result
* MeeGo1.2 feature verification

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-05-24T03:33:41Z

Cindy: /* Test execution schedule */

= Meego Developer Edition Quality Assurance =
Quality Assurance for Meego [[ARM/N900|Developers Edition.]] [[ARM/N900|Developer Edition]] QA uses many same components as in core Meego, therefore remember to look [[Quality|Meego core quality page.]] Monitoring the [[ARM/N900|Developer Edition]] maturity is one of the main tasks of [[ARM/N900|Developer Edition]] QA. The current maturity status can be found from the [[ARM/N900/Status|Status page.]]

== Organization ==
* Error Management
** Error Manager: Iekku Huttunen
* QA Tools
* Core Testing
* User Experience Testing (UX)

== Reports ==
* [http://wiki.meego.com/ARM/N900/Status General DE N900 Feature status]
* [http://wiki.meego.com/ARM/N900/QA/Performance Performance results]
* [http://qa-reports.meego.com/1.2/Handset/Data%20Flow%20For%20Developer%20Edition Dataflow reports]
* [http://qa-reports.meego.com/1.2/Handset/Use%20Case%20Testing%20For%20Developer%20Edition Use case reports]

== Meetings ==
QA IRC meeting every Tuesday:
* [http://wiki.meego.com/MeeGo-Meeting_IRC_Schedule MeeGo-Meeting IRC Schedule]

N900 DE Blocker Bug Triage meeting minutes:
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-19-06.59.html Meeting minutes 19-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-12-06.59.html Meeting minutes 12-05-2011]
* [http://irclogs.meego.com/meetbot/meego-meeting/2011/meego-meeting.2011-05-05-07.00.html Meeting minutes 05-05-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

= QA Tools =
Developers Edition uses same QA Tools as in core Meego. For more information please refer to the [[Quality/QA-tools|Quality/QA-tools]].

== QA-Tools Task List ==
List of tasks the QA-Tools are doing for Meego Developer Edition.
If you need something from QA please tell it to us :)
=== OTS setup ===
(Open Testing System)
* [http://194.136.64.78/logger/view/ OTS server] DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - DONE
* Minimize automatic installation time to 10mins - DONE
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - ONGOING (timakima)
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)
* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)
* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
We are using mainly MCTS test assets, so please refer to the [[Quality/TestSuite/MCTS|MCTS page.]] You can find list of open bugs also from there.
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

[[File:Text9867-0-4-8.png|350px|thumb|right|Automated testing and crash reporting]]

Core dump processing and backtraces from crashing ARM processes (click the image on right).

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (https://bugs.meego.com/show_bug.cgi?id=17134) - ONGOING (alkuznet)
** Oopslog (and lifelog) functionality - NOT STARTED

* Set up back-end server for core processing - ONGOING (rikhalon, sampos, timakima)

* Image building and test run triggered from cron - trigger-testrun.sh -u <URL> - DONE (timakima)

* Implement OTS Conductor plugin to fetch debug package list (before test run) and upload rich-core dumps to post-processing (after testrun )- DONE (rikhalon)

* testrunner-lite sets a UID for each test case on DUT kernel core pattern. So that a coredump can be matched with a test case - DONE (rikhalon)

* Debug image is built (simultaneuosly) on the core proscessing backend server - build-autotest-image.sh -f fs -d -p debug-packet-list -s 8000 -u <URL>, and saved as target for core prosessing - DONE (sampos)

* After each test case, cores matched against the UID are fetched from the DUT by testrunner-lite. - NOT STARTED
** testrunner-lite needs to write unique identifier to results.xml e.g. md5 hash from rich-core.
[[File:backend.png|350px|thumb|right|Core Processing Backend]]
<crashes>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
<crash-id>1234567890ABCDEF</crash-id>
</crashes>

* The core processing backend extracts the rich core and looks for proper target (if not available waits...) - PofC DONE (sampos)

* The core processing backend chroots to the target with debug symbols and executes statically linked cross gdb for backtrace - DONE (sampos)

* Upload processed crash data using [[Quality/QA-tools/CrashReports/API|Crash Reports API]] - ONGOING (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]

= Core Testing =
== QA Tasks For Developer Edition ==
QA tasks for the [[ARM/N900|Developer Edition]] differ from the usual N900 approach in that there are less features to be tested. This is described in more detail in [[ARM/N900#Target|Developer Edition Targets.]] There are currently 2 test sets for the [[ARM/N900|Developer Edition]], these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time, a CPU/Memory benchmark, files system, and power measurement, but this will be explained later.

Suggestions are welcome.

=== Testing schedule ===
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Feature
| DE Weekly
|-
| Tuesday
| Sanity
| DE Trunk testing
|-
| Wednesday
| Sanity
| DE Trunk testing
|-
| Thursday
| Sanity
| DE Trunk testing
|-
| Friday
| Sanity
| DE Trunk Testing
|-
|}

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* Bug verification

==== Backlog ====
* Testcase automation list

==== In progress ====
* DE Hourly Automation improvement
* Wiki clean/update (waiting for comments)

==== Done ====
* * Plan Feature testset (.xml updated)
* Create weekly schedule for MRT (currently in draft form)
* Week 15 DE Sanity Testing
* Maturity statement of Alpha RELEASE (result in QA-report)
* Alpha RELEASE testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.3(Tu Qingqing)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.2 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script

==== In progress ====
* MeeGo1.2 feature verification
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-04-28T02:25:20Z

Cindy: /* META team backlog for MeeGo1.2 N900 */

= General info =

== MeeGo.com IRC meetings ==

QA IRC meeting every Tuesday:
* [[MeeGo-Meeting_IRC_Schedule|MeeGo-Meeting IRC Schedule]]

== QA Organization ==

== Error Management ==
* Error Manager Iekku Huttunen

N900 DE Blocker Bug Triage meeting minutes:
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-21-07.01.html Meeting minutes 21-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

= Tools =

== OTS setup and automated hourly testing ==

=== OTS setup ===
* [http://194.136.64.78/logger/view/ OTS server] - Ville Ilvonen/Riku Halonen, DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
* Minimize automatic installation time to 10mins - Timo Makimattila, ONGOING
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - NOTSTARTED
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)

* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)

* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

Core dump processing and backtraces from crashing ARM processes.

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (Something goes wrong, when reducer processes coredump.) - DONE (sampos)
** Oopslog (and lifelog) functionality - NOT STARTED

* Back-end server
** Set up back-end server for core processing - ONGOING (rikhalon, sampos)

* Script to produce backtraces locally
** Extend rich-core-extract to proceduce backtrace from rich-core file - NOT STARTED
** Write howto guide in wiki - NOT STARTED

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

= UX testing =
== Test execution schedule ==
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== Performance Testing ==
Performance testing results done from UI can be found [[ARM/N900/QA/Performance|here]]

= Core Testing =
== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* Bug verification

==== Backlog ====
* Maturity statement of Alpha RELEASE
* Wiki clean/update
* Plan Feature testset

==== In progress ====
* Alpha RELEASE testing
* Create weekly schedule for MRT (currently in draft form)
* DE Hourly Automation improvement

==== Done ====
* Week 15 DE Sanity Testing

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.2(Tu Qingqing)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
|Monday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Monday, Tuesday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Wednesday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Wednesday, Thursday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.2 N900 daily validation

==== Backlog ====
* Optimize acceptance automation script

==== In progress ====
* MeeGo1.2 feature verification
* Automation testing for trunk:test and trunk image

==== Done ====
* Publish automation test result

= Usefull links =

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-04-20T01:30:23Z

Cindy: /* Test execution schedule */

= QA TODOs (in priority order) =

== MeeGo.com IRC meetings ==

There are no special QA meeting, other DE team meetings can be found from here:
* [[MeeGo-Meeting_IRC_Schedule|MeeGo-Meeting IRC Schedule]]

== OTS setup and automated hourly testing ==

=== OTS setup ===
* [http://194.136.64.78/logger/view/ OTS server] - Ville Ilvonen/Riku Halonen, DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
* Minimize automatic installation time to 10mins - Timo Makimattila, ONGOING
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - NOTSTARTED
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)

* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)

* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

Core dump processing and backtraces from crashing ARM processes.

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (Something goes wrong, when reducer processes coredump.) - DONE (sampos)
** Oopslog (and lifelog) functionality - NOT STARTED

* Back-end server
** Set up back-end server for core processing - ONGOING (rikhalon, sampos)

* Script to produce backtraces locally
** Extend rich-core-extract to proceduce backtrace from rich-core file - NOT STARTED
** Write howto guide in wiki - NOT STARTED

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

=== Test execution schedule ===
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* Bug verification

==== Backlog ====
* Maturity statement of Alpha RELEASE
* Wiki clean/update
* Plan Feature testset

==== In progress ====
* Alpha RELEASE testing
* Create weekly schedule for MRT (currently in draft form)
* DE Hourly Automation improvement

==== Done ====
* Week 15 DE Sanity Testing

== QA Organization ==

== Error Management ==
* Error Manager Iekku Huttunen

N900 DE Blocker Bug Triage meeting minutes:
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.2(Tu Qingqing)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
|Monday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Monday, Tuesday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P2
|-
| Wednesday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Wednesday, Thursday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Thursday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)& test for changes
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.2 N900 daily validation

==== Backlog ====
* Automation testing for trunk:test and trunk image

==== In progress ====
* MeeGo1.2 feature verification
* Publish automation test result

==== Done ====

== Usefull links ==

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-04-20T01:20:15Z

Cindy: /* Test execution schedule */

= QA TODOs (in priority order) =

== MeeGo.com IRC meetings ==

There are no special QA meeting, other DE team meetings can be found from here:
* [[MeeGo-Meeting_IRC_Schedule|MeeGo-Meeting IRC Schedule]]

== OTS setup and automated hourly testing ==

=== OTS setup ===
* [http://194.136.64.78/logger/view/ OTS server] - Ville Ilvonen/Riku Halonen, DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
* Minimize automatic installation time to 10mins - Timo Makimattila, ONGOING
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - NOTSTARTED
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)

* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)

* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

Core dump processing and backtraces from crashing ARM processes.

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (Something goes wrong, when reducer processes coredump.) - DONE (sampos)
** Oopslog (and lifelog) functionality - NOT STARTED

* Back-end server
** Set up back-end server for core processing - ONGOING (rikhalon, sampos)

* Script to produce backtraces locally
** Extend rich-core-extract to proceduce backtrace from rich-core file - NOT STARTED
** Write howto guide in wiki - NOT STARTED

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

=== Test execution schedule ===
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* Bug verification

==== Backlog ====
* Maturity statement of Alpha RELEASE
* Wiki clean/update
* Plan Feature testset

==== In progress ====
* Alpha RELEASE testing
* Create weekly schedule for MRT (currently in draft form)
* DE Hourly Automation improvement

==== Done ====
* Week 15 DE Sanity Testing

== QA Organization ==

== Error Management ==
* Error Manager Iekku Huttunen

N900 DE Blocker Bug Triage meeting minutes:
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.2(Tu Qingqing)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)
| MeeGo.com trunk testing
| P1
|-
|Monday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Monday, Tuesday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
| Tuesday
| Acceptance test (OK)
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Acceptance test (OK)
| MeeGo.com trunk testing
| P2
|-
| Wednesday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Wednesday, Thursday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Thursday
| Acceptance test (OK)
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.2 N900 daily validation

==== Backlog ====
* Automation testing for trunk:test and trunk image

==== In progress ====
* MeeGo1.2 feature verification
* Publish automation test result

==== Done ====

== Usefull links ==

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-04-20T01:09:53Z

Cindy: /* Test execution schedule */

= QA TODOs (in priority order) =

== MeeGo.com IRC meetings ==

There are no special QA meeting, other DE team meetings can be found from here:
* [[MeeGo-Meeting_IRC_Schedule|MeeGo-Meeting IRC Schedule]]

== OTS setup and automated hourly testing ==

=== OTS setup ===
* [http://194.136.64.78/logger/view/ OTS server] - Ville Ilvonen/Riku Halonen, DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
* Minimize automatic installation time to 10mins - Timo Makimattila, ONGOING
* OTS worker for UX tests - NOTSTARTED
* OTS Worker(s) for DE tests - NOTSTARTED
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)

* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)

* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

Core dump processing and backtraces from crashing ARM processes.

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (Something goes wrong, when reducer processes coredump.) - DONE (sampos)
** Oopslog (and lifelog) functionality - NOT STARTED

* Back-end server
** Set up back-end server for core processing - ONGOING (rikhalon, sampos)

* Script to produce backtraces locally
** Extend rich-core-extract to proceduce backtrace from rich-core file - NOT STARTED
** Write howto guide in wiki - NOT STARTED

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

=== Test execution schedule ===
* UX testing schedule: DE / Meego.com testing
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Dataflow
| DE Weekly
| P1
|-
| Monday
| Use cases
| DE Weekly
| P2
|-
|Monday
| Key feature
| DE Weekly
| P3
|-
| Monday
| Performance
| DE Weekly
| P5
|-
| Monday
| Reliability / Iterative
| DE Weekly
| P6
|-
| Monday
| Dataflow
| DE Trunk testing
| P4
|-
| Tuesday
| Dataflow
| DE Trunk testing
| P1
|-
| Tuesday
| Dataflow
| DE Trunk
| P2
|-
| Tuesday
| Acceptance
| Meego Trunk testing
| P3
|-
| Tuesday
| Key feature
| DE Tablet (N900)
| P4
|-
| Wednesday
| Dataflow
| DE Trunk testing
| P1
|-
| Wednesday
| Dataflow
| DE Trunk
| P2
|-
| Wednesday
| Key feature
| Meego.com weekly
| P3
|-
| Wednesday
| Sanity
| Meego.com weekly
| P4
|-
| Thursday
| Dataflow
| DE Trunk testing
| P1
|-
| Thursday
| Dataflow
| DE Trunk
| P2
|-
| Thursday
| Acceptance
| Meego.com Trunk testing
| P3
|-
| Friday
| Dataflow
| DE Trunk Testing
| P1
|-
| Friday
| Dataflow
| DE Trunk
| P2
|-
| Friday
| Acceptance
| Meego.com Trunk testing
| P3
|}

== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* Bug verification

==== Backlog ====
* Maturity statement of Alpha RELEASE
* Wiki clean/update
* Plan Feature testset

==== In progress ====
* Alpha RELEASE testing
* Create weekly schedule for MRT (currently in draft form)
* DE Hourly Automation improvement

==== Done ====
* Week 15 DE Sanity Testing

== QA Organization ==

== Error Management ==
* Error Manager Iekku Huttunen

N900 DE Blocker Bug Triage meeting minutes:
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.2(Tu Qingqing)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance test (OK)
| MeeGo.com trunk testing
| P1
|-
| Monday
| Basic feature test (OK)
| MeeGo.com pre-weekly
| P2
|-
|Monday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Tuesday
| Acceptance test (OK)
| MeeGo.com trunk testing
| P1
|-
| Tuesday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Wednesday
| Dataflow (OK)
| MeeGo.com weekly
| P1
|-
| Wednesday
| Acceptance test (OK)
| MeeGo.com trunk testing
| P2
|-
| Wednesday
| Sanity test (OK)
| MeeGo.com trunk
| P3
|-
| Thursday
| Acceptance test (OK)
| MeeGo.com trunk testing
| P1
|-
| Thursday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|-
| Friday
| Acceptance test (OK)
| MeeGo.com trunk testing
| P1
|-
| Friday
| Sanity test (OK)
| MeeGo.com trunk
| P2
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.2 N900 daily validation

==== Backlog ====
* Automation testing for trunk:test and trunk image

==== In progress ====
* MeeGo1.2 feature verification
* Publish automation test result

==== Done ====

== Usefull links ==

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-04-15T05:09:11Z

Cindy: /* MeeGo.com N900 QA Tasks */

= QA TODOs (in priority order) =

== MeeGo.com IRC meetings ==

There are no special QA meeting, other DE team meetings can be found from here:
* [[MeeGo-Meeting_IRC_Schedule|MeeGo-Meeting IRC Schedule]]

== OTS setup and automated hourly testing ==

=== OTS setup ===
* [http://194.136.64.78/logger/view/ OTS server] - Ville Ilvonen/Riku Halonen, DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
* Minimize automatic installation time to 10mins - Timo Makimattila, ONGOING
* OTS worker for UX tests - NOTSTARTED
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)

* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)

* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

Core dump processing and backtraces from crashing ARM processes.

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (Something goes wrong, when reducer processes coredump.) - DONE (sampos)
** Oopslog (and lifelog) functionality - NOT STARTED

* Back-end server
** Set up back-end server for core processing - ONGOING (rikhalon, sampos)

* Script to produce backtraces locally
** Extend rich-core-extract to proceduce backtrace from rich-core file - NOT STARTED
** Write howto guide in wiki - NOT STARTED

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

== Test Execution Schedule ==
* Core (Teivas)
* Handset UX weekly testing schedule (Rajala)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Key feature (OK)
| Preview
|-
| Monday
| Acceptance (OK)
| Testing trunk
|-
|Tuesday
|Acceptance (Ok)
|Testing trunk
|-
|Tuesday
|Sanity (Ok)
|Daily trunk
|-
|Tuesday
|DE Dataflow (Ok)
|Preview
|-
|Tuesday
|DE use cases (Ok)
|Preview
|-
|Wednesday
|Acceptance (Ok)
|Testing trunk
|-
|Wednesday
|Key feature (Ok)
|Weekly
|-
|Thursday
|DE Dataflow (Ok)
|Weekly
|-
|Thursday
|DE use cases (Ok)
|Weekly
|-
|Thursday
|Acceptance (Ok)
|Testing trunk
|-
|Thursday
|Sanity Ok
|Daily trunk
|-
|Thursday
|DE Reliability (Ongoing)
|Weekly
|-
|Thursday
|DE Performance (Ongoing)
|Weekly
|-
|Friday
|Acceptance (Ok)
|Testing trunk
|-
|Friday
|Sanity (Ok)
|Daily trunk
|}

== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* Bug verification

==== Backlog ====
* Wiki clean/update
* Plan Feature testset

==== In progress ====
* Create weekly schedule for MRT (currently in draft form)
* DE Hourly Automation improvement
* Week 15 DE Sanity Testing

==== Done ====

== QA Organization ==

== Error Management ==
* Error Manager Iekku Huttunen

N900 DE Blocker Bug Triage meeting minutes:
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== QA Tasks for MeeGo.com N900 ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule for MeeGo1.2(Tu Qingqing)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Monday
| Basic feature test(OK)
| Preview
| P2
|-
| Monday
| Sanity (OK)
| Trunk
| P3
|-
| Tuesday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Tuesday
| Sanity (OK)
| Trunk
| P2
|-
| Wednesday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Wednesday
| Data Flow(OK)
| Weekly
| P2
|-
| Wednesday
| Sanity (OK)
| Trunk
| P3
|-
| Thursday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Thursday
| Sanity (OK)
| Trunk
| P2
|-
| Friday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Friday
| Sanity (OK)
| Trunk
| P2
|-
|}

=== META team backlog for MeeGo1.2 N900 ===

==== Continuous tasks ====
* Bug verification
* MeeGo1.2 N900 daily validation

==== Backlog ====
* Automation testing for trunk:test and trunk image

==== In progress ====
* MeeGo1.2 feature verification
* Publish automation test result

==== Done ====

== Usefull links ==

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-04-15T04:53:02Z

Cindy: /* Test execution schedule */

= QA TODOs (in priority order) =

== MeeGo.com IRC meetings ==

There are no special QA meeting, other DE team meetings can be found from here:
* [[MeeGo-Meeting_IRC_Schedule|MeeGo-Meeting IRC Schedule]]

== OTS setup and automated hourly testing ==

=== OTS setup ===
* [http://194.136.64.78/logger/view/ OTS server] - Ville Ilvonen/Riku Halonen, DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
* Minimize automatic installation time to 10mins - Timo Makimattila, ONGOING
* OTS worker for UX tests - NOTSTARTED
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)

* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)

* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crash analysis support for ARM core dumps ==

Core dump processing and backtraces from crashing ARM processes.

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (Something goes wrong, when reducer processes coredump.) - DONE (sampos)
** Oopslog (and lifelog) functionality - NOT STARTED

* Back-end server
** Set up back-end server for core processing - ONGOING (rikhalon, sampos)

* Script to produce backtraces locally
** Extend rich-core-extract to proceduce backtrace from rich-core file - NOT STARTED
** Write howto guide in wiki - NOT STARTED

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

== Test Execution Schedule ==
* Core (Teivas)
* Handset UX weekly testing schedule (Rajala)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Key feature (OK)
| Preview
|-
| Monday
| Acceptance (OK)
| Testing trunk
|-
|Tuesday
|Acceptance (Ok)
|Testing trunk
|-
|Tuesday
|Sanity (Ok)
|Daily trunk
|-
|Tuesday
|DE Dataflow (Ok)
|Preview
|-
|Tuesday
|DE use cases (Ok)
|Preview
|-
|Wednesday
|Acceptance (Ok)
|Testing trunk
|-
|Wednesday
|Key feature (Ok)
|Weekly
|-
|Thursday
|DE Dataflow (Ok)
|Weekly
|-
|Thursday
|DE use cases (Ok)
|Weekly
|-
|Thursday
|Acceptance (Ok)
|Testing trunk
|-
|Thursday
|Sanity Ok
|Daily trunk
|-
|Thursday
|DE Reliability (Ongoing)
|Weekly
|-
|Thursday
|DE Performance (Ongoing)
|Weekly
|-
|Friday
|Acceptance (Ok)
|Testing trunk
|-
|Friday
|Sanity (Ok)
|Daily trunk
|}

== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

==== Continuous tasks ====
* Bug verification

==== Backlog ====
* Wiki clean/update
* Plan Feature testset

==== In progress ====
* Create weekly schedule for MRT (currently in draft form)
* DE Hourly Automation improvement
* Week 15 DE Sanity Testing

==== Done ====

== QA Organization ==

== Error Management ==
* Error Manager Iekku Huttunen

N900 DE Blocker Bug Triage meeting minutes:
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-14-07.00.html Meeting minutes 14-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-04-07-07.02.html Meeting minutes 07-04-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-31-07.00.html Meeting minutes 31-03-2011]
* [http://trac.tspre.org/meetbot/meego-meeting/2011/meego-meeting.2011-03-24-06.59.html Meeting minutes 24-03-2011]

== MeeGo.com N900 QA Tasks ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule (Tu Qingqing)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Monday
| Basic feature test(OK)
| Preview
| P2
|-
| Monday
| Sanity (OK)
| Trunk
| P3
|-
| Tuesday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Tuesday
| Sanity (OK)
| Trunk
| P2
|-
| Wednesday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Wednesday
| Data Flow(OK)
| Weekly
| P2
|-
| Wednesday
| Sanity (OK)
| Trunk
| P3
|-
| Thursday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Thursday
| Sanity (OK)
| Trunk
| P2
|-
| Friday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Friday
| Sanity (OK)
| Trunk
| P2
|-
|}

== Usefull links ==

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-04-08T05:01:34Z

Cindy: /* QA TODOs (in priority order) */

= QA TODOs (in priority order) =

== MeeGo.com IRC meetings ==

There are no special QA meeting, other DE team meetings can be found from here:
* [[MeeGo-Meeting_IRC_Schedule|MeeGo-Meeting IRC Schedule]]

== OTS setup and automated hourly testing ==

=== OTS setup ===
* [http://194.136.64.78/logger/view/ OTS server] - Ville Ilvonen/Riku Halonen, DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
* Minimize automatic installation time to 10mins - Timo Makimattila, ONGOING
* OTS worker for UX tests - NOTSTARTED
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)

* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)

* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crashdb support for ARM core dumps ==

Core dump processing and backtraces from crashing ARM processes.

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (Something goes wrong, when reducer processes coredump.) - DONE (sampos)
** Oopslog (and lifelog) functionality - NOT STARTED

* Back-end server
** Set up back-end server for core processing - ONGOING (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

== Test Execution Schedule ==
* Core (Teivas)
* Handset UX weekly testing schedule (Rajala)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Key feature (OK)
| Preview
|-
| Monday
| Acceptance (OK)
| Testing trunk
|-
|Tuesday
|Acceptance (Ok)
|Testing trunk
|-
|Tuesday
|Sanity (Ok)
|Daily trunk
|-
|Tuesday
|DE Dataflow (Ok)
|Preview
|-
|Tuesday
|DE use cases (Ok)
|Preview
|-
|Wednesday
|Acceptance (Ok)
|Testing trunk
|-
|Wednesday
|Key feature (Ok)
|Weekly
|-
|Thursday
|DE Dataflow (Ok)
|Weekly
|-
|Thursday
|DE use cases (Ok)
|Weekly
|-
|Thursday
|Acceptance (Ok)
|Testing trunk
|-
|Thursday
|Sanity Ok
|Daily trunk
|-
|Thursday
|DE Reliability (Ongoing)
|Weekly
|-
|Thursday
|DE Performance (Ongoing)
|Weekly
|-
|Friday
|Acceptance (Ok)
|Testing trunk
|-
|Friday
|Sanity (Ok)
|Daily trunk
|}

== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

* Create weekly schedule for MRT (currently in draft form)
* Wiki clean/update
* Hourly Automation improvement
* Plan Feature testset
* Bug verification

== QA Organization ==

== Error Management ==
* Error Manager Iekku Huttunen

== MeeGo.com N900 QA Tasks ==

=== Test execution schedule ===
* MeeGo.com N900 Core weekly test schedule (Tu Qingqing)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Monday
| Basic feature test
| Preview
| P2
|-
| Monday
| Sanity (OK)
| Trunk
| P3
|-
| Tuesday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Tuesday
| Sanity (OK)
| Trunk
| P2
|-
| Wednesday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Wednesday
| Data Flow
| Weekly
| P2
|-
| Wednesday
| Sanity (OK)
| Trunk
| P3
|-
| Thursday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Thursday
| Sanity (OK)
| Trunk
| P2
|-
| Friday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Friday
| Sanity (OK)
| Trunk
| P2
|-
|}

== Usefull links ==

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-04-08T04:56:07Z

Cindy: /* Test Execution Schedule */

= QA TODOs (in priority order) =

== MeeGo.com IRC meetings ==

There are no special QA meeting, other DE team meetings can be found from here:
* [[MeeGo-Meeting_IRC_Schedule|MeeGo-Meeting IRC Schedule]]

== OTS setup and automated hourly testing ==

=== OTS setup ===
* [http://194.136.64.78/logger/view/ OTS server] - Ville Ilvonen/Riku Halonen, DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
* Minimize automatic installation time to 10mins - Timo Makimattila, ONGOING
* OTS worker for UX tests - NOTSTARTED
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)

* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)

* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crashdb support for ARM core dumps ==

Core dump processing and backtraces from crashing ARM processes.

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (Something goes wrong, when reducer processes coredump.) - DONE (sampos)
** Oopslog (and lifelog) functionality - NOT STARTED

* Back-end server
** Set up back-end server for core processing - ONGOING (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

== Test Execution Schedule ==
* Core (Teivas)
* MeeGo.com N900 Core weekly test schedule (Tu Qingqing)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Monday
| Basic feature test
| Preview
| P2
|-
| Monday
| Sanity (OK)
| Trunk
| P3
|-
| Tuesday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Tuesday
| Sanity (OK)
| Trunk
| P2
|-
| Wednesday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Wednesday
| Data Flow
| Weekly
| P2
|-
| Wednesday
| Sanity (OK)
| Trunk
| P3
|-
| Thursday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Thursday
| Sanity (OK)
| Trunk
| P2
|-
| Friday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Friday
| Sanity (OK)
| Trunk
| P2
|-
|}
* Handset UX weekly testing schedule (Rajala)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Key feature (OK)
| Preview
|-
| Monday
| Acceptance (OK)
| Testing trunk
|-
|Tuesday
|Acceptance (Ok)
|Testing trunk
|-
|Tuesday
|Sanity (Ok)
|Daily trunk
|-
|Tuesday
|DE Dataflow (Ok)
|Preview
|-
|Tuesday
|DE use cases (Ok)
|Preview
|-
|Wednesday
|Acceptance (Ok)
|Testing trunk
|-
|Wednesday
|Key feature (Ok)
|Weekly
|-
|Thursday
|DE Dataflow (Ok)
|Weekly
|-
|Thursday
|DE use cases (Ok)
|Weekly
|-
|Thursday
|Acceptance (Ok)
|Testing trunk
|-
|Thursday
|Sanity Ok
|Daily trunk
|-
|Thursday
|DE Reliability (Ongoing)
|Weekly
|-
|Thursday
|DE Performance (Ongoing)
|Weekly
|-
|Friday
|Acceptance (Ok)
|Testing trunk
|-
|Friday
|Sanity (Ok)
|Daily trunk
|}

== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

* Create weekly schedule for MRT (currently in draft form)
* Wiki clean/update
* Hourly Automation improvement
* Plan Feature testset
* Bug verification

== QA Organization ==

== Error Management ==
* Error Manager Iekku Huttunen

== Usefull links ==

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-04-07T09:00:53Z

Cindy: /* Test Execution Schedule */

= QA TODOs (in priority order) =

== MeeGo.com IRC meetings ==

There are no special QA meeting, other DE team meetings can be found from here:
* [[MeeGo-Meeting_IRC_Schedule|MeeGo-Meeting IRC Schedule]]

== OTS setup and automated hourly testing ==

=== OTS setup ===
* [http://194.136.64.78/logger/view/ OTS server] - Ville Ilvonen/Riku Halonen, DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
* Minimize automatic installation time to 10mins - Timo Makimattila, ONGOING
* OTS worker for UX tests - NOTSTARTED
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)

* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)

* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crashdb support for ARM core dumps ==

Core dump processing and backtraces from crashing ARM processes.

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (Something goes wrong, when reducer processes coredump.) - DONE (sampos)
** Oopslog (and lifelog) functionality - NOT STARTED

* Back-end server
** Set up back-end server for core processing - ONGOING (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

== Test Execution Schedule ==
* MeeGo.com N900 Core weekly test schedule (Tu Qingqing)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Monday
| Basic feature test
| Preview
| P2
|-
| Monday
| Sanity (OK)
| Trunk
| P3
|-
| Tuesday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Tuesday
| Sanity (OK)
| Trunk
| P2
|-
| Wednesday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Wednesday
| Data Flow
| Weekly
| P2
|-
| Wednesday
| Sanity (OK)
| Trunk
| P3
|-
| Thursday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Thursday
| Sanity (OK)
| Trunk
| P2
|-
| Friday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Friday
| Sanity (OK)
| Trunk
| P2
|-
|}
* Handset UX weekly testing schedule (Rajala)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Key feature (OK)
| Preview
|-
| Monday
| Acceptance (OK)
| Testing trunk
|-
|Tuesday
|Acceptance (Ok)
|Testing trunk
|-
|Tuesday
|Sanity (Ok)
|Daily trunk
|-
|Tuesday
|DE Dataflow (Ok)
|Preview
|-
|Tuesday
|DE use cases (Ok)
|Preview
|-
|Wednesday
|Acceptance (Ok)
|Testing trunk
|-
|Wednesday
|Key feature (Ok)
|Weekly
|-
|Thursday
|DE Dataflow (Ok)
|Weekly
|-
|Thursday
|DE use cases (Ok)
|Weekly
|-
|Thursday
|Acceptance (Ok)
|Testing trunk
|-
|Thursday
|Sanity Ok
|Daily trunk
|-
|Thursday
|DE Reliability (Ongoing)
|Weekly
|-
|Thursday
|DE Performance (Ongoing)
|Weekly
|-
|Friday
|Acceptance (Ok)
|Testing trunk
|-
|Friday
|Sanity (Ok)
|Daily trunk
|}

== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

- Create weekly schedule for MRT
- Wiki clean/update
- Hourly Automation improvement
- Plan Feature testset
- Bug verification

== QA Organization ==

== Error Management ==
* Error Manager Iekku Huttunen

== Usefull links ==

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

ARM/N900/QA

2011-04-07T08:54:11Z

Cindy: /* Test Execution Schedule */

= QA TODOs (in priority order) =

== MeeGo.com IRC meetings ==

There are no special QA meeting, other DE team meetings can be found from here:
* [[MeeGo-Meeting_IRC_Schedule|MeeGo-Meeting IRC Schedule]]

== OTS setup and automated hourly testing ==

=== OTS setup ===
* [http://194.136.64.78/logger/view/ OTS server] - Ville Ilvonen/Riku Halonen, DONE
* [http://194.136.64.78/logger/view/workers/ OTS worker(s) for core tests]- Ville Ilvonen/Riku Halonen/Timo Harkonen - DONE
* [http://qa-reports.meego.com/1.2/Core/Hourly%20-%20Automated Reporting of hourly tests to QA-reports] - Ville Ilvonen/Esa-Pekka Miettinen/Timo Harkonen - DONE
* Minimize automatic installation time to 10mins - Timo Makimattila, ONGOING
* OTS worker for UX tests - NOTSTARTED
* Power consumption measurements - NOTSTARTED

=== Test automation images ===
* Setup hourly image building for autotest image on own setup - Riku Halonen/Timo Harkonen/Ville Ilvonen - DONE (in OTS subnet, see above)

* We need to be able to control included test packages - ONGOING (http://meego.gitorious.org/meego-quality-assurance/handset-hourly-automated-tests)

* Move to use images from release engineering (1. download image, 2. install automation enablers, core dumping enablers etc. using mic-chroot, 3. install image to device 4. test) - NOT STARTED

If you need something else from QA please tell it to us :)

== Applications for testing ==
* qt-demos already available from the repos
* Small applications that use Qt mobility APIs to access things like sensors to help manual testing

== Test asset ==
* [https://bugs.meego.com/buglist.cgi?quicksearch=mcts List of open bugs for MCTS]
** see priorities below - discuss with Iekku about priorities of open bugs for MCTS tests
# WLAN cases
# Call/SMS cases
# Audio policy framework cases (lower priority)
# Camera cases (lower priority)
# Sensor data cases (Qt Mobility, lower priority)

== Crashdb support for ARM core dumps ==

Core dump processing and backtraces from crashing ARM processes.

* Rich Core dumping
** Make rich-core dumping to work in MeeGo N900 DE (basic functionality) - DONE (sampos, rikhalon)
*** Changes in MeeGo Gitorious ([https://meego.gitorious.org/meego-quality-assurance/rich-core/commits/meego-n900de meego-n900de] branch).
*** Dumps are generated in /home/meego/core-dumps
*** In file name, string "xxxx" is used instead of IMEI digits (privacy issue)
*** Get latest packages [http://repo.pub.meego.com/home:/rha/Project_DE_Trunk_Testing_standard/armv7l/ here]
*** Add "-corewatcher" and "-corewatcher-applet" to .ks file to remove overlapping corewatcher.
** Fix core-reducer (Something goes wrong, when reducer processes coredump.) - DONE (sampos)
** Oopslog (and lifelog) functionality - NOT STARTED

* Back-end server
** Set up back-end server for core processing - ONGOING (rikhalon)

== Boot time measurement ==
* Measure and optimize N900 boot time (timakima, ONGOING)

== CPU load measurement during audio/video playback ==

== Application Manager ==
* Implement an application manager (similar to one in N900/Fremantle) to control install/uninstall/update applications and other packages. (kyranto, ONGOING).

== Test Execution Schedule ==
* Core (Teivas)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release !! Priority
|-
| Monday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Monday
| Basic feature test
| Preview
| P2
|-
| Monday
| Sanity (OK)
| Trunk
| P3
|-
| Tuesday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Tuesday
| Sanity (OK)
| Trunk
| P2
|-
| Wednesday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Wednesday
| Data Flow
| Weekly
| P2
|-
| Wednesday
| Sanity (OK)
| Trunk
| P3
|-
| Thursday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Thursday
| Sanity (OK)
| Trunk
| P2
|-
| Friday
| Acceptance (OK)
| Trunk:Testing
| P1
|-
| Friday
| Sanity (OK)
| Trunk
| P2
|-
|}
* Handset UX weekly testing schedule (Rajala)
{| cellpadding="5" cellspacing="0" border="1"
! Day !! Test set (status) !! Release
|-
| Monday
| Key feature (OK)
| Preview
|-
| Monday
| Acceptance (OK)
| Testing trunk
|-
|Tuesday
|Acceptance (Ok)
|Testing trunk
|-
|Tuesday
|Sanity (Ok)
|Daily trunk
|-
|Tuesday
|DE Dataflow (Ok)
|Preview
|-
|Tuesday
|DE use cases (Ok)
|Preview
|-
|Wednesday
|Acceptance (Ok)
|Testing trunk
|-
|Wednesday
|Key feature (Ok)
|Weekly
|-
|Thursday
|DE Dataflow (Ok)
|Weekly
|-
|Thursday
|DE use cases (Ok)
|Weekly
|-
|Thursday
|Acceptance (Ok)
|Testing trunk
|-
|Thursday
|Sanity Ok
|Daily trunk
|-
|Thursday
|DE Reliability (Ongoing)
|Weekly
|-
|Thursday
|DE Performance (Ongoing)
|Weekly
|-
|Friday
|Acceptance (Ok)
|Testing trunk
|-
|Friday
|Sanity (Ok)
|Daily trunk
|}

== QA Tasks For Developer Edition ==
There is a wiki article about the [[ARM/N900/DeveloperEdition|Developer Edition]].

QA tasks for the Developer Edition differ from the usual N900 approach in that there are less features to be tested. There are currently 2 test sets for the Developer Edition, these are the Sanity Test Set and the Feature Test Set. They are described below.

=== Test Sets ===
==== Sanity Test Set ====
The sanity set should be run automatically on every image. As such it must meet the following requirements:
* 100% automated
* Testing only basic features
==== Feature Test Set ====
The feature set will be run periodically and will test the basic features as well as enablers for those features (e.g. PIM for phoning contacts). Performance will also be analysed, at the moment, this will include browser startup time and a CPU benchmark but this will be expanded later.

Suggestions are welcome.

=== Core QA Team Backlog ===
The Core QA team intends to keep its backlog as public as possible. The limitation on this is the amount of work generated by doing this. We will attempt to keep an up-to-date lists of tasks and progress on these tasks. At the moment, the task list looks like this:

- Create weekly schedule for MRT
- Wiki clean/update
- Hourly Automation improvement
- Plan Feature testset
- Bug verification

== QA Organization ==

== Error Management ==
* Error Manager Iekku Huttunen

== Usefull links ==

* [[ARM/N900/Install/MMC|Flashing instructions]]
Flashing tested with http://download.meego.com/testing-daily/builds/trunk/1.1.90.8.20110318.89/

[[Category:N900]]

Quality/Plans/Test-execution

2011-01-19T00:54:41Z

Cindy: /* Weekly executions */

== Test execution roadmap towards 1.2 ==

=== Daily executions ===

* Core OS Acceptance (against Trunk:Testing) on [[http://qa-reports.meego.com/1.2/Core/Acceptance/nCDK nCDK]], [[http://qa-reports.meego.com/1.2/Core/Acceptance/iCDK iCDK]], [[http://qa-reports.meego.com/1.2/Core/Acceptance/N900 N900]], [[http://qa-reports.meego.com/1.2/Core/Acceptance/Netbook Netbook]] and [[http://qa-reports.meego.com/1.2/Core/Acceptance/IVI IVI]]
* Core OS Sanity (against Trunk) on [[http://qa-reports.meego.com/1.2/Core/Sanity/nCDK nCKD]], [[http://qa-reports.meego.com/1.2/Core/Sanity/iCDK iCDK]], [[http://qa-reports.meego.com/1.2/Core/Sanity/N900 N900]], [[http://qa-reports.meego.com/1.2/Core/Sanity/Netbook Netbook]] and [[http://qa-reports.meego.com/1.2/Core/Sanity/Ivi IVI]]
* Handset UX Acceptance (against Trunk:Testing) on [[http://qa-reports.meego.com/1.2/Handset/Acceptance/Ncdk nCDK]] and [[http://qa-reports.meego.com/1.2/Handset/Acceptance/N900 N900]]
* Handset UX Sanity (against Trunk) on [[http://qa-reports.meego.com/1.2/Handset/sanity/Ncdk nCDK]] and [[http://qa-reports.meego.com/1.2/Handset/Sanity/N900 N900]]
* Netbook UX Acceptance (against Trunk:Testing) on [[http://qa-reports.meego.com/1.2/Netbook/Acceptance/Pinetrail Pinetrail]]
* Netbook UX Sanity (against Trunk) on [[http://qa-reports.meego.com/1.2/Netbook/Sanity/Pinetrail Pinetrail]]

=== Weekly executions ===

{| cellpadding="5" cellspacing="0" border="1"
! Iteration !! Iteration start -- end dates !! nCDK !! N900 !! iCDK !! Netbook !! IVI
|-
| style="background-color:#ffff00" | 1.1.80.10
| 2010-12-09 -- 2010-12-15
| Preview: [[http://qa-reports.meego.com/1.2/Core/Basic%20feature%20testing/Ncdk Core OS Basic Feature Testing]], 
Weekly: [[http://qa-reports.meego.com/1.2/Core/Dataflow/Ncdk Core OS Dataflow]], Preview and Weekly: [[http://wiki.meego.com/Quality/HandsetTestReport/HandsetGroupedKeyFeature1.2nCDK Handset UX QA Key Feature Testing]]
| Preview: [[http://qa-reports.meego.com/1.2/Core/Basic%20Feature%20Testing/N900 Core OS Basic Feature Testing]] Weekly: [[http://qa-reports.meego.com/1.2/Core/Dataflow/N900 Core OS Dataflow]], [[http://wiki.meego.com/Quality/HandsetTestReport/HandsetGroupedKeyFeature1.2 Handset Key Feature Testing]], [[http://qa-reports.meego.com/1.2/Handset/Sanity%20-%20automated/N900 Sanity (automated ramp-up)]]
| Preview: [[http://wiki.meego.com/Quality/HandsetTestReport/KeyBasicFeatureTestFornCDK_1.2/20101129 Handset Key Basic Feature Testing]]
|
|
|-
| style="background-color:#ffff00" | 1.1.80.11 MM2.5
| 2010-12-16 -- 2010-12-22
| Preview: [[http://qa-reports.meego.com/1.2/Core/Basic%20feature%20testing/Ncdk Core OS Basic Feature Testing]], 
Weekly: [[http://qa-reports.meego.com/1.2/Core/Dataflow/Ncdk Core OS Dataflow]], Preview and Weekly: [[http://wiki.meego.com/Quality/HandsetTestReport/HandsetGroupedKeyFeature1.2nCDK Handset UX QA Key Feature Testing]]
| Preview: [[http://qa-reports.meego.com/1.2/Core/Basic%20Feature%20Testing/N900 Core OS Basic Feature Testing]] Weekly: [[http://qa-reports.meego.com/1.2/Core/Dataflow/N900 Core OS Dataflow]], [[http://wiki.meego.com/Quality/HandsetTestReport/HandsetGroupedKeyFeature1.2 Handset Key Feature Testing]], [[http://qa-reports.meego.com/1.2/Handset/Sanity%20-%20automated/N900 Sanity (automated ramp-up)]]
| Preview: [[http://wiki.meego.com/Quality/HandsetTestReport/KeyBasicFeatureTestFornCDK_1.2/20101129 Handset Key Basic Feature Testing]]
|
| Weekly: [[http://qa-reports.meego.com/1.2/Ivi/Basic%20feature%20testing/Ia-russellville/423 IVI Basic Feature Testing]]
|-
| style="background-color:#ffff00" | 1.1.80.12
| 2010-12-23 -- 2010-12-29
| No Releases during this week.
| No Releases during this week.
| No Releases during this week.
|
| No Releases during this week.
|-
| style="background-color:#ffff00" | 1.1.80.13
| 2010-12-30 -- 2011-01-05
| Preview: [[http://qa-reports.meego.com/1.2/Core/Basic%20feature%20testing/Ncdk Core OS Basic Feature Testing]], 
Weekly: [[http://qa-reports.meego.com/1.2/Core/Dataflow/Ncdk Core OS Dataflow]], Preview and Weekly: [[http://wiki.meego.com/Quality/HandsetTestReport/HandsetGroupedKeyFeature1.2nCDK Handset UX QA Key Feature Testing]]
| Preview: [[http://qa-reports.meego.com/1.2/Core/Basic%20Feature%20Testing/N900 Core OS Basic Feature Testing]] Weekly: [[http://qa-reports.meego.com/1.2/Core/Dataflow/N900 Core OS Dataflow]], [[http://wiki.meego.com/Quality/HandsetTestReport/HandsetGroupedKeyFeature1.2 Handset Key Feature Testing]], [[http://qa-reports.meego.com/1.2/Handset/Sanity%20-%20automated/N900 Sanity (automated)]]
| Preview: [[http://wiki.meego.com/Quality/HandsetTestReport/KeyBasicFeatureTestFornCDK_1.2/20101129 Handset Key Basic Feature Testing]]
|
| Weekly: [[http://qa-reports.meego.com/1.2/Ivi/Basic%20feature%20testing/Ia-russellville/573 IVI Basic Feature Testing]]
|-
| style="background-color:#ffff00" | 1.1.80.14
| 2011-01-06 -- 2011-01-12
| Preview: [[http://qa-reports.meego.com/1.2/Core/Basic%20feature%20testing/Ncdk Core OS Basic Feature Testing]], 
Weekly: [[http://qa-reports.meego.com/1.2/Core/Dataflow/Ncdk Core OS Dataflow]], Preview and Weekly: [[http://wiki.meego.com/Quality/HandsetTestReport/HandsetGroupedKeyFeature1.2nCDK Handset UX QA Key Feature Testing]]
| Preview: [[http://qa-reports.meego.com/1.2/Core/Basic%20Feature%20Testing/N900 Core OS Basic Feature Testing]] Weekly: [[http://qa-reports.meego.com/1.2/Core/Dataflow/N900 Core OS Dataflow]], [[http://wiki.meego.com/Quality/HandsetTestReport/HandsetGroupedKeyFeature1.2 Handset Key Feature Testing]], [[http://qa-reports.meego.com/1.2/Handset/Sanity%20-%20automated/N900 Sanity (automated)]]
| Preview: [[http://wiki.meego.com/Quality/HandsetTestReport/KeyBasicFeatureTestFornCDK_1.2/20101129 Handset Key Basic Feature Testing]]
|
| Weekly: [[http://qa-reports.meego.com/1.2/Ivi/Basic%20feature%20testing/Ia-russellville/714 IVI Basic Feature Testing]]
|-
| style="background-color:#ffff00" | 1.1.80.15
| 2011-01-13 -- 2011-01-19
| Preview: [[http://qa-reports.meego.com/1.2/Core/Basic%20feature%20testing/Ncdk Core OS Basic Feature Testing]], 
Weekly: [[http://qa-reports.meego.com/1.2/Core/Dataflow/Ncdk Core OS Dataflow]], Preview and Weekly: [[http://wiki.meego.com/Quality/HandsetTestReport/HandsetGroupedKeyFeature1.2nCDK Handset UX QA Key Feature Testing]]
| Preview: [[http://qa-reports.meego.com/1.2/Core/Basic%20Feature%20Testing/N900 Core OS Basic Feature Testing]] Weekly: [[http://qa-reports.meego.com/1.2/Core/Dataflow/N900 Core OS Dataflow]], [[http://wiki.meego.com/Quality/HandsetTestReport/HandsetGroupedKeyFeature1.2 Handset Key Feature Testing]], [[http://qa-reports.meego.com/1.2/Handset/Sanity%20-%20automated/N900 Sanity (automated)]]
| Preview: [[http://wiki.meego.com/Quality/HandsetTestReport/KeyBasicFeatureTestFornCDK_1.2/20101129 Handset Key Basic Feature Testing]]
|
|
|-
| style="background-color:#99cc00" | 1.1.90.0 MM3
| 2011-01-20 -- 2011-01-26
|
|Preview: [[http://qa-reports.meego.com/1.2/Core/Basic%20Feature%20Testing/N900 Core OS Basic Feature Testing]] Weekly: [[http://qa-reports.meego.com/1.2/Core/Dataflow/N900 Core OS Dataflow]]
|
|
|
|-
| style="background-color:#99cc00" | 1.1.90.1
| 2011-01-27 -- 2011-02-02
|
|Preview: [[http://qa-reports.meego.com/1.2/Core/Basic%20Feature%20Testing/N900 Core OS Basic Feature Testing]] Weekly: [[http://qa-reports.meego.com/1.2/Core/Dataflow/N900 Core OS Dataflow]]
|
|
|
|-
| style="background-color:#99cc00" | 1.1.90.2
| 2011-02-03 -- 2011-02-09
|
|
|
|
|
|-
| style="background-color:#99cc00" | 1.1.90.3
| 2011-02-10 -- 2011-02-16
|
|Preview: [[http://qa-reports.meego.com/1.2/Core/Basic%20Feature%20Testing/N900 Core OS Basic Feature Testing]] Weekly: [[http://qa-reports.meego.com/1.2/Core/Dataflow/N900 Core OS Dataflow]]
|
|
|
|-
| style="background-color:#99cc00" | 1.1.90.4
| 2011-02-17 -- 2011-02-23
|
|Preview: [[http://qa-reports.meego.com/1.2/Core/Basic%20Feature%20Testing/N900 Core OS Basic Feature Testing]] Weekly: [[http://qa-reports.meego.com/1.2/Core/Dataflow/N900 Core OS Dataflow]]
|
|
|
|-
| style="background-color:#99cc00" | 1.1.90.5
| 2011-02-24 -- 2011-03-02
|
|Preview: [[http://qa-reports.meego.com/1.2/Core/Basic%20Feature%20Testing/N900 Core OS Basic Feature Testing]] Weekly: [[http://qa-reports.meego.com/1.2/Core/Dataflow/N900 Core OS Dataflow]]
|
|
|
|-
| style="background-color:#99cc00" | 1.1.90.6
| 2011-03-03 -- 2011-03-09
|
|
|
|
|
|-
| style="background-color:#99cc00" | 1.1.90.7
| 2011-03-10 -- 2011-03-16
|
|
|
|
|
|-
| style="background-color:#99cc00" | 1.1.90.8
| 2011-03-17 -- 2011-03-23
|
|
|
|
|
|-
| style="background-color:#00ff00" | 1.1.99.0 MM4
| 2011-03-24 -- 2011-03-30
|
|
|
|
|
|-
| style="background-color:#00ff00" | 1.1.99.1
| 2011-03-31 -- 2011-04-06
|
|
|
|
|
|-
| style="background-color:#00ff00" | 1.1.99.2
| 2011-04-07 -- 2011-04-13
|
|
|
|
|
|-
| style="background-color:#00ff00" | 1.1.99.3
| 2011-04-14 -- 2011-04-20
|
|
|
|
|
|-
| style="background-color:#339966" | 1.2.0.0 MM5
| 2011-04-21 -- 2011-04-27
|
|
|
|
|
|}

== Test execution roadmap for 1.1 update ==
=== Daily executions ===

* Netbook&Core UX Sanity on [[http://qa-reports.meego.com/1.1/Netbook/Update%20Sanity/Pinetrail Pinetrail]]

=== Monthly executions ===
{| cellpadding="3" cellspacing="0" border="1"
! Item !! Release Time !! Netbook
|-
| Update Release 1
| 2010-11
| System Functional: [[http://wiki.meego.com/Quality/1.0_sw_update_test_reports/Fullpass20100630 Netbook 1.1 Update Release 1 System Functional Test]]
|-
| Update Release 2
| 2011-01
| System Functional:
|}

== Test execution roadmap for 1.0 update ==
=== Daily executions ===

* Netbook UX Sanity on [[http://qa-reports.meego.com/1.0/Netbook/Update%20Sanity/Pinetrail Pinetrail]]

=== Monthly executions ===
{| cellpadding="3" cellspacing="0" border="1"
! Item !! Release Time !! Netbook
|-
| Update Release 1
| 2010-07
| Fullpass: [[http://wiki.meego.com/Quality/1.0_sw_update_test_reports/Fullpass20100630 Netbook 1.0 Update Release 1 Fullpass Test]]
|-
| Update Release 2
| 2010-08
| Fullpass: [[http://wiki.meego.com/Quality/1.0_sw_update_test_reports/Fullpass20100805 Netbook 1.0 Update Release 2 Fullpass Test]]
|-
| Update Release 3
| 2010-09
| Fullpass: [[http://wiki.meego.com/Quality/1.0_sw_update_test_reports/Fullpass20100909 Netbook 1.0 Update Release 3 Fullpass Test]] 
System Functional: [[http://wiki.meego.com/Quality/1.0_sw_update_test_reports/1.0UpdateSystem20100909 Netbook 1.0 Update Release 3 System Functional Test]]
|-
| Update Release 4
| 2010-10
| Fullpass: [[http://wiki.meego.com/Quality/1.0_sw_update_test_reports/Fullpass20101013 Netbook 1.0 Update Release 4 Fullpass Test]] 
System Functional: [[http://wiki.meego.com/Quality/1.0_sw_update_test_reports/1.0UpdateSystem20101013 Netbook 1.0 Update Release 4 System Functional Test]]
|-
| Update Release 5
| 2010-11
| System Functional: [[http://wiki.meego.com/Quality/1.0_sw_update_test_reports/Fullpass20101013 Netbook 1.0 Update Release 5 Fullpass Test]]
|-
| Update Release 6
| 2011-01
| System Functional:
|}

Quality/Glossary

2010-12-23T05:51:43Z

Cindy: /* Test case verdict */

=== Glossary ===

==== Smoke Test (a.k.a Sanity Test) ====

Smoke test ~ intake+smoke test from ISTQB.
smoke test (set): A subset of all defined/planned test cases that cover the main functionality of a daily build for MeeGo distribution, to ascertaining that the most crucial features of a distribution work, but not bothering with finer details. Smoke test is carried out at the start of test execution phase to decide if the distribution is ready for detailed and further testing.

The purpose of Sanity Test for MeeGo:
* Provide basic health status of the whole system on daily basis, so people have basic understanding where we are in terms of quality
* Report out outstanding issues and regressions and track them fixed quickly
* Sanity test results are used to measure if the MeeGo repositories are in good shape so that decisions can be made if the software is ready for further testing or release. Of course, the release decision cannot be made only based on sanity test results, further testing results will also be referred to.

==== Test Collection maintainer (a.k.a Test Suite maintainer) ====

Test collection maintainer ~ package maintainer (archlinux) + technical test analyst (ISTQB):
* The role of the test collection maintainer is to update tests as new versions become available upstream and to field support questions relating to bugs in said tests. The term may be applied to any of the following:
** A test analyst who maintains a test collection in one of the official test repositories (tests.meego.com). Test analyst shall be capable to:
*** Analyze tests and the internal structure of the system in sufficient detail to meet the expected quality level
*** Evaluate tests in terms of technical quality attributes as performance, security, etc.
** A trusted tester of the community who maintains test collections in the unsupported/unofficial community test repository.
* The maintainer of a test collection is the person currently responsible for the test collection. Previous maintainers should be listed as contributors along with others who have contributed to the test collection.

Test collection ~ tests testing on specific areas of the distribution (e.g. Application Group, Application, Domain, Component).
You can replace the test collection maintainer with test package maintainer – if you wish.

==== Test case ====

* (Low level) Test case: A test case with concrete (implementation level) values for input data and expected results.
* High level test case (a.k.a test idea): A test case without concrete (implementation level) values for input data and expected results. Logical operators are used; instances of the actual values are not yet defined and/or available.

==== Test case verdict ====

* Pass: A test is deemed to pass if its actual result matches its expected result.
* Fail: A test is deemed to fail if its actual result does not match its expected result.
* N/A - Not Applicable - can be seen as a initial value which is left as a verdict for a test case when no other verdict cannot be given due to some reason (e.g. test case could not be executed due to failure in test infra, etc.).
* Blocked: This is special value required for the situations where verdict cannot be given due to failure in preconditions.
* Measured: Non-functional cases whose data has been collected but verdict cannot be given due to missing pass/fail criteria

For all Blocked and N/A verdicts it would be preferable that reason why pass/fail verdict could not be given is documented either by comment “Test case X verdict not given due to reason Y" or bug ID.

For the case which functionality is implemented but case is missing, it is recommended to find case substitute to cover the check point. If no substitute is available to cover, mark the case as N/A.

=== References ===

* [[http://www.istqb.org/downloads/glossary-current.pdf ISTQB Glossary]]

[[Category:QA]]

Quality/Glossary

2010-12-01T05:56:13Z

Cindy: /* Test case verdict */

=== Glossary ===

==== Smoke Test (a.k.a Sanity Test) ====

Smoke test ~ intake+smoke test from ISTQB.
smoke test (set): A subset of all defined/planned test cases that cover the main functionality of a daily build for MeeGo distribution, to ascertaining that the most crucial features of a distribution work, but not bothering with finer details. Smoke test is carried out at the start of test execution phase to decide if the distribution is ready for detailed and further testing.

The purpose of Sanity Test for MeeGo:
* Provide basic health status of the whole system on daily basis, so people have basic understanding where we are in terms of quality
* Report out outstanding issues and regressions and track them fixed quickly
* Sanity test results are used to measure if the MeeGo repositories are in good shape so that decisions can be made if the software is ready for further testing or release. Of course, the release decision cannot be made only based on sanity test results, further testing results will also be referred to.

==== Test Collection maintainer (a.k.a Test Suite maintainer) ====

Test collection maintainer ~ package maintainer (archlinux) + technical test analyst (ISTQB):
* The role of the test collection maintainer is to update tests as new versions become available upstream and to field support questions relating to bugs in said tests. The term may be applied to any of the following:
** A test analyst who maintains a test collection in one of the official test repositories (tests.meego.com). Test analyst shall be capable to:
*** Analyze tests and the internal structure of the system in sufficient detail to meet the expected quality level
*** Evaluate tests in terms of technical quality attributes as performance, security, etc.
** A trusted tester of the community who maintains test collections in the unsupported/unofficial community test repository.
* The maintainer of a test collection is the person currently responsible for the test collection. Previous maintainers should be listed as contributors along with others who have contributed to the test collection.

Test collection ~ tests testing on specific areas of the distribution (e.g. Application Group, Application, Domain, Component).
You can replace the test collection maintainer with test package maintainer – if you wish.

==== Test case ====

* (Low level) Test case: A test case with concrete (implementation level) values for input data and expected results.
* High level test case (a.k.a test idea): A test case without concrete (implementation level) values for input data and expected results. Logical operators are used; instances of the actual values are not yet defined and/or available.

==== Test case verdict ====

* Pass: A test is deemed to pass if its actual result matches its expected result.
* Fail: A test is deemed to fail if its actual result does not match its expected result.
* N/A - Not Applicable - can be seen as a initial value which is left as a verdict for a test case when no other verdict cannot be given due to some reason (e.g. test case could not be executed due to failure in test infra, etc.).
* Blocked: This is special value required for the situations where verdict cannot be given due to failure in preconditions.
* Measured: Non-functional cases whose data has been collected but verdict cannot be given due to missing pass/fail criteria

For all Blocked and N/A verdicts it would be preferable that reason why pass/fail verdict could not be given is documented either by comment “Test case X verdict not given due to reason Y" or bug ID.

=== References ===

* [[http://www.istqb.org/downloads/glossary-current.pdf ISTQB Glossary]]

[[Category:QA]]

Quality/Plans/MeeGo Core Test Plan

2010-11-16T08:12:27Z

Cindy: /* Risks */

'''Proposal For MeeGo 1.2. This plan is under review. actively revising and updating'''
= MeeGo Core Overall Test Plan =

== Introduction ==
This is overall test plan for MeeGo Core of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo Core. It will give readers an overview of validation activities for MeeGo Core of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo Core stack. It will be the joint effort from MeeGo community.

This plan describes MeeGo Core OS QA for MeeGo 1.2 and onwards. Test plan is revised for each MeeGo Release and thus this test plan should be considered as evolving document. New testing methods, practicalities and approaches are described in revisions.

== Objectives ==
Objectives in MeeGo Core OS QA is to validate the functionality of entire MeeGo Core OS software delivery by performing hourly, daily and weekly testing for weekly releases. More information about testing cycle and test types can be found later in this document. Target is to ensure that:

* New features introduced in specific Core OS weekly release are working as specified as a part of system.
* Stakeholders and community get visibility to release quality.
* Validate that relevant bugs are fixed in software release.

Weekly testing is cumulative in terms of test coverage. Week to week test cases included in test run will vary and new test cases are introduced. Thus test case coverage increases constantly. Increase is dependent on release content (how many new features there was released in specific weekly release).

For these activities MeeGo Core OS QA follows iteration cycle and process described in [[Release_Engineering/Process|Release Engineering’s Process]].

As addition to fast cycle testing more thorough testing (Full Pass) is done for MeeGo Releases. Full Pass testing is massive test execution for entire test asset available at the moment. With full pass all features are re-verified and regression is measured. This activity is taking place after MeeGo Release Feature Complete. Target is to ensure that:

* Delivered features for MeeGo Core OS are working as specified as a part of system.
* MeeGo Core OS is performing well and is reliable.
* Program maturity statement can be given.

In order to verify features of MeeGo Core OS requires exhaustive documentation of feature under test. insufficient documentation has negative impact to test asset quality as stated in [[Quality/TestSuite/MCTS/MCTS_Development_Guideline|MCTS Development Guidelines]].

For these activities MeeGo Core OS QA follows release cycle described in [[Release_Engineering/Release_Timeline|Release Engineering’s release timeline]].

== Test strategy and approach ==

=== Introduction ===

The overall objective of MeeGo Core QA is to ensure that MeeGo middleware and OS Base provide stable hardware and usage model independent application services and APIs for building the vertical specific user experience. Each core component has different quality risk regarding to MeeGo integration. For example, some core component is mature in upstream and MeeGo do the integration without heavy new feature development; some component is contributed and open-sourced from proprietary component with heavy development. Considering most of MeeGo Core components will be adopted by multiple vertical usages and run on a number of MeeGo devices, Test execution efficiency shall taken into account when creating the test cases. Given this, there are following strategy considerations:

*Unified test suite (MeeGo Core Test Suite) for MeeGo Core validation. It's open-sourced and could start with the first batch of test cases contributed from community. MeeGo Core Test Suite (MCTS) docs can be found at MeeGo Quality page, and the code in Gitorious.
*Test suites from different sources are utilized as much as possible (e.g. Qt Mobility API test cases)
*If test cases are available in upstream project they will be used in order to keep maintainability.
*Appropriate test coverage in defined (density) categories. The test coverage of each component is based on the quality risk regarding to MeeGo integration.
*Test cases are automated s much as possible and able to run under MeeGo automated testing framework (OTS).
*MeeGo Core test cases are independent from vertical specific UX’s

MeeGo Core OS is verified with test assets available in MeeGo GIT in different projects and other open source projects. One can divide used test cases to two different types:

*Component tests. Unit/module like test cases verifying API’s of specific component.
*System tests. E2E type system level test cases verifying MeeGo Core OS stack as whole.

Typical characteristics for these component test cases are that they verify specific component or library with extensive set of parameters. Often they are missing E2E approach where entire stack is exercised. Test cases may not necessarily leave from component under test.

Typical characteristics for these system test cases are that they are based on use cases or user stories and often testing entire stack from top most interfaces provided by MeeGo Core OS Middleware and exercises HW peripheral beneath SW stack. These types of test suites are the most efficient ones for measuring and providing visibility to maturity of MeeGo Core OS.

MeeGo Core OS QA uses mainly test framework and other testing tools provided by [[Quality/QA-tools|MeeGo QA Tools Team]].

In order to ensure that MeeGo is competitive SW platform MeeGo Core OS QA is executing Performance test cases and driving performance improvements to MeeGo Core OS stack. Majority of the performance test cases are measuring raw performance of the system and do not necessarily measure end user experience. End user experience (response time measurements) is measured by Handset UX QA. For more detailed information of End User Experience testing see [[Quality#MeeGo_Handset_Testing|Handset UX Test Plan]].

In order to ensure reliability of MeeGo, MeeGo Core OS QA is executing Reliability test cases and driving reliability improvements to MeeGo Core OS stack. As addition to conventional test types such as Long-lasting and iterative, also Feature Interaction Testing is done as part of reliability testing. Feature Interaction Testing is based on user scenarios.

Test cases are following test type definition Aligned with ISO/IEC 9126-1 Software Quality Model and ISTQB Advanced Level Syllabus. Test types are defined in [[Quality/TestAreas|test areas]].

=== Development teams and MeeGo Core OS QA ===

It is a fact that Quality Assurance cannot create quality of the release by doing exhaustive testing. Quality is build in development phase by developers contributing to MeeGo.

Developers has significant role also in QA.Here are QA recommendations for developers contributing to MeeGo:

* Developer contributing to MeeGo release content shall verify their deliveries prior to commit.
* Test results shall be posted to appropriate MeeGo mailing list and results shall be stored to punblicly available location (link to be added to MeeGo Core testing results).
* Provide test asset with the component including detailed documentation.
* Propose sanity test cases for the component To QA Contact in order to include them to Sanity and to weekly testing.
* Maintain test asset.

At the end of the day, developer is responsible of Quality of the his/hers delivery.

== Quality requirements for MCTS ==

Even though there are certain problematic when testing code with a code it is very efficient of testing while:

*Fast test execution time & feedback
*Enables high automation rate (less resource hungry)
*Error debugging is fast (focused errors with appropriate log files and root cause analysis can be provided)
*Test coverage can be defined easily.

In order to take advantage of items described above test asset shall follow demanding quality standards. Test asset producing lots of false positives and negatives confuses community, provides wrong information about release quality and sends developers to wild goose hunt. This shall never happen. To ensure this MCTS code will follow quality requirements described [[Quality/TestSuite/MCTS/MCTS_Development_Guideline|MCTS Development Guidelines]].

== Feature test and System test ==
QA target is to validate MeeGo distribution

*Feature functionality
*System functionality (Interaction and negative scenarios)
*System performance (Data Throughput, Latencies, Framerate, etc.)
*System reliability

=== Feature Testing ===
*In this context known also as Component Testing.
*Target is to verify feature provided by one or more component with specific test cases designed for this purpose. Found 3G Device
*Every component (formed by features) basic functionality is tested in weekly testing and after feature complete phase as full pass testing.

=== System Testing ===
*Target is to test basic (functional) system testing of several components/features simultaneously. Test case example: Bluetooth, send 1MB file L2Cap PC.
*Target is to test system testing (performance). Test case example: NFT-BT-L2CAP_Send_10MB-THRO-MS (target: 1.9Mbit/s)
*Target is to test system testing (reliability). Test case example: NFT-BT-L2CAP_Send_2h-LOLA-MS (target: Data transfer for 120mins without performance decrease)
*Not tested NFT types: Performance (Memory Consumption and Power Management) and Reliability (Aging and Long Period)

== Testability ==
Testability of MeeGo 1.2 Core OS features is ensured.

* Features are defined by Product Management and relevant stakeholders to Bugzilla tool.
* Defined QA Owner checks features through from Bugzilla against defined Testability Checklist and adding comment to feature in Bugzilla that can feature be used as QA input and it is possible validate in software release with relevant test case(s). Also more information is requested from Feature owner if it is seen insufficient from QA perspective.
* QA does not does not accept that features without proper information to ensure testability are assigned to development.

Testability can be seen as main key for the success of QA. In order to ensure high quality QA, testability percentage of the MeeGo 1.2. features shall be 90% or higher.

Relevant Links
* http://bugs.meego.com/ (MeeGo 1.2 Core OS Features stored in Bugzilla)
* [[Quality/Plans/Testability-checklist|Testability Checklist]]

== Test cases ==
Well defined test cases are the key to success in MeeGo Core OS Testing. While the objective of testing is assist developers in creating software that functions correctly, quite often testing falls into the trap of attempting to demonstrate that the software works. This shall be avoided. In test case development following should be considered:

*Do not attempt to find work-arounds for problems in the API under test: if A + B = C and that is not the obtained result, then the test must fail
*Tests should be written against the specification/documentation of software instead of against the implementation as it is the implementation itself that is under test.
*Always attempt to 'break' the software under test: Do not only test things that are known to be working, testing the unknown areas is as important as testing the commonly used functions.
*The final objective of testing is to provide information/evidence to developers, therefore, the more (and more detailed) information that the test can provide, the better.

MeeGo Core OS Test Design follows spirit of [[Quality/TestDesignProcessAndGuideline|MeeGo QA Common Test Design Process and Guidelines]]. Specifics being:
*Test Cases are designed by test asset developers based on QA Owners input.
*QA Owners input is based on existing features and which have been approved from testability point of view.
*Test Cases are described according to [[Quality/QA-tools/Test plan|test definition]] in tests.xml files
*Tests.xml files and are stored to MeeGo Gitorious.
*[[TestCaseTemplate|Common Test Case Template]] is used when designing test cases.

== Features to Be Tested ==

Overall the MeeGo Core Testing will cover the '''MeeGo OS Middlewares layer''' and '''MeeGo OS Base layer''' in [http://meego.com/developers/meego-architecture MeeGo Architecture]:
[[File:MeeGoArch.png]]

Specific features to be tested will be aligned with the features under '''MeeGo Core OS Features''' product in [http://bugs.meego.com MeeGo Featurezilla]

== What will not be tested ==
Following feature category won't be covered in MeeGo Core validation for MeeGo open source releases.
* Any proprietary component which is not included in the open source MeeGo release
* Certification testing like: Ofono, Bluetooth, USB, WLAN, and similar.
* Any UI or application related testing.

=== Test Coverage Definition ===
In order to use resources efficiently and control risk level on component maturity. Testing is done in different levels as follows:

* '''Thorough'''
** Full testing service for API
** Target is to have 100% function/method coverage provided by API
** Parameter coverage defined by using testing techniques as follows:
*** All Pairs (Pair wise)
*** Equivalence Partitioning
*** Boundary Value Analysis
*** Subjective test case selection (according to Expert opinion or similar)

* '''Average'''
** Between Thorough and Light
** Full API coverage (according to rules for Thorough) for new functionality.
** Integration level testing for legacy features.

* '''Light'''
** Integration level testing
** Upstream Middleware components shall be tested with limited amount of test cases (sub set) ensuring that integration of component to MeeGo is successful and component is working as expected

* '''Not Covered'''
** No Middleware testing, covered indirectly by UX Testing
** There will be API’s not feasible to cover with test cases at middleware level but more efficiently tested indirectly by UX testing

=== Test Coverage Measurement ===
In order to understand how well certain component is covered with test cases there shall be test coverage measurement done. This is directly linking to risk level of specific component. Test coverage is based on Function/Method coverage per API.

[[Quality/TestSuite/MCTS/MCTS_API_analysis|MCTS API analysis]] describes methods to be used for test coverage measurement.

=== Component Test Plans and Coverage ===
Each component will have detailed test plan for the specific component. Test Plans are available in [[../../TestSuite/MCTS#Test_Suite_Status_For_Middleware |MeeGo Core Test Suite]] for details (each component name is link to test plan).

Test plan shall follow structure described below: :

* 1 Introduction
* 2 Test Coverage
** 2.1 Software Coverage
** 2.2 Hardware Coverage
** 2.3 Items not Covered
* 3 Running the tests
** 3.1 Executing tests
** 3.2 Test case arguments
** 3.3 Test cases
** 3.4 Configuration file
* 4 Module Design and Architecture
** 4.1 Architecture diagram
* 5 References
* 6 Change History

See [[Quality/TestSuite/Video_Playback_Driver_Test_Specification|Video playback driver test plan]] for reference.

=== Test Cycle ===
MeeGo Core will be tested from the following different test execution levels. Testing Gear Box is as follows.

[[File:Coreosqawkcycle.PNG]]

==== Hourly Testing ====
Testing is done against Trunk:Testing. It will run a portion of fully automated test cases for core components and aims to provide quick acceptance testing to support incremental packages integration. It will be conducted under OTS (Open Test System).

==== Sanity (Daily) Testing ====
Testing is done against Trunk and also for weekly release prior to release announcement to provide visibility to release quality and to ensure that last fixes does not cause regression to release. Release Engineering includes links to test reports in release announcement. Sanity testing is static set of test cases which is modified on need basis. Thus Sanity test set may contain test cases for functionality which are not introduced yet. These test cases are marked as N/A with comment that feature not integrated yet. Sanity testing consists of:

* Set of fully automated test cases for release content developed under Bugzilla components
* Set of fully automated system level test cases verifying functionality from top most interfaces provided by MeeGo Core OS.

Daily Sanity testing aims to identify the regressions as early as possible and provide easy to understand visibility to SW maturity. This testing is answering to questions like:

* Is it possible to scan and connect to WLAN Access Point and conduct data transfer as a enabler for browsing?
* How fast specific file is downloaded over WLAN as a enabler for good user experience for browsing.

While test cycle needs to be fast, reliability is not reasonable to measure in daily testing.

==== Weekly Testing ====
Weekly Testing is a test cycle against the weekly preview images released from Release Engineering. Test frequency is once a week, which aligns with distribution's weekly image release cadence.
Weekly testing is incremental testing and target for weekly testing is to:

* Test the new features early
* Bug verification in order to track critical/major bugs got fixed in timely fashion
* Measure regression

'''New features''' are verified as soon as they are ready for testing. QA Owners follows release engineering’s release plans and feature status in Featurezilla. When feature is turned to Released sate in Featurezilla, test cases mapped to this feature are taken as part of next weekly testing execution. If test cases for specific feature are passing, Feature shall be marked as verified in Featurezilla.

'''Regression''' test cases are chosen amongst test cases designed for newly verified feature and are included in next weekly testing round.
Regression is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.
The regression test will be taken in following test cycle:

* In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
* In the intermediate regular release, only the modified features need to be heavily tested.

'''Bug verification''' on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Performance or reliability test cases by default are not included in weekly testing. Performance ro Reliability test cases are included only if bug fix has been provided against performance reliability related Bugzilla item or there are other suspicious changes in release content which may have a effect to performance or reliability

==== Full Pass (Milestone) Testing ====
In the formal test cycle for milestone test, after a new build passed the weekly test, QA will start the full validation testing for it, following test will be involved:

* Component Functional Test – which will follow the Bugzilla component test plan to run all tests.
* System functional Test – which are defined in the individual Bugzilla component test plan.
* System Non-functional Test – which are defined in the individual Bugzilla component test plan.

Purpose of the Full Pass is to measure release maturity in detailed level. In Full Pass testing entire test asset is executed for all the features released and previously marked as verified. Thus visibility and detailed information about release maturity is gained. Target is to have two Full Pass testing cycles during release life cycle. First Pull Pass test round starts at feature complete and last round ends few days prior to release date.

Between these two rounds failing cases and related bugs are followed closely in weekly testing. If there are very good grounds Full Pass testing can be executed more than twice during release cycle life cycle.

=== Test Report ===
* Send Test report to mailing list
** Weekly MeeGo Core Test Report (Bug verification, new feature exploration...)
** Distribution Milestone Full Pass Test report
* Archive reports at [[Quality/CoreTestReports|MeeGo Quality]] wiki

== Environmental Needs ==
=== Hardware Platforms ===
MeeGo Core OS is tested with numerous reference devices. The public reference configurations used are:
* Aava DV1/DV2
* N900
* MRST CDK
* Netbook

=== Test enviromental requirements ===

Each test suite shall contain README file describing test environment in detailed level. In a complex cases specific test environment description can be provided with reference in README file.

Test environment description includes everything needed to run the test or tests. This included hardware and software configuration of the device under test as well as any equipment (and its configuration) outside the device itself.

== Risks ==

One main expectation from program management is that QA is able to pinpoint risks concerning Quality Risks. For MeeGo 1.2 QA sees following risks:

{|cellspacing="0" border="1"
!|Risk #
!|Description
!|impact
!|Severity
!|Mitigation actions

|-
|| 1 || Insufficient documentation and/or low quality feature descriptions not meeting that testability requirements || Bad quality test asset not verifying feature in detailed level. || '''High''' || Significant effort put to feature testability in early phase of release life cycle. Review test assets with feature owner and assignees.
|-
|| 2 || Insufficient amount of system level test cases to show functionality of the release || Unclear release quality statement. Community confused and developers hunting errors which do not exist|| '''High''' || Significant effort to system level test case development. Each functionality (main success scenario and it's variants tested at least with one set of parameters.
|-
|| 3 || Changed components are verified in weekly testing but if dependencies are not understood impact to indirectly affected component not verified. || Surprises in release quality || Medium || Developers to address dependencies in detailed level showing significant dependencies
|}

== References ==
* MCTS (MeeGo Core Test Suite) code git tree: http://gitorious.org/qa-tools (to be moved meego.gitorious.org)
* MCTS docs http://wiki.meego.com/Quality/TestSuite/MCTS
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MeeGo Core test reports archive: http://wiki.meego.com/Quality/CoreTestReports
* MeeGo bugzilla: http://bugs.meego.com
* MeeGo 1.2 release planning: http://wiki.meego.com/Core_OS_Program_Dashboard#1.2_Release
* MeeGo test area terminology: http://wiki.meego.com/Quality/TestAreas
* Release Engineering Process: http://wiki.meego.com/Release_Engineering/Process
* Release Engineering/Release Timeline: http://wiki.meego.com/Release_Engineering/Release_Timeline
* MCTS Development Guideline: http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_Development_Guideline
* QA tools team: http://wiki.meego.com/index.php?title=Quality/TestPlan/MeeGo_Core_Test_Plan&action=edit&section=28
* MeeGo Handset Testing: http://wiki.meego.com/Quality#MeeGo_Handset_Testing
* Testability Checklist: http://wiki.meego.com/Quality/TestabilityChecklist
* Test Design Process And Guideline: http://wiki.meego.com/Quality/TestDesignProcessAndGuideline
* Test Packaging: http://wiki.meego.com/Test_Packaging/Test_Plan
* Common Test Plan Template: http://wiki.meego.com/TestCaseTemplate
* MeeGo Architecture: http://meego.com/developers/meego-architecture
* MCTS API analysis: http://wiki.meego.com/Quality/TestSuite/MCTS/MCTS_API_analysis
* Video Playback Driver Test Specification: http://wiki.meego.com/Quality/TestSuite/Video_Playback_Driver_Test_Specification