Views

SDKTestPlan

From MeeGo wiki

(Difference between revisions)

Latest revision as of 16:21, 15 June 2011

1 MeeGo SDK Test Plan

MeeGo SDK Test Plan

Scope of this Document

This is overall test plan for MeeGo SDK(Software Development Kit) of MeeGo open source project, which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo SDK. It will give readers an overview of validation activities for MeeGo SDK of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo SDK.

Features to Be Tested

Overall the MeeGo Core Testing will cover below host, target and testpoints:

P1: Test focus with main functionality, make sure common case work
P2: Nice to have for coming release
TBD: not covered in coming release

Tools to set up develop environment

Tools	Priority
Emulator: Qemu	P1
Simulator: Xephyr + chroot	P1
VirtualBox	TBD

Test Host & Target

Host\Target	Netbook	Handset	CDK	ARM based platform
Fedora13	P1	P1	TBD	*
Fedora12	P2	P2	TBD	*
Ubuntu10.04	P1	P1	TBD	*
Ubuntu9.10	P2	P2	TBD	*
OpenSuse 11.2	TBD	TBD	TBD	*
OpenSuse 11.3	TBD	TBD	TBD	*
WindowsXP	P1	P1	TBD	*
Win 7	TBD	TBD	TBD	*
Meego Netbook	TBD	TBD	TBD	*
Mac OS	TBD	TBD	TBD	*

means to be filled

Check Pointer

Check Point		Priority
SDK installer works		P1
SDK un-installer works		P1
OBS remote build works		TBD
OBS local build works		TBD
SDK images compliance with Meego images		P1
Boot SDK images with Qemu VMM Test Plan		P1
Boot SDK images with Xephyr		P2
Basic function check for SDK image	check basic services: network, telephond, dbus, etc.	P1
Qt Creator	Compiling plugins: OBS plugin, MADDE plugin Run plugins: Run on Device plugin, Qemu plugin, Xephyr/chroot plugin Debug plugins:GDB plugin, oprofile plugin, valgrind plugin	P1
MADDE		P1
sysroots		P1
toolchains		P1
App development program, build, debug inside simulator or emulator deploy to target and remote debug on target	CLI QT GUI QT MTF QT (handset only) WRT	P1 P1 P1 TBD
Documents & Toturials		P1
Additional tools		TBD
Memory/performance/power check		TBD

What will not Be Tested

Following feature category won't be covered in MeeGo SDK testing.

Only lightly test for SDK image, not check all of the functionalities or applications inside it, but only major deamon and service of it.
Not test for SDK package or image building, only check the built out package or images.
Not cover Meego compliance test, but only check key package versions which should align with Meego release.
64bit support is in plan, will be 2nd priority for testing.
VirtualBox support is still under investiagtion, will be taken into consideration for next step.

Test Strategy and Approach

The overall objective of MeeGo SDK testing is to ensure MeeGo SDK provide stable environment for developers to program on host machine within simulator or emulator, conduct build and debugging inside it, deploy the application to target and remote debug on target. The test cycle, bug lifecycle defined below may not apply to SDK testing because of not clear about if distribution will be involved for short term, but it can be refered and modified after we get synchronized on these opens.

Test Cycle

MeeGo SDK will be tested from the following different test execution levels according to detail development, release status.

Weekly Testing

Weekly Testing is a test cycle against the weekly images released from distribution team. Usually the test frequency is twice a week, which aligns with distribution's weekly image release cadence.

It's an incremental testing and includes two parts in general: test the basic distribution health to report out regression, test new features when they are ready for testing. We will also do bug verification and regression test which is set of tests to verify that changes from the last build (code enhancements, bug fixes) don't introduce new issues to the previous working code. The purpose of weekly testing is to early test the new features, to track critical/major issues got fixed in timely fashion, and to report out the overall quality status of the whole image so that people could understand where we are.

Milestone Testing

In the formal test cycle for milestone test, after a new build passed the BAT test, test team will start the full validation testing for it.

Regression Testing

This is a set of tests to verify that changes from the last build (code enhancements, bug fixes) don’t introduce new issues to the previous working code as well as new features work as expected. This cycle include the tests for previous major bug fixes and areas of the code that will be affected by new implementation.

The regression test will be taken in following test cycle:

In the test cycle for milestone release, before the release criteria is satisfied, it is possible that there are several release candidates which includes bug fixes for previous release candidates. These bug-fixes need be tested
In the intermediate regular release, only the modified features need to be heavily tested.
Bug verification on weekly basis to make sure the bug fixes be verified within one week after bug fixing

Bug life cycle

Bug life cycle follow Meego general definition, refer to MeeGo bugzilla guide
SDK has customized bugzilla to meet project characters, refer to SDK Bugzilla Components

Test Report

Send Test report to mailing list
- Weekly MeeGo SDK Test Report (Bug verification, new feature exploration...)
- Distribution Milestone Full Pass Test report
Archive reports at MeeGo SDK Quality wiki

Environmental Needs

Hardware Platforms

Host machine
- Fedora 13
- Fedora 12
- Ubuntu9.10
- Ubuntu10.04
- OpenSuse 11.2
- OpenSuse 11.3
- windows XP
- Win7
Target machine
- Netbook -- AspireOne NAV50
- Handset -- Aava
- ARM based Devices-- N900

Peripherals required

Category	Peripherals
USB devices	keyboard, mouse, USB storage device , ...

Network Environment

The networking arrangements needed to conduct the testing:

WLAN
LAN
USB

References

Meego SDK Development Guide: http://wiki.meego.com/MeeGo_SDK_Development_Guide
Meego1.1 Application SDK Project Plan: http://wiki.meego.com/SDK/MeeGo_1.1_Application_SDK_Project_Plan
Meego SDK requirement document File:MeeGo SDK Requirements.xls
MeeGo Architecture: http://meego.com/developers/meego-architecture
MeeGo bugzilla: http://bugs.meego.com

Retrieved from "http://wiki.meego.com/SDKTestPlan"

@@ Line 2: / Line 2: @@
 == Scope of this Document ==
-This is overall test plan for MeeGo SDK(Software Development Kit) of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo SDK. It will give readers an overview of validation activities for MeeGo SDK of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo SDK. It will be the joint effort from Nokia, Intel and MeeGo community.
+This is overall test plan for MeeGo SDK(Software Development Kit) of '''MeeGo open source project''', which defines test scope, test strategy, test configurations as well as test execution cycle for MeeGo SDK. It will give readers an overview of validation activities for MeeGo SDK of MeeGo open source releases. A series of component test plans will also be linked in this overall test plan to cover the detailed test approaches and test design for ingredients of MeeGo SDK.
 == Features to Be Tested ==
@@ Line 16: / Line 16: @@
 | Emulator: Qemu || <font color="blue">P1</font>
 |-
-| Simulator: Xephyr + chroot || <font color="orange">P2</font>
+| Simulator: Xephyr + chroot || <font color="orange">P1</font>
+|-
+| VirtualBox || TBD
 |}
 === Test Host & Target ===
 {|cellspacing="0" border="1"
@@ Line 23: / Line 26: @@
 !|Netbook
 !|Handset
-!|Tablet
+!|CDK
+!|ARM based platform
 |-
-|Fedora13 || <font color="blue">P1</font> || <font color="blue">P1</font> || <font color="blue">P1</font>
+|Fedora13 || <font color="blue">P1</font> || <font color="blue">P1</font> || TBD || *
 |-
-|Fedora12 || <font color="orange">P2</font> || <font color="orange">P2</font> || <font color="orange">P2</font>
+|Fedora12 || <font color="orange">P2</font> || <font color="orange">P2</font> || TBD || *
 |-
-|Ubuntu10.04 || <font color="blue">P1</font> || <font color="blue">P1</font> || <font color="blue">P1</font>
+|Ubuntu10.04 || <font color="blue">P1</font> || <font color="blue">P1</font> || TBD || *
 |-
-|Ubuntu9.10 || <font color="orange">P2</font> || <font color="orange">P2</font> || <font color="orange">P2</font>
+|Ubuntu9.10 || <font color="orange">P2</font> || <font color="orange">P2</font> || TBD || *
 |-
-|OpenSuse 11.2 || TBD || TBD || TBD
+|OpenSuse 11.2 || TBD || TBD || TBD || *
 |-
-|OpenSuse 11.3 || TBD || TBD || TBD
+|OpenSuse 11.3 || TBD || TBD || TBD || *
 |-
-|WindowsXP || <font color="blue">P1</font> || <font color="blue">P1</font> || <font color="blue">P1</font>
+|WindowsXP || <font color="blue">P1</font> || <font color="blue">P1</font> || TBD || *
 |-
-|Win 7 || TBD || TBD || TBD
+|Win 7 || TBD || TBD || TBD || *
 |-
-|Meego Netbook || TBD || TBD || TBD
+|Meego Netbook || TBD || TBD || TBD || *
 |-
-|Mac OS || TBD ||TBD ||TBD
+|Mac OS || TBD ||TBD ||TBD || *
 |}
+* means to be filled
 === Check Pointer ===
 {|cellspacing="0" border="1"
@@ Line 52: / Line 59: @@
 |-
 |colspan="2"|SDK installer works|| <font color="blue">P1</font>
+|-
+|colspan="2"|SDK un-installer works|| <font color="blue">P1</font>
+|-
+|colspan="2"|OBS remote build works|| TBD
+|-
+|colspan="2"|OBS local build works|| TBD
 |-
 |colspan="2"|SDK images compliance with Meego images || <font color="blue">P1</font>
 |-
-|colspan="2"|Boot SDK images with Qemu || <font color="blue">P1</font>
+|colspan="2"|Boot SDK images with Qemu [http://wiki.meego.com/Quality/Plans/VMM_test_plan VMM Test Plan]|| <font color="blue">P1</font>
 |-
 |colspan="2"|Boot SDK images with Xephyr || <font color="orange">P2</font>
 |-
 |Basic function check for SDK image ||
-*e.g. network services, dbus services, etc.
+*check basic services: network, telephond, dbus, etc.
-*'''need more detail checklist here'''|| <font color="blue">P1</font>
+|<font color="blue">P1</font>
 |-
-|colspan="2"|Qt Creator || <font color="blue">P1</font>
+|Qt Creator ||
+*Compiling plugins: OBS plugin, MADDE plugin
+*Run plugins: Run on Device plugin, Qemu plugin, Xephyr/chroot plugin
+*Debug plugins:GDB plugin, oprofile plugin, valgrind plugin
+|<font color="blue">P1</font>
 |-
 |colspan="2"|MADDE || <font color="blue">P1</font>
@@ Line 77: / Line 94: @@
 * CLI QT
 * GUI QT
-* MTF QT
+* MTF QT (handset only)
 * WRT
 ||
@@ Line 85: / Line 102: @@
 * TBD
 |-
-|Documents & Toturials ||'''need detail checklist here'''|| <font color="blue">P1</font>
+|colspan="2"|Documents & Toturials || <font color="blue">P1</font>
 |-
 |colspan="2"|Additional tools || TBD
@@ Line 97: / Line 114: @@
 * Not test for SDK package or image building, only check the built out package or images.
 * Not cover Meego compliance test, but only check key package versions which should align with Meego release.
-* ARM support are not tracked in this document yet, which should be owned by Nokia.
+* 64bit support is in plan, will be 2nd priority for testing.
+* VirtualBox support is still under investiagtion, will be taken into consideration for next step.
 == Test Strategy and Approach ==
@@ Line 123: / Line 141: @@
 === Bug life cycle ===
-Refer to [http://moblin.intel.com/wiki/MeeGo_Bugzilla_and_Bug_Triage MeeGo bugzilla guide]
+*Bug life cycle follow Meego general definition, refer to [http://moblin.intel.com/wiki/MeeGo_Bugzilla_and_Bug_Triage MeeGo bugzilla guide]
+*SDK has customized bugzilla to meet project characters, refer to [http://wiki.meego.com/SDK/Bugzilla_Components SDK Bugzilla Components]
 === Test Report ===
@@ Line 145: / Line 164: @@
 #* Netbook -- AspireOne NAV50
 #* Handset -- Aava
-#* Tablet -- TBD
+#* ARM based Devices-- N900
 === Peripherals required ===
@@ Line 157: / Line 176: @@
 === Network Environment ===
 The networking arrangements needed to conduct the testing:
-* LAN
+*WLAN
+*LAN
+*USB
 == References ==