MeeGo wiki - User contributions [en]

User:Edmondas

2011-09-23T12:27:50Z

Edmondas: /* About me */

=== About me ===
MeeGo SDK bugs analysis & fixes, workarounds, developers support on #meego-sdk and mailinglist.
You can find me on freenode as eg81.

=== Pages created by me ===
[[User:Edmondas/Building ARM sysroot|Building ARM sysroot]]

[[SDK/Sysroot_extension]]

[[User:Edmondas/Meego_SDK_on_Ubuntu_11.04|Meego SDK on Ubuntu 11.04]]

=== MeeGo SDK ===
[[SDKBugTriageMinutesArchive|SDK bug triages]]

[https://bugs.meego.com/buglist.cgi?query_format=advanced&emailtype2=substring&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&email2=ext-edmondas.girkantas%40nokia.com&emailassigned_to2=1 Bugs assigned to me]

[https://bugs.meego.com/show_bug.cgi?id=14245 METABUG: Bugs assigned to @nokia.com]

=== OBS ===
[[Build_Infrastructure/Sysadmin_Distro/OBS_setup_openSUSE114|OBS setup on OpenSuse 11.4]]

[[User:Stskeeps/10_easy_steps_to_a_local_OBS|10 easy steps to a local OBS]]

User:Edmondas

2011-09-13T11:38:34Z

Edmondas: /* MeeGo SDK */

=== About me ===
MeeGo SDK bugs analysis & fixes, workarounds, developers support on #meego-sdk and mailinglist.

=== Pages created by me ===
[[User:Edmondas/Building ARM sysroot|Building ARM sysroot]]

[[SDK/Sysroot_extension]]

[[User:Edmondas/Meego_SDK_on_Ubuntu_11.04|Meego SDK on Ubuntu 11.04]]

=== MeeGo SDK ===
[[SDKBugTriageMinutesArchive|SDK bug triages]]

[https://bugs.meego.com/buglist.cgi?query_format=advanced&emailtype2=substring&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&email2=ext-edmondas.girkantas%40nokia.com&emailassigned_to2=1 Bugs assigned to me]

[https://bugs.meego.com/show_bug.cgi?id=14245 METABUG: Bugs assigned to @nokia.com]

=== OBS ===
[[Build_Infrastructure/Sysadmin_Distro/OBS_setup_openSUSE114|OBS setup on OpenSuse 11.4]]

[[User:Stskeeps/10_easy_steps_to_a_local_OBS|10 easy steps to a local OBS]]

User:Edmondas

2011-09-13T11:38:12Z

Edmondas: /* MeeGo SDK */

=== About me ===
MeeGo SDK bugs analysis & fixes, workarounds, developers support on #meego-sdk and mailinglist.

=== Pages created by me ===
[[User:Edmondas/Building ARM sysroot|Building ARM sysroot]]

[[SDK/Sysroot_extension]]

[[User:Edmondas/Meego_SDK_on_Ubuntu_11.04|Meego SDK on Ubuntu 11.04]]

=== MeeGo SDK ===
[[SDKBugTriageMinutesArchive|SDK bug triages]]

[https://bugs.meego.com/buglist.cgi?query_format=advanced&emailtype2=substring&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&email2=ext-edmondas.girkantas%40nokia.com&emailassigned_to2=1 Bugs assigned to me]

[https://bugs.meego.com/show_bug.cgi?id=14245|METABUG: bugs assigned to @nokia.com]

=== OBS ===
[[Build_Infrastructure/Sysadmin_Distro/OBS_setup_openSUSE114|OBS setup on OpenSuse 11.4]]

[[User:Stskeeps/10_easy_steps_to_a_local_OBS|10 easy steps to a local OBS]]

User:Edmondas

2011-09-13T11:36:33Z

Edmondas: /* Pages created by me */

=== About me ===
MeeGo SDK bugs analysis & fixes, workarounds, developers support on #meego-sdk and mailinglist.

=== Pages created by me ===
[[User:Edmondas/Building ARM sysroot|Building ARM sysroot]]

[[SDK/Sysroot_extension]]

[[User:Edmondas/Meego_SDK_on_Ubuntu_11.04|Meego SDK on Ubuntu 11.04]]

=== MeeGo SDK ===
[[SDKBugTriageMinutesArchive|SDK bug triages]]

[https://bugs.meego.com/buglist.cgi?query_format=advanced&emailtype2=substring&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&email2=ext-edmondas.girkantas%40nokia.com&emailassigned_to2=1 Bugs assigned to me]

=== OBS ===
[[Build_Infrastructure/Sysadmin_Distro/OBS_setup_openSUSE114|OBS setup on OpenSuse 11.4]]

[[User:Stskeeps/10_easy_steps_to_a_local_OBS|10 easy steps to a local OBS]]

User:Edmondas/Meego SDK on Ubuntu 11.04

2011-09-13T11:35:37Z

Edmondas: added initial version

'''Note:''' the latest installer has support for Ubuntu 11.04 (get it from here http://wiki.meego.com/SDK/1.2/Installer_Status)

Some notes how you can install MeeGo SDK 1.2 on Ubuntu 11.04:

1. Download MeeGo SDK installer:
wget http://sdk.meego.com/installer/Linux/meego-sdk-installer-online-20110519
chmod u+x meego-sdk-installer-online-20110519

2. Start installation process and wait till you will get "Error during installation" message, do not close this window:
./meego-sdk-installer-online-20110519

3. Open terminal and fix MeeGo repository definition, replace "Invalid" with "Ubuntu/10.10" in /etc/apt/sources.list.d/meego-sdk.list file, e.g.:
sudo sed -i 's/Invalid/ubuntu\/10.10/' /etc/apt/sources.list.d/meego-sdk.list

4. Add repository key:
gpg --recv 0BC7BEC479FC1F8A
gpg --export --armor 0BC7BEC479FC1F8A | sudo apt-key add -

5. Update the packages database:
sudo apt-get update

6. Press "retry" button, then installation process will continue.

User:Edmondas

2011-09-13T11:07:43Z

Edmondas: /* Other useful pages */

=== About me ===
MeeGo SDK bugs analysis & fixes, workarounds, developers support on #meego-sdk and mailinglist.

=== Pages created by me ===
[[User:Edmondas/Building ARM sysroot|Building ARM sysroot]]

[[SDK/Sysroot_extension]]

=== MeeGo SDK ===
[[SDKBugTriageMinutesArchive|SDK bug triages]]

[https://bugs.meego.com/buglist.cgi?query_format=advanced&emailtype2=substring&bug_status=NEW&bug_status=NEEDINFO&bug_status=ASSIGNED&bug_status=WAITING%20FOR%20UPSTREAM&bug_status=REOPENED&email2=ext-edmondas.girkantas%40nokia.com&emailassigned_to2=1 Bugs assigned to me]

=== OBS ===
[[Build_Infrastructure/Sysadmin_Distro/OBS_setup_openSUSE114|OBS setup on OpenSuse 11.4]]

[[User:Stskeeps/10_easy_steps_to_a_local_OBS|10 easy steps to a local OBS]]

User:Edmondas

2011-09-13T07:18:09Z

Edmondas: added new link

=== About me ===
MeeGo SDK bugs analysis & fixes, workarounds, developers support on #meego-sdk and mailinglist.

=== Pages created by me ===
[[User:Edmondas/Building ARM sysroot|Building ARM sysroot]]

[[SDK/Sysroot_extension]]

=== Other useful pages ===
[[Build_Infrastructure/Sysadmin_Distro/OBS_setup_openSUSE114|OBS setup on OpenSuse 11.4]]

[[User:Stskeeps/10_easy_steps_to_a_local_OBS|10 easy steps to a local OBS]]

User:Edmondas

2011-09-13T07:16:58Z

Edmondas: minor changes, added new links

=== About me ===
MeeGo SDK bugs analysis & fixes, workarounds, developers support on #meego-sdk and mailinglist.

=== Pages created by me ===
[[User:Edmondas/Building ARM sysroot|Building ARM sysroot]]

[[SDK/Sysroot_extension]]

=== Other useful pages ===
[[User:Stskeeps/10_easy_steps_to_a_local_OBS|10 easy steps to a local OBS]]

User:Edmondas

2011-09-13T07:02:52Z

Edmondas: added link to new howto

MeeGo SDK bugs analysis & fixes, workarounds, developer support on #meego-sdk and mailinglist

[[User:Edmondas/Building ARM sysroot|Building ARM sysroot]]

User:Edmondas/Building ARM sysroot

2011-09-13T06:54:42Z

Edmondas: added note about system used during testing

'''Building ARM sysroot'''

These steps were tested on fresh Ubuntu 11.04 system.

1. Add bellow line to /etc/apt/sources.list.d/meego-sdk.list
deb http://repo.meego.com/MeeGo/tools/repos/ubuntu/10.10/ /

Note: if needed import MeeGo GPG key:
gpg --keyserver pgpkeys.mit.edu --recv 0BC7BEC479FC1F8A
gpg --export --armor 0BC7BEC479FC1F8A | sudo apt-key add -

2. Update available packages
sudo apt-get update

3. Install required tools
sudo apt-get install mic2 btrfs-tools qemu-arm-static

Note: On Ubuntu 11.04 install qemu-kvm-extras-static instead of
qemu-arm-static package

4. Get latest available kickstart file to create ARM sysroot from http://repo.meego.com/MeeGo/builds/1.2.0.90/latest/builddata/image-configs/, e.g.:
wget http://repo.meego.com/MeeGo/builds/1.2.0.90/1.2.0.90.12.20110809.2/builddata/image-configs/handset-armv7hl-madde-sysroot.ks

5. Try to build tarball with ARM sysroot:
sudo mic-image-creator --format=fs --compress-disk-image=none --package=tar.bz2 --arch=armv7hl --save-kernel --config=handset-armv7hl-madde-sysroot.ks --release=1.2.0.90.12.20110809.2

'''Troubleshooting:'''

Add parameters --verbose --debug if it will fail for any reason

If you are behind a proxy, you can set it to mic by adding
proxy=http://proxy.yourcompany.com:8080
to /etc/mic2/mic2.conf.

If installation of packages fails for some reason then change package manager to yum:
--pkgmgr=yum

Disable vdso:
echo 0 | sudo tee /proc/sys/vm/vdso_enabled

If something goes wrong try to remove old meego bootstrap:
rm -rf /var/cache/meego-bootstrap

User:Edmondas/Building ARM sysroot

2011-09-13T06:53:49Z

Edmondas: fixed typo

'''Building ARM sysroot'''

1. Add bellow line to /etc/apt/sources.list.d/meego-sdk.list
deb http://repo.meego.com/MeeGo/tools/repos/ubuntu/10.10/ /

Note: if needed import MeeGo GPG key:
gpg --keyserver pgpkeys.mit.edu --recv 0BC7BEC479FC1F8A
gpg --export --armor 0BC7BEC479FC1F8A | sudo apt-key add -

2. Update available packages
sudo apt-get update

3. Install required tools
sudo apt-get install mic2 btrfs-tools qemu-arm-static

Note: On Ubuntu 11.04 install qemu-kvm-extras-static instead of
qemu-arm-static package

4. Get latest available kickstart file to create ARM sysroot from http://repo.meego.com/MeeGo/builds/1.2.0.90/latest/builddata/image-configs/, e.g.:
wget http://repo.meego.com/MeeGo/builds/1.2.0.90/1.2.0.90.12.20110809.2/builddata/image-configs/handset-armv7hl-madde-sysroot.ks

5. Try to build tarball with ARM sysroot:
sudo mic-image-creator --format=fs --compress-disk-image=none --package=tar.bz2 --arch=armv7hl --save-kernel --config=handset-armv7hl-madde-sysroot.ks --release=1.2.0.90.12.20110809.2

'''Troubleshooting:'''

Add parameters --verbose --debug if it will fail for any reason

If you are behind a proxy, you can set it to mic by adding
proxy=http://proxy.yourcompany.com:8080
to /etc/mic2/mic2.conf.

If installation of packages fails for some reason then change package manager to yum:
--pkgmgr=yum

Disable vdso:
echo 0 | sudo tee /proc/sys/vm/vdso_enabled

If something goes wrong try to remove old meego bootstrap:
rm -rf /var/cache/meego-bootstrap

User:Edmondas/Building ARM sysroot

2011-09-13T06:53:19Z

Edmondas: removed notice about draft version

'''Building ARM sysroot'''

1. Add bellow line to /etc/apt/sources.list.d/meego-sdk.list
deb http://repo.meego.com/MeeGo/tools/repos/ubuntu/10.10/ /

Not: if needed import MeeGo GPG key:
gpg --keyserver pgpkeys.mit.edu --recv 0BC7BEC479FC1F8A
gpg --export --armor 0BC7BEC479FC1F8A | sudo apt-key add -

2. Update available packages
sudo apt-get update

3. Install required tools
sudo apt-get install mic2 btrfs-tools qemu-arm-static

Note: On Ubuntu 11.04 install qemu-kvm-extras-static instead of
qemu-arm-static package

4. Get latest available kickstart file to create ARM sysroot from http://repo.meego.com/MeeGo/builds/1.2.0.90/latest/builddata/image-configs/, e.g.:
wget http://repo.meego.com/MeeGo/builds/1.2.0.90/1.2.0.90.12.20110809.2/builddata/image-configs/handset-armv7hl-madde-sysroot.ks

5. Try to build tarball with ARM sysroot:
sudo mic-image-creator --format=fs --compress-disk-image=none --package=tar.bz2 --arch=armv7hl --save-kernel --config=handset-armv7hl-madde-sysroot.ks --release=1.2.0.90.12.20110809.2

'''Troubleshooting:'''

Add parameters --verbose --debug if it will fail for any reason

If you are behind a proxy, you can set it to mic by adding
proxy=http://proxy.yourcompany.com:8080
to /etc/mic2/mic2.conf.

If installation of packages fails for some reason then change package manager to yum:
--pkgmgr=yum

Disable vdso:
echo 0 | sudo tee /proc/sys/vm/vdso_enabled

If something goes wrong try to remove old meego bootstrap:
rm -rf /var/cache/meego-bootstrap

User:Edmondas/Building ARM sysroot

2011-09-13T06:52:16Z

Edmondas: minor fixes to style

'''this is draft version!'''

1. Add bellow line to /etc/apt/sources.list.d/meego-sdk.list
deb http://repo.meego.com/MeeGo/tools/repos/ubuntu/10.10/ /

Not: if needed import MeeGo GPG key:
gpg --keyserver pgpkeys.mit.edu --recv 0BC7BEC479FC1F8A
gpg --export --armor 0BC7BEC479FC1F8A | sudo apt-key add -

2. Update available packages
sudo apt-get update

3. Install required tools
sudo apt-get install mic2 btrfs-tools qemu-arm-static

Note: On Ubuntu 11.04 install qemu-kvm-extras-static instead of
qemu-arm-static package

4. Get latest available kickstart file to create ARM sysroot from http://repo.meego.com/MeeGo/builds/1.2.0.90/latest/builddata/image-configs/, e.g.:
wget http://repo.meego.com/MeeGo/builds/1.2.0.90/1.2.0.90.12.20110809.2/builddata/image-configs/handset-armv7hl-madde-sysroot.ks

5. Try to build tarball with ARM sysroot:
sudo mic-image-creator --format=fs --compress-disk-image=none --package=tar.bz2 --arch=armv7hl --save-kernel --config=handset-armv7hl-madde-sysroot.ks --release=1.2.0.90.12.20110809.2

'''Troubleshooting:'''

Add parameters --verbose --debug if it will fail for any reason

If you are behind a proxy, you can set it to mic by adding
proxy=http://proxy.yourcompany.com:8080
to /etc/mic2/mic2.conf.

If installation of packages fails for some reason then change package manager to yum:
--pkgmgr=yum

Disable vdso:
echo 0 | sudo tee /proc/sys/vm/vdso_enabled

If something goes wrong try to remove old meego bootstrap:
rm -rf /var/cache/meego-bootstrap

User:Edmondas

2011-09-12T15:20:10Z

Edmondas: updated my info

MeeGo SDK bugs analysis & fixes, workarounds, developer support on #meego-sdk and mailinglist

User:Edmondas

2011-09-12T15:19:20Z

Edmondas: added info about myself

MeeGo SDK bug fixer

User:Edmondas/Building ARM sysroot

2011-09-12T15:17:30Z

Edmondas: added initial version of ARM sysroot building howto

'''this is draft version!'''

1. Add bellow line to /etc/apt/sources.list.d/meego-sdk.list
deb http://repo.meego.com/MeeGo/tools/repos/ubuntu/10.10/ /

Not: if needed import required GPG key:
gpg --keyserver pgpkeys.mit.edu --recv 0BC7BEC479FC1F8A
gpg --export --armor 0BC7BEC479FC1F8A | sudo apt-key add -

2. Update available packages
sudo apt-get update

3. Install required tools
sudo apt-get install mic2 btrfs-tools qemu-arm-static

Note: On Ubuntu 11.04 install qemu-kvm-extras-static instead of
qemu-arm-static package

4. Get latest available kickstart file for arm sysroot creation, e.g.:
wget http://repo.meego.com/MeeGo/builds/1.2.0.90/latest/builddata/image-configs/handset-armv7hl-madde-sysroot.ks

5. Try to build sysroot tarball:
sudo mic-image-creator --format=fs --compress-disk-image=none --package=tar.bz2 --arch=armv7hl --save-kernel --config=handset-armv7hl-madde-sysroot.ks --release=1.2.0.90.12.20110809.2

Troubleshooting:
Add parameters --verbose --debug if it will fail for any reason

If you are behind a proxy, you can set it to mic by adding
proxy=http://proxy.yourcompany.com:8080
to /etc/mic2/mic2.conf.

If installation of packages fails for some reason then change package manager to yum:
--pkgmgr=yum

Disable vdso:
echo 0 | sudo tee /proc/sys/vm/vdso_enabled

If something goes wrong try to remove old meego bootstrap:
rm -rf /var/cache/meego-bootstrap

SDKBugTriageMinutesArchive

2011-07-27T07:38:14Z

Edmondas: added log from latest meeting

*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-07-27-05.55.html MeeGo SDK Bug Triage 20110727]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-07-20-05.55.html MeeGo SDK Bug Triage 20110720]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-07-06-05.57.html MeeGo SDK Bug Triage 20110706]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-06-29-06.00.html MeeGo SDK Bug Triage 20110629]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-06-22-05.57.html MeeGo SDK Bug Triage 20110622]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-06-15-05.58.html MeeGo SDK Bug Triage 20110615]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-06-08-05.57.html MeeGo SDK Bug Triage 20110608]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-05-25-05.58.html MeeGo SDK Bug Triage 20110525]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-05-18-05.58.html MeeGo SDK Bug Triage 20110518]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-05-11-05.58.html MeeGo SDK Bug Triage 20110511]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-05-04-05.59.html MeeGo SDK Bug Triage 20110504]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-04-27-05.59.html MeeGo SDK Bug Triage 20110427]

SDK/Meetings/SDK Weekly

2011-07-18T11:50:51Z

Edmondas: /* Attendees */

= MeeGo SDK Weekly Meeting Info =

MeeGo SDK weekly meeting is held every Tuesday at 7:00 am PDT. Duration is 1 hour. Meeting is for syncing up between different parties and agreeing on actions to follow.

Agenda and minutes of individual meetings can be found from [[SDK/Meetings]].

== Attendees ==

* [http://meego.com/users/veli Veli Kaksonen] (veli) - Project Manager
* Marianne Vaiciulis - Program Manager
* Ville Lavonius - The Man
* * Al Nikolov (alnikolov) - Windows & Mac port + Toolchain packaging
* Jaya Uppalapati (?) - QA / Testing
* Maurice Kalinowski (mauricek) - QtCreator
* Bob Spencer (bspencer) - Architect
* Jacki Wu - SDK Engineering Manager
* Kevin Wang (KevinWang) - Packaging and Integration
* John Chen (?) - Integration with Qt Creator
* Ionut Gavaz - SDK QA
* Jan-Simon Möller (dl9pf_) - OBS cross-compilation integration & support
* Edmondas Girkantas (eg81) - MeeGo SDK bugs analysis & fixes, workarounds, developer support on #meego-sdk and mailinglist

== Default Agenda ==

1. Meeting Objective Review (5 min)
2. Team Lead Report Out (30 min)
3. Architecture Update (25 min)

SDK/Meetings/SDK Weekly

2011-07-18T11:41:57Z

Edmondas: /* Attendees */

= MeeGo SDK Weekly Meeting Info =

MeeGo SDK weekly meeting is held every Tuesday at 7:00 am PDT. Duration is 1 hour. Meeting is for syncing up between different parties and agreeing on actions to follow.

Agenda and minutes of individual meetings can be found from [[SDK/Meetings]].

== Attendees ==

* [http://meego.com/users/veli Veli Kaksonen] (veli) - Project Manager
* Marianne Vaiciulis - Program Manager
* Ville Lavonius - The Man
* * Al Nikolov (alnikolov) - Windows & Mac port + Toolchain packaging
* Jaya Uppalapati (?) - QA / Testing
* Maurice Kalinowski (mauricek) - QtCreator
* Bob Spencer (bspencer) - Architect
* Jacki Wu - SDK Engineering Manager
* Kevin Wang (KevinWang) - Packaging and Integration
* John Chen (?) - Integration with Qt Creator
* Ionut Gavaz - SDK QA
* Jan-Simon Möller (dl9pf_) - OBS cross-compilation integration & support
* Edmondas Girkantas (eg81) - ?

== Default Agenda ==

1. Meeting Objective Review (5 min)
2. Team Lead Report Out (30 min)
3. Architecture Update (25 min)

ARM/N900/Install/MMC

2011-07-14T08:56:27Z

Edmondas: /* Notes about flashing the kernel */

= Installing MeeGo to N900 on external MMC card =

'''NOTE: Read the whole guide before doing any steps in this guide to make sure you understand everything.'''

If you do not want to erase the NAND partition or the internal 32 G eMMC from the N900 device, this installation is for you. What you need for this installation is a raw image that can be put to the MMC card and kernel (same kernel that is installed to the MMC).

=== Images ===

Raw images and corresponding kernel images can be found at http://repo.meego.com/MeeGo/builds/ (development builds) and http://repo.meego.com/MeeGo/releases/ (stable releases)

Find official release images here ([[ARM/N900#Releases]])

You may want to check for the most recent N900 test reports at [[Quality#MeeGo_Handset_Testing]] to see how much functionality is known to work on the N900.

== Installing Rootfs on external MMC card ==

First, what you need, is a microSD memory card which does not contain any information that you need, as it will be erased during this operation.

The steps in this guide require an MMC card of at least 4gb.

=== Linux ===

When inserting the microSD memory card in the card reader, you need to find out what the proper device for the card is. You can also plug the N900 into your computer's USB slot, using the package-provided cable. Make sure the external microSD card is unmounted, as with most modern linux distributions today it will get auto-mounted if the has a valid filesystem (FAT32 or ExtX). To unmount, you can try this example:
sudo umount /dev/sdX

And use the mount command to determine if/ where it is mounted:
sudo mount (for an example how the microSD would look go [[MountOutput here]].

Finding out the device node can also be done with, for example, fdisk:
sudo fdisk -l

An example output ('''NOTE: The /dev/sdX is used as an example on your PC. This might be also called /dev/mmcblk0, /dev/sdd or something else''')
$ sudo fdisk -l
...
Disk /dev/sdX: 3965 MB, 3965714432 bytes
194 heads, 30 sectors/track, 1330 cylinders
Units = cylinders of 5820 * 512 = 2979840 bytes
Disk identifier: 0x0001ab40

Device Boot Start End Blocks Id System
/dev/sdX1 1 588 1708984 83 Linux

'''NOTE: The .raw image contains the partition table, as well. So the image needs to be written to /dev/sdX not /dev/sdX1.'''

After you are 100% sure that the /dev/sdX is the microSD memory card you just inserted in the card reader, you can use, for example, dd to put the image onto the card:
$ sudo dd bs=4096 if='''<raw_image>''' of=/dev/sdX

If you are low on disk space, you can use

$ bzcat <raw_image>.bz2 | sudo dd bs=4096 of=/dev/sdX

to decompress the compressed raw image on the fly, without having to unpack it on you computer first. And if you have pv(1) installed, you can add it in between to display the progress (the image is ~ 2GB in size as of 2010-10-04):

$ bzcat <raw_image>.bz2 | pv | sudo dd bs=4096 of=/dev/sdX

The dd does not show any progress until the file is written to the device, so be patient.

Although sending a USR1 signal to a running dd process makes it print I/O statistics to standard error and then resume copying:
$ dd if=/dev/zero of=/dev/null& pid=$!
from another term:
$ kill -USR1 $pid

dd will output:

328356+0 records in
328356+0 records out
1344946176 bytes (1.3 GB) copied, 105.625 s, 12.7 MB/s

After this, you can insert the card in the N900.

=== Windows ===

# Download and install bzip2 to Windows http://gnuwin32.sourceforge.net/packages/bzip2.htm
# Uncompress raw image in command prompt "bunzip2.exe <compressed raw image>"
# Change (rename) extracted file extension from .raw to .img
# Download the Win32DiskImager.exe program: https://launchpad.net/win32-image-writer/+download (zip file)
# Unzip the file and extract the contents to a known directory
# Run W32DiskImager.exe
# Select the MeeGo image file
# Select the drive letter which corresponds to the microSD memory card.
# Click the "Write" button to byte-copy the image to the microSD memory card.
# insert the memory card into the device and attach the back cover
# load/flash kernel image

==== Notes about flashing the kernel ====

Get the N900's flasher from [http://tablets-dev.nokia.com/maemo-dev-env-downloads.php tablets-dev.nokia.com]

To load the kernel:
flasher-3.5.exe -l -b -k [kernel image] to load it

To flash it
flasher-3.5.exe -f -b -k [kernel image] to flash it

Flashing replace the Maemo5 kernel and you won't be able to boot into maemo5 without flashing the original kernel back

Then connect N900 to the computer via the USB cable and the Meego OS will boot. You may need to keep the 'U' key pressed (on the phone's keyboard) when connecting the cable.

When booting from a memory card make sure that the device's back cover is on. Otherwise the card will not mount and you'll see a kernel panic

''''''NOTE!!''' If you get error 'Not enough space on disk'. Try again with external memorycard reader.'''

=== Mac OS X ===

When inserting the microSD memory card in the card reader, you need to find out what the proper device for the card is. This can be done with disktool:
diskutil list

An example output ('''NOTE: The /dev/diskX is used as an example on your Mac. This might be also called /dev/disk2, /dev/disk3 or something else''')
$ diskutil list
...
/dev/diskX
#: TYPE NAME SIZE IDENTIFIER
0: FDisk_partition_scheme *4.1 GB disk3
1: DOS_FAT_32 DISKETTE 4.1 GB disk3s1

'''NOTE: The .raw image contains the partition table, as well. So the image needs to be written to /dev/diskX not /dev/diskX1.'''

'''NOTE: If you want to improve the speed writing your mmc card, you can use the raw device, avoiding the use of the os buffer, add 'r' add the name of your mmc interface, example: /dev/rdiskX'''

After you are 100% sure that the /dev/diskX is the microSD memory card you just inserted in the card reader, you can use, for example, dd to put the image onto the card:
$ sudo dd bs=4096 if='''<raw_image>''' of=/dev/diskX

If you are low on disk space, you can use

$ bzcat <raw_image>.bz2 | sudo dd bs=4096 of=/dev/diskX

to decompress the compressed raw image on the fly without having to unpack it on you computer first.

The dd does not show any progress until the file is written to the device, so be patient.

Although sending a SIGINFO signal to a running dd process makes it print I/O statistics to standard error and then resume copying:
$ dd if=/dev/zero of=/dev/null& pid=$!
from another term:
$ kill -SIGINFO $pid

After dd is done, dd will output something like this:

475136+1 records in
475136+1 records out
1946157057 bytes transferred in 2606.033611 secs (746789 bytes/sec)

After this, you can eject the microSD and insert the card in the N900.

=== On the N900 itself ===

This should be a last resort solution.

# Download and unpack the raw image to a desktop computer (see above).
# un-mount the microSD memory card
sudo gainroot
umount /dev/mmcblkXpY
dd if=/home/user/MyDocs/'''<raw_image>''' of=/dev/mmcblk1

==== Notes ====
* X, Y in umount command should be the device number and partition number, usually 1, 1. '''Do not take information from /proc/partitions!'''
N900 swaps devices during boot and /proc/partitions is what the kernel initially sees when it is loaded. N900's SD-Card is /dev/mmcblk1 for Maemo5-Standard-Install and not like shown in /proc/partitions, /dev/mmcblk0! NITdroid's SD-install has SD-Card as /dev/mmcblk0, instead. If you want to see what is mounted at the moment, type "mount|grep mmc" or "df|grep mmc".
* There might be more than one partition on the microSD memory card, though it is unlikely and depends on your own setup. You need to unmount all partitions before you proceed to `dd`.

* It is possible to download the compressed image to your N900, but it is not recommended as it takes a long time to unpack.
* `dd` does not give any output while it is copying, so be patient.
* If you aren't sure about any of these steps you should not proceed without consulting a professional.

== Load or flash kernel on N900 ==

Before MeeGo is able to boot, you also need to load the kernel (vmlinuz) provided with the raw image to the device. This can be done with the [[ARM/N900/tools/flasher|flasher]].

=== Dual-boot with maemo using u-boot ===

See [[ARM/N900/Install/Dual_Boot]] instead.

=== Load or flash manually ===
====Prerequisites====

* the flasher application needs to be installed on your computer. You can get it from [http://tablets-dev.nokia.com/maemo-dev-env-downloads.php here].
On 64-bit Ubuntu/Debian this command would do the trick, after downloading the .deb file:

$ sudo dpkg --force-architecture -i maemo_flasher-*i386.deb

====Running Meego====

'''NOTE: First, open the back cover of N900 and insert the MMC card to the slot. Reinstall the back cover again.

'''NOTE: Back cover must be closed to boot from MMC.'''

'''NOTE: If you have a flashing jig, you will need to put a magnet at the red location marked [http://www.daimi.au.dk/~cvm/magnet.png here]'''

You need to execute the following command on your host system.

'''NOTE: The device must be turned off and disconnected from the computer. Connect it to the computer using the USB cable, only after executing the command'''

$ sudo flasher-3.5 -l -k '''<kernel>''' -b

A message like:
flasher v2.5.2 (Oct 21 2009)
Suitable USB device not found, waiting.
is shown on the terminal of the computer.

Then connect N900 to the computer via the USB cable and the Meego OS will boot.
You may need to keep the 'U' key pressed (on the phone's keyboard) when connecting the cable.

'''NOTE: The command above will only load the kernel to the device, so next time you boot the device, the original kernel should be used and your Maemo 5 OS should boot normally.'''

If you want to flash the kernel to your device so that it is not forgotten when it is powered off, use option -f instead of option -l on command above.

'''NOTE: It's highly recommended NOT to flash the kernel on your device, unless you really know what you're doing (so please use -l instead of -f). Don't blame us if you brick your device, you have been warned!'''

{| class="wikitable" border="1" cellpadding="5"
|+ Examples of files that can be used with this guide
! MeeGo Version
! <kernel>
! <raw_image>
|-
| 1.0.99.2.20101019.1
| [http://repo.meego.com/MeeGo/builds/1.0.99/1.0.99.2.20101019.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.0.99.2.20101019.1-vmlinuz-2.6.35.3-10.3-n900 meego-handset-armv7l-n900-1.0.99.2.20101019.1-vmlinuz-2.6.35.3-10.3-n900]
| [http://repo.meego.com/MeeGo/builds/1.0.99/1.0.99.2.20101019.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.0.99.2.20101019.1-mmcblk0p.raw.bz2 meego-handset-armv7l-n900-1.0.99.2.20101019.1-mmcblk0p.raw.bz2]
|-
| 1.1.80.0.20101001.1
| [http://repo.meego.com/MeeGo/builds/trunk/1.1.80.0.20101001.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.1.80.0.20101001.1-vmlinuz-2.6.35.3-8.5-n900 meego-handset-armv7l-n900-1.1.80.0.20101001.1-vmlinuz-2.6.35.3-8.5-n900]
| [http://repo.meego.com/MeeGo/builds/trunk/1.1.80.0.20101001.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.1.80.0.20101001.1-mmcblk0p.raw.bz2 meego-handset-armv7l-n900-1.1.80.0.20101001.1-mmcblk0p.raw.bz2]
|}

[[Category:N900]]

SDK/Docs/1.2/MeeGo SDK 1.2 Preview for Linux

2011-07-08T10:44:03Z

Edmondas: /* Download the MeeGo SDK */

== Download the MeeGo SDK ==
<ol>
<li>'''Download and install the SDK'''

{| class="wikitable"
|+ Download information
|-
! File !! File Size !! MD5SUM !! Installed Size
|-
| [http://sdk.meego.com/installer/Linux/meego-sdk-installer-online-20110519 MeeGo SDK 1.2] || 26MB || 76d0442e060628b6aa696470a254dedc || 5.2 GB
|}

$ chmod +x ./meego-sdk-installer-online
$ ./meego-sdk-installer-online --verbose

Note that --verbose is optional. 
SDK content details are described on the [[SDK/Docs/1.2/MeeGo SDK 1.2 Components|SDK Component Summary]] page.
</li>
<li>'''Select the components that will be installed. Mainly select the target for development. During downloading, once one of components failed to download due to the connection issue, just click 'Retry' button in the notification dialog to resume.'''</li>
[[File:Installer-select component.png]]
</ol>

== Install a MeeGo Target ==
A ''Target'' is used to build programs using the MeeGo toolchain and libraries (sysroot). Except you can install target in the process of SDK, you also can install target by launching Maintain MeeGo SDK.
'''MeeGo SDK 1.2 Preview > Maintain MeeGo SDk''' . Select targets from SDK targest and click next. It will install targets you selected 
[[File:Select target.png]]

 

'''Next Step:[[SDK/Docs/1.2/MeeGo_SDK_1.2_Preview_for_Linux/Configure_Qt_Creator_to_Develop_MeeGo_Applications]]'''

[[Category:SDK]]
[[Category:Meego-1.2]]
[[Category:Linux]]

ARM/N900/Install/MMC

2011-06-29T12:13:33Z

Edmondas: /* Notes about flashing the kernel */

= Installing MeeGo to N900 on external MMC card =

'''NOTE: Read the whole guide before doing any steps in this guide to make sure you understand everything.'''

If you do not want to erase the NAND partition or the internal 32 G eMMC from the N900 device, this installation is for you. What you need for this installation is a raw image that can be put to the MMC card and kernel (same kernel that is installed to the MMC).

=== Images ===

Raw images and corresponding kernel images can be found at http://repo.meego.com/MeeGo/builds/ (development builds) and http://repo.meego.com/MeeGo/releases/ (stable releases)

Find official release images here ([[ARM/N900#Releases]])

You may want to check for the most recent N900 test reports at [[Quality#MeeGo_Handset_Testing]] to see how much functionality is known to work on the N900.

== Installing Rootfs on external MMC card ==

First, what you need, is a microSD memory card which does not contain any information that you need, as it will be erased during this operation.

The steps in this guide require an MMC card of at least 4gb.

=== Linux ===

When inserting the microSD memory card in the card reader, you need to find out what the proper device for the card is. You can also plug the N900 into your computer's USB slot, using the package-provided cable. Make sure the external microSD card is unmounted, as with most modern linux distributions today it will get auto-mounted if the has a valid filesystem (FAT32 or ExtX). To unmount, you can try this example:
sudo umount /dev/sdX

And use the mount command to determine if/ where it is mounted:
sudo mount (for an example how the microSD would look go [[MountOutput here]].

Finding out the device node can also be done with, for example, fdisk:
sudo fdisk -l

An example output ('''NOTE: The /dev/sdX is used as an example on your PC. This might be also called /dev/mmcblk0, /dev/sdd or something else''')
$ sudo fdisk -l
...
Disk /dev/sdX: 3965 MB, 3965714432 bytes
194 heads, 30 sectors/track, 1330 cylinders
Units = cylinders of 5820 * 512 = 2979840 bytes
Disk identifier: 0x0001ab40

Device Boot Start End Blocks Id System
/dev/sdX1 1 588 1708984 83 Linux

'''NOTE: The .raw image contains the partition table, as well. So the image needs to be written to /dev/sdX not /dev/sdX1.'''

After you are 100% sure that the /dev/sdX is the microSD memory card you just inserted in the card reader, you can use, for example, dd to put the image onto the card:
$ sudo dd bs=4096 if='''<raw_image>''' of=/dev/sdX

If you are low on disk space, you can use

$ bzcat <raw_image>.bz2 | sudo dd bs=4096 of=/dev/sdX

to decompress the compressed raw image on the fly, without having to unpack it on you computer first. And if you have pv(1) installed, you can add it in between to display the progress (the image is ~ 2GB in size as of 2010-10-04):

$ bzcat <raw_image>.bz2 | pv | sudo dd bs=4096 of=/dev/sdX

The dd does not show any progress until the file is written to the device, so be patient.

Although sending a USR1 signal to a running dd process makes it print I/O statistics to standard error and then resume copying:
$ dd if=/dev/zero of=/dev/null& pid=$!
from another term:
$ kill -USR1 $pid

dd will output:

328356+0 records in
328356+0 records out
1344946176 bytes (1.3 GB) copied, 105.625 s, 12.7 MB/s

After this, you can insert the card in the N900.

=== Windows ===

# Download and install bzip2 to Windows http://gnuwin32.sourceforge.net/packages/bzip2.htm
# Uncompress raw image in command prompt "bunzip2.exe <compressed raw image>"
# Change (rename) extracted file extension from .raw to .img
# Download the Win32DiskImager.exe program: https://launchpad.net/win32-image-writer/+download (zip file)
# Unzip the file and extract the contents to a known directory
# Run W32DiskImager.exe
# Select the MeeGo image file
# Select the drive letter which corresponds to the microSD memory card.
# Click the "Write" button to byte-copy the image to the microSD memory card.
# insert the memory card into the device and attach the back cover
# load/flash kernel image

==== Notes about flashing the kernel ====

Get the N900's flasher from [http://tablets-dev.nokia.com/maemo-dev-env-downloads.php tablets-dev.nokia.com]

To load the kernel:
flasher-3.5.exe -l -b -k [kernel image] to load it

To flash it
flasher-3.5.exe -f -b -k [kernel image] to flash it

Flashing replace the Maemo5 kernel and you won't be able to boot into maemo5 without flashing the original kernel back

When booting from a memory card make sure that the device's back cover is on. Otherwise the card will not mount and you'll see a kernel panic

''''''NOTE!!''' If you get error 'Not enough space on disk'. Try again with external memorycard reader.'''

=== Mac OS X ===

When inserting the microSD memory card in the card reader, you need to find out what the proper device for the card is. This can be done with disktool:
diskutil list

An example output ('''NOTE: The /dev/diskX is used as an example on your Mac. This might be also called /dev/disk2, /dev/disk3 or something else''')
$ diskutil list
...
/dev/diskX
#: TYPE NAME SIZE IDENTIFIER
0: FDisk_partition_scheme *4.1 GB disk3
1: DOS_FAT_32 DISKETTE 4.1 GB disk3s1

'''NOTE: The .raw image contains the partition table, as well. So the image needs to be written to /dev/diskX not /dev/diskX1.'''

'''NOTE: If you want to improve the speed writing your mmc card, you can use the raw device, avoiding the use of the os buffer, add 'r' add the name of your mmc interface, example: /dev/rdiskX'''

After you are 100% sure that the /dev/diskX is the microSD memory card you just inserted in the card reader, you can use, for example, dd to put the image onto the card:
$ sudo dd bs=4096 if='''<raw_image>''' of=/dev/diskX

If you are low on disk space, you can use

$ bzcat <raw_image>.bz2 | sudo dd bs=4096 of=/dev/diskX

to decompress the compressed raw image on the fly without having to unpack it on you computer first.

The dd does not show any progress until the file is written to the device, so be patient.

Although sending a SIGINFO signal to a running dd process makes it print I/O statistics to standard error and then resume copying:
$ dd if=/dev/zero of=/dev/null& pid=$!
from another term:
$ kill -SIGINFO $pid

After dd is done, dd will output something like this:

475136+1 records in
475136+1 records out
1946157057 bytes transferred in 2606.033611 secs (746789 bytes/sec)

After this, you can eject the microSD and insert the card in the N900.

=== On the N900 itself ===

This should be a last resort solution.

# Download and unpack the raw image to a desktop computer (see above).
# un-mount the microSD memory card
sudo gainroot
umount /dev/mmcblkXpY
dd if=/home/user/MyDocs/'''<raw_image>''' of=/dev/mmcblk1

==== Notes ====
* X, Y in umount command should be the device number and partition number, usually 1, 1. '''Do not take information from /proc/partitions!'''
N900 swaps devices during boot and /proc/partitions is what the kernel initially sees when it is loaded. N900's SD-Card is /dev/mmcblk1 for Maemo5-Standard-Install and not like shown in /proc/partitions, /dev/mmcblk0! NITdroid's SD-install has SD-Card as /dev/mmcblk0, instead. If you want to see what is mounted at the moment, type "mount|grep mmc" or "df|grep mmc".
* There might be more than one partition on the microSD memory card, though it is unlikely and depends on your own setup. You need to unmount all partitions before you proceed to `dd`.

* It is possible to download the compressed image to your N900, but it is not recommended as it takes a long time to unpack.
* `dd` does not give any output while it is copying, so be patient.
* If you aren't sure about any of these steps you should not proceed without consulting a professional.

== Load or flash kernel on N900 ==

Before MeeGo is able to boot, you also need to load the kernel (vmlinuz) provided with the raw image to the device. This can be done with the [[ARM/N900/tools/flasher|flasher]].

=== Dual-boot with maemo using u-boot ===

See [[ARM/N900/Install/Dual_Boot]] instead.

=== Load or flash manually ===
====Prerequisites====

* the flasher application needs to be installed on your computer. You can get it from [http://tablets-dev.nokia.com/maemo-dev-env-downloads.php here].
On 64-bit Ubuntu/Debian this command would do the trick, after downloading the .deb file:

$ sudo dpkg --force-architecture -i maemo_flasher-*i386.deb

====Running Meego====

'''NOTE: First, open the back cover of N900 and insert the MMC card to the slot. Reinstall the back cover again.

'''NOTE: Back cover must be closed to boot from MMC.'''

'''NOTE: If you have a flashing jig, you will need to put a magnet at the red location marked [http://www.daimi.au.dk/~cvm/magnet.png here]'''

You need to execute the following command on your host system.

'''NOTE: The device must be turned off and disconnected from the computer. Connect it to the computer using the USB cable, only after executing the command'''

$ sudo flasher-3.5 -l -k '''<kernel>''' -b

A message like:
flasher v2.5.2 (Oct 21 2009)
Suitable USB device not found, waiting.
is shown on the terminal of the computer.

Then connect N900 to the computer via the USB cable and the Meego OS will boot.
You may need to keep the 'U' key pressed (on the phone's keyboard) when connecting the cable.

'''NOTE: The command above will only load the kernel to the device, so next time you boot the device, the original kernel should be used and your Maemo 5 OS should boot normally.'''

If you want to flash the kernel to your device so that it is not forgotten when it is powered off, use option -f instead of option -l on command above.

'''NOTE: It's highly recommended NOT to flash the kernel on your device, unless you really know what you're doing (so please use -l instead of -f). Don't blame us if you brick your device, you have been warned!'''

{| class="wikitable" border="1" cellpadding="5"
|+ Examples of files that can be used with this guide
! MeeGo Version
! <kernel>
! <raw_image>
|-
| 1.0.99.2.20101019.1
| [http://repo.meego.com/MeeGo/builds/1.0.99/1.0.99.2.20101019.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.0.99.2.20101019.1-vmlinuz-2.6.35.3-10.3-n900 meego-handset-armv7l-n900-1.0.99.2.20101019.1-vmlinuz-2.6.35.3-10.3-n900]
| [http://repo.meego.com/MeeGo/builds/1.0.99/1.0.99.2.20101019.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.0.99.2.20101019.1-mmcblk0p.raw.bz2 meego-handset-armv7l-n900-1.0.99.2.20101019.1-mmcblk0p.raw.bz2]
|-
| 1.1.80.0.20101001.1
| [http://repo.meego.com/MeeGo/builds/trunk/1.1.80.0.20101001.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.1.80.0.20101001.1-vmlinuz-2.6.35.3-8.5-n900 meego-handset-armv7l-n900-1.1.80.0.20101001.1-vmlinuz-2.6.35.3-8.5-n900]
| [http://repo.meego.com/MeeGo/builds/trunk/1.1.80.0.20101001.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.1.80.0.20101001.1-mmcblk0p.raw.bz2 meego-handset-armv7l-n900-1.1.80.0.20101001.1-mmcblk0p.raw.bz2]
|}

[[Category:N900]]

ARM/N900/Install/MMC

2011-06-29T12:12:36Z

Edmondas: /* Notes about flashing the kernel */

= Installing MeeGo to N900 on external MMC card =

'''NOTE: Read the whole guide before doing any steps in this guide to make sure you understand everything.'''

If you do not want to erase the NAND partition or the internal 32 G eMMC from the N900 device, this installation is for you. What you need for this installation is a raw image that can be put to the MMC card and kernel (same kernel that is installed to the MMC).

=== Images ===

Raw images and corresponding kernel images can be found at http://repo.meego.com/MeeGo/builds/ (development builds) and http://repo.meego.com/MeeGo/releases/ (stable releases)

Find official release images here ([[ARM/N900#Releases]])

You may want to check for the most recent N900 test reports at [[Quality#MeeGo_Handset_Testing]] to see how much functionality is known to work on the N900.

== Installing Rootfs on external MMC card ==

First, what you need, is a microSD memory card which does not contain any information that you need, as it will be erased during this operation.

The steps in this guide require an MMC card of at least 4gb.

=== Linux ===

When inserting the microSD memory card in the card reader, you need to find out what the proper device for the card is. You can also plug the N900 into your computer's USB slot, using the package-provided cable. Make sure the external microSD card is unmounted, as with most modern linux distributions today it will get auto-mounted if the has a valid filesystem (FAT32 or ExtX). To unmount, you can try this example:
sudo umount /dev/sdX

And use the mount command to determine if/ where it is mounted:
sudo mount (for an example how the microSD would look go [[MountOutput here]].

Finding out the device node can also be done with, for example, fdisk:
sudo fdisk -l

An example output ('''NOTE: The /dev/sdX is used as an example on your PC. This might be also called /dev/mmcblk0, /dev/sdd or something else''')
$ sudo fdisk -l
...
Disk /dev/sdX: 3965 MB, 3965714432 bytes
194 heads, 30 sectors/track, 1330 cylinders
Units = cylinders of 5820 * 512 = 2979840 bytes
Disk identifier: 0x0001ab40

Device Boot Start End Blocks Id System
/dev/sdX1 1 588 1708984 83 Linux

'''NOTE: The .raw image contains the partition table, as well. So the image needs to be written to /dev/sdX not /dev/sdX1.'''

After you are 100% sure that the /dev/sdX is the microSD memory card you just inserted in the card reader, you can use, for example, dd to put the image onto the card:
$ sudo dd bs=4096 if='''<raw_image>''' of=/dev/sdX

If you are low on disk space, you can use

$ bzcat <raw_image>.bz2 | sudo dd bs=4096 of=/dev/sdX

to decompress the compressed raw image on the fly, without having to unpack it on you computer first. And if you have pv(1) installed, you can add it in between to display the progress (the image is ~ 2GB in size as of 2010-10-04):

$ bzcat <raw_image>.bz2 | pv | sudo dd bs=4096 of=/dev/sdX

The dd does not show any progress until the file is written to the device, so be patient.

Although sending a USR1 signal to a running dd process makes it print I/O statistics to standard error and then resume copying:
$ dd if=/dev/zero of=/dev/null& pid=$!
from another term:
$ kill -USR1 $pid

dd will output:

328356+0 records in
328356+0 records out
1344946176 bytes (1.3 GB) copied, 105.625 s, 12.7 MB/s

After this, you can insert the card in the N900.

=== Windows ===

# Download and install bzip2 to Windows http://gnuwin32.sourceforge.net/packages/bzip2.htm
# Uncompress raw image in command prompt "bunzip2.exe <compressed raw image>"
# Change (rename) extracted file extension from .raw to .img
# Download the Win32DiskImager.exe program: https://launchpad.net/win32-image-writer/+download (zip file)
# Unzip the file and extract the contents to a known directory
# Run W32DiskImager.exe
# Select the MeeGo image file
# Select the drive letter which corresponds to the microSD memory card.
# Click the "Write" button to byte-copy the image to the microSD memory card.
# insert the memory card into the device and attach the back cover
# load/flash kernel image

==== Notes about flashing the kernel ====

Get the N900's flasher from [http://tablets-dev.nokia.com/maemo-dev-env-downloads.php tablets-dev.nokia.com]

To load the kernel:
flasher-3.5.exe -l -b -k` [kernel image] to load it

To flash it
flasher-3.5.exe -f -b -k [kernel image] to flash it

Flashing replace the Maemo5 kernel and you won't be able to boot into maemo5 without flashing the original kernel back

When booting from a memory card make sure that the device's back cover is on. Otherwise the card will not mount and you'll see a kernel panic

''''''NOTE!!''' If you get error 'Not enough space on disk'. Try again with external memorycard reader.'''

=== Mac OS X ===

When inserting the microSD memory card in the card reader, you need to find out what the proper device for the card is. This can be done with disktool:
diskutil list

An example output ('''NOTE: The /dev/diskX is used as an example on your Mac. This might be also called /dev/disk2, /dev/disk3 or something else''')
$ diskutil list
...
/dev/diskX
#: TYPE NAME SIZE IDENTIFIER
0: FDisk_partition_scheme *4.1 GB disk3
1: DOS_FAT_32 DISKETTE 4.1 GB disk3s1

'''NOTE: The .raw image contains the partition table, as well. So the image needs to be written to /dev/diskX not /dev/diskX1.'''

'''NOTE: If you want to improve the speed writing your mmc card, you can use the raw device, avoiding the use of the os buffer, add 'r' add the name of your mmc interface, example: /dev/rdiskX'''

After you are 100% sure that the /dev/diskX is the microSD memory card you just inserted in the card reader, you can use, for example, dd to put the image onto the card:
$ sudo dd bs=4096 if='''<raw_image>''' of=/dev/diskX

If you are low on disk space, you can use

$ bzcat <raw_image>.bz2 | sudo dd bs=4096 of=/dev/diskX

to decompress the compressed raw image on the fly without having to unpack it on you computer first.

The dd does not show any progress until the file is written to the device, so be patient.

Although sending a SIGINFO signal to a running dd process makes it print I/O statistics to standard error and then resume copying:
$ dd if=/dev/zero of=/dev/null& pid=$!
from another term:
$ kill -SIGINFO $pid

After dd is done, dd will output something like this:

475136+1 records in
475136+1 records out
1946157057 bytes transferred in 2606.033611 secs (746789 bytes/sec)

After this, you can eject the microSD and insert the card in the N900.

=== On the N900 itself ===

This should be a last resort solution.

# Download and unpack the raw image to a desktop computer (see above).
# un-mount the microSD memory card
sudo gainroot
umount /dev/mmcblkXpY
dd if=/home/user/MyDocs/'''<raw_image>''' of=/dev/mmcblk1

==== Notes ====
* X, Y in umount command should be the device number and partition number, usually 1, 1. '''Do not take information from /proc/partitions!'''
N900 swaps devices during boot and /proc/partitions is what the kernel initially sees when it is loaded. N900's SD-Card is /dev/mmcblk1 for Maemo5-Standard-Install and not like shown in /proc/partitions, /dev/mmcblk0! NITdroid's SD-install has SD-Card as /dev/mmcblk0, instead. If you want to see what is mounted at the moment, type "mount|grep mmc" or "df|grep mmc".
* There might be more than one partition on the microSD memory card, though it is unlikely and depends on your own setup. You need to unmount all partitions before you proceed to `dd`.

* It is possible to download the compressed image to your N900, but it is not recommended as it takes a long time to unpack.
* `dd` does not give any output while it is copying, so be patient.
* If you aren't sure about any of these steps you should not proceed without consulting a professional.

== Load or flash kernel on N900 ==

Before MeeGo is able to boot, you also need to load the kernel (vmlinuz) provided with the raw image to the device. This can be done with the [[ARM/N900/tools/flasher|flasher]].

=== Dual-boot with maemo using u-boot ===

See [[ARM/N900/Install/Dual_Boot]] instead.

=== Load or flash manually ===
====Prerequisites====

* the flasher application needs to be installed on your computer. You can get it from [http://tablets-dev.nokia.com/maemo-dev-env-downloads.php here].
On 64-bit Ubuntu/Debian this command would do the trick, after downloading the .deb file:

$ sudo dpkg --force-architecture -i maemo_flasher-*i386.deb

====Running Meego====

'''NOTE: First, open the back cover of N900 and insert the MMC card to the slot. Reinstall the back cover again.

'''NOTE: Back cover must be closed to boot from MMC.'''

'''NOTE: If you have a flashing jig, you will need to put a magnet at the red location marked [http://www.daimi.au.dk/~cvm/magnet.png here]'''

You need to execute the following command on your host system.

'''NOTE: The device must be turned off and disconnected from the computer. Connect it to the computer using the USB cable, only after executing the command'''

$ sudo flasher-3.5 -l -k '''<kernel>''' -b

A message like:
flasher v2.5.2 (Oct 21 2009)
Suitable USB device not found, waiting.
is shown on the terminal of the computer.

Then connect N900 to the computer via the USB cable and the Meego OS will boot.
You may need to keep the 'U' key pressed (on the phone's keyboard) when connecting the cable.

'''NOTE: The command above will only load the kernel to the device, so next time you boot the device, the original kernel should be used and your Maemo 5 OS should boot normally.'''

If you want to flash the kernel to your device so that it is not forgotten when it is powered off, use option -f instead of option -l on command above.

'''NOTE: It's highly recommended NOT to flash the kernel on your device, unless you really know what you're doing (so please use -l instead of -f). Don't blame us if you brick your device, you have been warned!'''

{| class="wikitable" border="1" cellpadding="5"
|+ Examples of files that can be used with this guide
! MeeGo Version
! <kernel>
! <raw_image>
|-
| 1.0.99.2.20101019.1
| [http://repo.meego.com/MeeGo/builds/1.0.99/1.0.99.2.20101019.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.0.99.2.20101019.1-vmlinuz-2.6.35.3-10.3-n900 meego-handset-armv7l-n900-1.0.99.2.20101019.1-vmlinuz-2.6.35.3-10.3-n900]
| [http://repo.meego.com/MeeGo/builds/1.0.99/1.0.99.2.20101019.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.0.99.2.20101019.1-mmcblk0p.raw.bz2 meego-handset-armv7l-n900-1.0.99.2.20101019.1-mmcblk0p.raw.bz2]
|-
| 1.1.80.0.20101001.1
| [http://repo.meego.com/MeeGo/builds/trunk/1.1.80.0.20101001.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.1.80.0.20101001.1-vmlinuz-2.6.35.3-8.5-n900 meego-handset-armv7l-n900-1.1.80.0.20101001.1-vmlinuz-2.6.35.3-8.5-n900]
| [http://repo.meego.com/MeeGo/builds/trunk/1.1.80.0.20101001.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.1.80.0.20101001.1-mmcblk0p.raw.bz2 meego-handset-armv7l-n900-1.1.80.0.20101001.1-mmcblk0p.raw.bz2]
|}

[[Category:N900]]

ARM/N900/Install/MMC

2011-06-29T12:04:46Z

Edmondas: /* Windows */

= Installing MeeGo to N900 on external MMC card =

'''NOTE: Read the whole guide before doing any steps in this guide to make sure you understand everything.'''

If you do not want to erase the NAND partition or the internal 32 G eMMC from the N900 device, this installation is for you. What you need for this installation is a raw image that can be put to the MMC card and kernel (same kernel that is installed to the MMC).

=== Images ===

Raw images and corresponding kernel images can be found at http://repo.meego.com/MeeGo/builds/ (development builds) and http://repo.meego.com/MeeGo/releases/ (stable releases)

Find official release images here ([[ARM/N900#Releases]])

You may want to check for the most recent N900 test reports at [[Quality#MeeGo_Handset_Testing]] to see how much functionality is known to work on the N900.

== Installing Rootfs on external MMC card ==

First, what you need, is a microSD memory card which does not contain any information that you need, as it will be erased during this operation.

The steps in this guide require an MMC card of at least 4gb.

=== Linux ===

When inserting the microSD memory card in the card reader, you need to find out what the proper device for the card is. You can also plug the N900 into your computer's USB slot, using the package-provided cable. Make sure the external microSD card is unmounted, as with most modern linux distributions today it will get auto-mounted if the has a valid filesystem (FAT32 or ExtX). To unmount, you can try this example:
sudo umount /dev/sdX

And use the mount command to determine if/ where it is mounted:
sudo mount (for an example how the microSD would look go [[MountOutput here]].

Finding out the device node can also be done with, for example, fdisk:
sudo fdisk -l

An example output ('''NOTE: The /dev/sdX is used as an example on your PC. This might be also called /dev/mmcblk0, /dev/sdd or something else''')
$ sudo fdisk -l
...
Disk /dev/sdX: 3965 MB, 3965714432 bytes
194 heads, 30 sectors/track, 1330 cylinders
Units = cylinders of 5820 * 512 = 2979840 bytes
Disk identifier: 0x0001ab40

Device Boot Start End Blocks Id System
/dev/sdX1 1 588 1708984 83 Linux

'''NOTE: The .raw image contains the partition table, as well. So the image needs to be written to /dev/sdX not /dev/sdX1.'''

After you are 100% sure that the /dev/sdX is the microSD memory card you just inserted in the card reader, you can use, for example, dd to put the image onto the card:
$ sudo dd bs=4096 if='''<raw_image>''' of=/dev/sdX

If you are low on disk space, you can use

$ bzcat <raw_image>.bz2 | sudo dd bs=4096 of=/dev/sdX

to decompress the compressed raw image on the fly, without having to unpack it on you computer first. And if you have pv(1) installed, you can add it in between to display the progress (the image is ~ 2GB in size as of 2010-10-04):

$ bzcat <raw_image>.bz2 | pv | sudo dd bs=4096 of=/dev/sdX

The dd does not show any progress until the file is written to the device, so be patient.

Although sending a USR1 signal to a running dd process makes it print I/O statistics to standard error and then resume copying:
$ dd if=/dev/zero of=/dev/null& pid=$!
from another term:
$ kill -USR1 $pid

dd will output:

328356+0 records in
328356+0 records out
1344946176 bytes (1.3 GB) copied, 105.625 s, 12.7 MB/s

After this, you can insert the card in the N900.

=== Windows ===

# Download and install bzip2 to Windows http://gnuwin32.sourceforge.net/packages/bzip2.htm
# Uncompress raw image in command prompt "bunzip2.exe <compressed raw image>"
# Change (rename) extracted file extension from .raw to .img
# Download the Win32DiskImager.exe program: https://launchpad.net/win32-image-writer/+download (zip file)
# Unzip the file and extract the contents to a known directory
# Run W32DiskImager.exe
# Select the MeeGo image file
# Select the drive letter which corresponds to the microSD memory card.
# Click the "Write" button to byte-copy the image to the microSD memory card.
# insert the memory card into the device and attach the back cover
# load/flash kernel image

==== Notes about flashing the kernel ====

Get the N900's flasher from tablets-dev.nokia.com

To load the kernel:
flasher-3.5.exe -l -b -l [kernel image] to load it

To flash it
flasher-3.5.exe -f -b -k [kernel image] to flash it

Flashing replace the Maemo5 kernel and you won't be able to boot into maemo5 without flashing the original kernel back

When booting from a memory card make sure that the device's back cover is on. Otherwise the card will not mount and you'll see a kernel panic

''''''NOTE!!''' If you get error 'Not enough space on disk'. Try again with external memorycard reader.'''

=== Mac OS X ===

When inserting the microSD memory card in the card reader, you need to find out what the proper device for the card is. This can be done with disktool:
diskutil list

An example output ('''NOTE: The /dev/diskX is used as an example on your Mac. This might be also called /dev/disk2, /dev/disk3 or something else''')
$ diskutil list
...
/dev/diskX
#: TYPE NAME SIZE IDENTIFIER
0: FDisk_partition_scheme *4.1 GB disk3
1: DOS_FAT_32 DISKETTE 4.1 GB disk3s1

'''NOTE: The .raw image contains the partition table, as well. So the image needs to be written to /dev/diskX not /dev/diskX1.'''

'''NOTE: If you want to improve the speed writing your mmc card, you can use the raw device, avoiding the use of the os buffer, add 'r' add the name of your mmc interface, example: /dev/rdiskX'''

After you are 100% sure that the /dev/diskX is the microSD memory card you just inserted in the card reader, you can use, for example, dd to put the image onto the card:
$ sudo dd bs=4096 if='''<raw_image>''' of=/dev/diskX

If you are low on disk space, you can use

$ bzcat <raw_image>.bz2 | sudo dd bs=4096 of=/dev/diskX

to decompress the compressed raw image on the fly without having to unpack it on you computer first.

The dd does not show any progress until the file is written to the device, so be patient.

Although sending a SIGINFO signal to a running dd process makes it print I/O statistics to standard error and then resume copying:
$ dd if=/dev/zero of=/dev/null& pid=$!
from another term:
$ kill -SIGINFO $pid

After dd is done, dd will output something like this:

475136+1 records in
475136+1 records out
1946157057 bytes transferred in 2606.033611 secs (746789 bytes/sec)

After this, you can eject the microSD and insert the card in the N900.

=== On the N900 itself ===

This should be a last resort solution.

# Download and unpack the raw image to a desktop computer (see above).
# un-mount the microSD memory card
sudo gainroot
umount /dev/mmcblkXpY
dd if=/home/user/MyDocs/'''<raw_image>''' of=/dev/mmcblk1

==== Notes ====
* X, Y in umount command should be the device number and partition number, usually 1, 1. '''Do not take information from /proc/partitions!'''
N900 swaps devices during boot and /proc/partitions is what the kernel initially sees when it is loaded. N900's SD-Card is /dev/mmcblk1 for Maemo5-Standard-Install and not like shown in /proc/partitions, /dev/mmcblk0! NITdroid's SD-install has SD-Card as /dev/mmcblk0, instead. If you want to see what is mounted at the moment, type "mount|grep mmc" or "df|grep mmc".
* There might be more than one partition on the microSD memory card, though it is unlikely and depends on your own setup. You need to unmount all partitions before you proceed to `dd`.

* It is possible to download the compressed image to your N900, but it is not recommended as it takes a long time to unpack.
* `dd` does not give any output while it is copying, so be patient.
* If you aren't sure about any of these steps you should not proceed without consulting a professional.

== Load or flash kernel on N900 ==

Before MeeGo is able to boot, you also need to load the kernel (vmlinuz) provided with the raw image to the device. This can be done with the [[ARM/N900/tools/flasher|flasher]].

=== Dual-boot with maemo using u-boot ===

See [[ARM/N900/Install/Dual_Boot]] instead.

=== Load or flash manually ===
====Prerequisites====

* the flasher application needs to be installed on your computer. You can get it from [http://tablets-dev.nokia.com/maemo-dev-env-downloads.php here].
On 64-bit Ubuntu/Debian this command would do the trick, after downloading the .deb file:

$ sudo dpkg --force-architecture -i maemo_flasher-*i386.deb

====Running Meego====

'''NOTE: First, open the back cover of N900 and insert the MMC card to the slot. Reinstall the back cover again.

'''NOTE: Back cover must be closed to boot from MMC.'''

'''NOTE: If you have a flashing jig, you will need to put a magnet at the red location marked [http://www.daimi.au.dk/~cvm/magnet.png here]'''

You need to execute the following command on your host system.

'''NOTE: The device must be turned off and disconnected from the computer. Connect it to the computer using the USB cable, only after executing the command'''

$ sudo flasher-3.5 -l -k '''<kernel>''' -b

A message like:
flasher v2.5.2 (Oct 21 2009)
Suitable USB device not found, waiting.
is shown on the terminal of the computer.

Then connect N900 to the computer via the USB cable and the Meego OS will boot.
You may need to keep the 'U' key pressed (on the phone's keyboard) when connecting the cable.

'''NOTE: The command above will only load the kernel to the device, so next time you boot the device, the original kernel should be used and your Maemo 5 OS should boot normally.'''

If you want to flash the kernel to your device so that it is not forgotten when it is powered off, use option -f instead of option -l on command above.

'''NOTE: It's highly recommended NOT to flash the kernel on your device, unless you really know what you're doing (so please use -l instead of -f). Don't blame us if you brick your device, you have been warned!'''

{| class="wikitable" border="1" cellpadding="5"
|+ Examples of files that can be used with this guide
! MeeGo Version
! <kernel>
! <raw_image>
|-
| 1.0.99.2.20101019.1
| [http://repo.meego.com/MeeGo/builds/1.0.99/1.0.99.2.20101019.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.0.99.2.20101019.1-vmlinuz-2.6.35.3-10.3-n900 meego-handset-armv7l-n900-1.0.99.2.20101019.1-vmlinuz-2.6.35.3-10.3-n900]
| [http://repo.meego.com/MeeGo/builds/1.0.99/1.0.99.2.20101019.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.0.99.2.20101019.1-mmcblk0p.raw.bz2 meego-handset-armv7l-n900-1.0.99.2.20101019.1-mmcblk0p.raw.bz2]
|-
| 1.1.80.0.20101001.1
| [http://repo.meego.com/MeeGo/builds/trunk/1.1.80.0.20101001.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.1.80.0.20101001.1-vmlinuz-2.6.35.3-8.5-n900 meego-handset-armv7l-n900-1.1.80.0.20101001.1-vmlinuz-2.6.35.3-8.5-n900]
| [http://repo.meego.com/MeeGo/builds/trunk/1.1.80.0.20101001.1/handset/images/meego-handset-armv7l-n900/meego-handset-armv7l-n900-1.1.80.0.20101001.1-mmcblk0p.raw.bz2 meego-handset-armv7l-n900-1.1.80.0.20101001.1-mmcblk0p.raw.bz2]
|}

[[Category:N900]]

SDKBugTriageMinutesArchive

2011-06-27T12:22:46Z

Edmondas:

*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-06-22-05.57.html MeeGo SDK Bug Triage 20110622]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-06-15-05.58.html MeeGo SDK Bug Triage 20110615]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-06-08-05.57.html MeeGo SDK Bug Triage 20110608]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-05-25-05.58.html MeeGo SDK Bug Triage 20110525]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-05-18-05.58.html MeeGo SDK Bug Triage 20110518]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-05-11-05.58.html MeeGo SDK Bug Triage 20110511]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-05-04-05.59.html MeeGo SDK Bug Triage 20110504]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-04-27-05.59.html MeeGo SDK Bug Triage 20110427]

SDKBugTriageMinutesArchive

2011-06-16T11:16:50Z

Edmondas: added latest irc logs

*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-06-15-05.58.html MeeGo SDK Bug Triage 20110615]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-06-08-05.57.html MeeGo SDK Bug Triage 20110608]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-05-25-05.58.html MeeGo SDK Bug Triage 20110525]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-05-18-05.58.html MeeGo SDK Bug Triage 20110518]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-05-11-05.58.html MeeGo SDK Bug Triage 20110511]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-05-04-05.59.html MeeGo SDK Bug Triage 20110504]
*[http://irclogs.meego.com/meetbot/meego-meeting2/2011/meego-meeting2.2011-04-27-05.59.html MeeGo SDK Bug Triage 20110427]

Quality/Bugtriage

2011-06-14T08:38:41Z

Edmondas: /* MeeGo Bug Triage Team */

== Bug Triage Definition ==
* Bug Triage is a process to:
** Ensure bug report completeness
** Analyze and assign bug to proper component
** Assign bug to proper bug owner
** Set appropriate bug priority
** Adjust bug severity properly (initially set by bug reporter)
** Resolve obvious invalid, duplication, won’t fix bugs etc.
** Forward bugs in upstream components to [[Quality/UpstreamBugTrackers | upstream bugtrackers]] and adding the upstream URL to the URL field of the report, if applicable

* Bug Triage Team
** A small team works on bug triage, could be experienced developer, distro engineer or QA
** Triage team members are expected to contribute significant time to bug triage

Please share your triaging knowledge by adding/editing [[Quality/Bugtriage_Stock_Answers | MeeGo Bug Triage Stock Answers]].

== Bug Triage Process ==
* Triage new incoming bug reports timely by each triage team (from twice a week to daily triage).
* Triage team members in each triage team could have different focus, such as IA arch bugs, ARM bugs or specific applications etc.
* Each triage team meet on IRC weekly to discuss controversial bug reports and any open reports
* Bug assignees accept bug reports by setting target milestones for triaged bug reports
* [http://meego.com/about/governance/program-office Program Managers] host bug report scrub meetings to discuss bug reports which do not have a target milestone set

Triage Process Flow as follows:
[[File:bug_triage_process.jpg]]

== MeeGo Bug Triage Guide ==
Check the [[Quality/Bugtriage_Guide|Triage Guide]] that explains good practices when triaging bug reports.

== MeeGo Bug Triage Team ==
{|cellspacing="0" border="1"
!|MeeGo Bug Triage Team
!|Member
|-
| MeeGo Core OS Triage || [http://meego.com/users/shuangeeer Yanshuang Zheng], [http://meego.com/users/ares2012 Jason Zhou], [http://meego.com/users/jarnoteivas Jarno Teivas], [http://meego.com/users/iekku Iekku Huttunen]
|-
| MeeGo Handset UX Triage || Fan Zhao, Cathy Li, [http://meego.com/users/srikanthyarlagadda Srikanth Yarlagadda],[http://meego.com/users/jylha Petri Jylha], [http://meego.com/users/ceferron Chris Ferron], [http://meego.com/users/iekku Iekku Huttunen]
|-
| MeeGo Netbook UX Triage || [http://meego.com/users/lingyu Ling Yu], Daniel Tao, [http://meego.com/users/yanglei Lei Yang], [http://meego.com/users/rossburton Ross Burton]
|-
| MeeGo Tablet UX Triage || [http://meego.com/users/lingyu Ling Yu], Daniel Tao, [http://meego.com/users/yanglei Lei Yang]
|-
| MeeGo IVI Triage || [http://meego.com/users/shuangeeer Yanshuang Zheng]
|-
| MeeGo Translation Triage || [http://meego.com/users/margie Margie Foster], [http://meego.com/users/pmccarty Patrick McCarty]
|-
| MeeGo SDK Triage || [http://meego.com/users/ionutgavaz Ionut Gavaz], Jackie Wu, [http://meego.com/users/edmondas Edmondas Girkantas], Azadeh Karimian, Max Yu, Juha Peisanen
|-
| MCTS (MeeGo Core System Testing) Triage || [http://meego.com/users/jarnoteivas Jarno Teivas], [http://meego.com/users/iekku Iekku Huttunen], Jeff Zheng
|}

=== MeeGo Bug Triage Meetings ===

* MeeGo Core Bug Triage
** Time: Every Monday at 07:00 [http://www.timeanddate.com/worldclock/converter.html UTC]
* MeeGo Handset Bug Triage
** Time: Every Wednesday at 07:00 [http://www.timeanddate.com/worldclock/converter.html UTC]
* MeeGo Netbook Bug Triage
** Time: Every Tuesday at 13:00 [http://www.timeanddate.com/worldclock/converter.html UTC]
* MeeGo IVI Bug Triage
** Time: TBD
* MeeGo Translation Bug Triage
** Time: TBD
* MeeGo SDK Bug Triage
** Time: Every Wednesday at 07:00 [http://www.timeanddate.com/worldclock/converter.html UTC]
* MCTS Bug Triage
** Time: Every Wednesday at 08:00 [http://www.timeanddate.com/worldclock/converter.html UTC]

Those meetings take place in the IRC channel'''#meego-meeting''' on [http://freenode.net irc.freenode.net].

* MeeGo Tablet Bug Triage
** Time: Every Thursday at 07:00 [http://www.timeanddate.com/worldclock/converter.html UTC]
** IRC channel '''#meego-meeting2''' on [http://freenode.net irc.freenode.net]

To discuss MeeGo bug triaging at any time feel free to visit the IRC channel '''#meego-bugs''' on [http://freenode.net irc.freenode.net].

* [[CoreBugTriageMinutesArchive|MeeGo Core Bug Triage Meeting Minutes Archive]]
* [[HandsetBugTriageMinutesArchive|MeeGo Handset Bug Triage Meeting Minutes Archive]]
* [[NetbookBugTriageMinutesArchive|MeeGo Netbook Bug Triage Meeting Minutes Archive]]
* [[SDKBugTriageMinutesArchive|MeeGo SDK Bug Triage Meeting Minutes Archive]]

=== Getting involved ===

Anyone can sign up for the triage team and start helping (see the [[Quality/Bugtriage_Guide|Triage Guide]] for information and steps). Just send an email to the [http://lists.meego.com/listinfo/meego-qa meego-qa mailing list] to get involved in. Thanks for your contribution to MeeGo!

== Other references ==
[[Quality/How_To_Report_Bugs|How to report MeeGo bugs?]]

[[Category:QA]]

Quality/Bugtriage

2011-06-14T08:35:48Z

Edmondas: /* MeeGo Bug Triage Team */

== Bug Triage Definition ==
* Bug Triage is a process to:
** Ensure bug report completeness
** Analyze and assign bug to proper component
** Assign bug to proper bug owner
** Set appropriate bug priority
** Adjust bug severity properly (initially set by bug reporter)
** Resolve obvious invalid, duplication, won’t fix bugs etc.
** Forward bugs in upstream components to [[Quality/UpstreamBugTrackers | upstream bugtrackers]] and adding the upstream URL to the URL field of the report, if applicable

* Bug Triage Team
** A small team works on bug triage, could be experienced developer, distro engineer or QA
** Triage team members are expected to contribute significant time to bug triage

Please share your triaging knowledge by adding/editing [[Quality/Bugtriage_Stock_Answers | MeeGo Bug Triage Stock Answers]].

== Bug Triage Process ==
* Triage new incoming bug reports timely by each triage team (from twice a week to daily triage).
* Triage team members in each triage team could have different focus, such as IA arch bugs, ARM bugs or specific applications etc.
* Each triage team meet on IRC weekly to discuss controversial bug reports and any open reports
* Bug assignees accept bug reports by setting target milestones for triaged bug reports
* [http://meego.com/about/governance/program-office Program Managers] host bug report scrub meetings to discuss bug reports which do not have a target milestone set

Triage Process Flow as follows:
[[File:bug_triage_process.jpg]]

== MeeGo Bug Triage Guide ==
Check the [[Quality/Bugtriage_Guide|Triage Guide]] that explains good practices when triaging bug reports.

== MeeGo Bug Triage Team ==
{|cellspacing="0" border="1"
!|MeeGo Bug Triage Team
!|Member
|-
| MeeGo Core OS Triage || [http://meego.com/users/shuangeeer Yanshuang Zheng], [http://meego.com/users/ares2012 Jason Zhou], [http://meego.com/users/jarnoteivas Jarno Teivas], [http://meego.com/users/iekku Iekku Huttunen]
|-
| MeeGo Handset UX Triage || Fan Zhao, Cathy Li, [http://meego.com/users/srikanthyarlagadda Srikanth Yarlagadda],[http://meego.com/users/jylha Petri Jylha], [http://meego.com/users/ceferron Chris Ferron], [http://meego.com/users/iekku Iekku Huttunen]
|-
| MeeGo Netbook UX Triage || [http://meego.com/users/lingyu Ling Yu], Daniel Tao, [http://meego.com/users/yanglei Lei Yang], [http://meego.com/users/rossburton Ross Burton]
|-
| MeeGo Tablet UX Triage || [http://meego.com/users/lingyu Ling Yu], Daniel Tao, [http://meego.com/users/yanglei Lei Yang]
|-
| MeeGo IVI Triage || [http://meego.com/users/shuangeeer Yanshuang Zheng]
|-
| MeeGo Translation Triage || [http://meego.com/users/margie Margie Foster], [http://meego.com/users/pmccarty Patrick McCarty]
|-
| MeeGo SDK Triage || Juha Peisanen, Ionut Gavaz, [http://meego.com/users/edmondas Edmondas Girkantas], Azadeh Karimian, Jackie Wu, Max Yu
|-
| MCTS (MeeGo Core System Testing) Triage || [http://meego.com/users/jarnoteivas Jarno Teivas], [http://meego.com/users/iekku Iekku Huttunen], Jeff Zheng
|}

=== MeeGo Bug Triage Meetings ===

* MeeGo Core Bug Triage
** Time: Every Monday at 07:00 [http://www.timeanddate.com/worldclock/converter.html UTC]
* MeeGo Handset Bug Triage
** Time: Every Wednesday at 07:00 [http://www.timeanddate.com/worldclock/converter.html UTC]
* MeeGo Netbook Bug Triage
** Time: Every Tuesday at 13:00 [http://www.timeanddate.com/worldclock/converter.html UTC]
* MeeGo IVI Bug Triage
** Time: TBD
* MeeGo Translation Bug Triage
** Time: TBD
* MeeGo SDK Bug Triage
** Time: Every Wednesday at 07:00 [http://www.timeanddate.com/worldclock/converter.html UTC]
* MCTS Bug Triage
** Time: Every Wednesday at 08:00 [http://www.timeanddate.com/worldclock/converter.html UTC]

Those meetings take place in the IRC channel'''#meego-meeting''' on [http://freenode.net irc.freenode.net].

* MeeGo Tablet Bug Triage
** Time: Every Thursday at 07:00 [http://www.timeanddate.com/worldclock/converter.html UTC]
** IRC channel '''#meego-meeting2''' on [http://freenode.net irc.freenode.net]

To discuss MeeGo bug triaging at any time feel free to visit the IRC channel '''#meego-bugs''' on [http://freenode.net irc.freenode.net].

* [[CoreBugTriageMinutesArchive|MeeGo Core Bug Triage Meeting Minutes Archive]]
* [[HandsetBugTriageMinutesArchive|MeeGo Handset Bug Triage Meeting Minutes Archive]]
* [[NetbookBugTriageMinutesArchive|MeeGo Netbook Bug Triage Meeting Minutes Archive]]
* [[SDKBugTriageMinutesArchive|MeeGo SDK Bug Triage Meeting Minutes Archive]]

=== Getting involved ===

Anyone can sign up for the triage team and start helping (see the [[Quality/Bugtriage_Guide|Triage Guide]] for information and steps). Just send an email to the [http://lists.meego.com/listinfo/meego-qa meego-qa mailing list] to get involved in. Thanks for your contribution to MeeGo!

== Other references ==
[[Quality/How_To_Report_Bugs|How to report MeeGo bugs?]]

[[Category:QA]]

Quality/Bugtriage

2011-06-14T08:16:14Z

Edmondas: /* MeeGo Bug Triage Team */

== Bug Triage Definition ==
* Bug Triage is a process to:
** Ensure bug report completeness
** Analyze and assign bug to proper component
** Assign bug to proper bug owner
** Set appropriate bug priority
** Adjust bug severity properly (initially set by bug reporter)
** Resolve obvious invalid, duplication, won’t fix bugs etc.
** Forward bugs in upstream components to [[Quality/UpstreamBugTrackers | upstream bugtrackers]] and adding the upstream URL to the URL field of the report, if applicable

* Bug Triage Team
** A small team works on bug triage, could be experienced developer, distro engineer or QA
** Triage team members are expected to contribute significant time to bug triage

Please share your triaging knowledge by adding/editing [[Quality/Bugtriage_Stock_Answers | MeeGo Bug Triage Stock Answers]].

== Bug Triage Process ==
* Triage new incoming bug reports timely by each triage team (from twice a week to daily triage).
* Triage team members in each triage team could have different focus, such as IA arch bugs, ARM bugs or specific applications etc.
* Each triage team meet on IRC weekly to discuss controversial bug reports and any open reports
* Bug assignees accept bug reports by setting target milestones for triaged bug reports
* [http://meego.com/about/governance/program-office Program Managers] host bug report scrub meetings to discuss bug reports which do not have a target milestone set

Triage Process Flow as follows:
[[File:bug_triage_process.jpg]]

== MeeGo Bug Triage Guide ==
Check the [[Quality/Bugtriage_Guide|Triage Guide]] that explains good practices when triaging bug reports.

== MeeGo Bug Triage Team ==
{|cellspacing="0" border="1"
!|MeeGo Bug Triage Team
!|Member
|-
| MeeGo Core OS Triage || [http://meego.com/users/shuangeeer Yanshuang Zheng], [http://meego.com/users/ares2012 Jason Zhou], [http://meego.com/users/jarnoteivas Jarno Teivas], [http://meego.com/users/iekku Iekku Huttunen]
|-
| MeeGo Handset UX Triage || Fan Zhao, Cathy Li, [http://meego.com/users/srikanthyarlagadda Srikanth Yarlagadda],[http://meego.com/users/jylha Petri Jylha], [http://meego.com/users/ceferron Chris Ferron], [http://meego.com/users/iekku Iekku Huttunen]
|-
| MeeGo Netbook UX Triage || [http://meego.com/users/lingyu Ling Yu], Daniel Tao, [http://meego.com/users/yanglei Lei Yang], [http://meego.com/users/rossburton Ross Burton]
|-
| MeeGo Tablet UX Triage || [http://meego.com/users/lingyu Ling Yu], Daniel Tao, [http://meego.com/users/yanglei Lei Yang]
|-
| MeeGo IVI Triage || [http://meego.com/users/shuangeeer Yanshuang Zheng]
|-
| MeeGo Translation Triage || [http://meego.com/users/margie Margie Foster], [http://meego.com/users/pmccarty Patrick McCarty]
|-
| MeeGo SDK Triage || Juha Peisanen, Ionut Gavaz, [http://meego.com/users/edmondas Edmondas Girkantas], Jackie Wu, Max Yu
|-
| MCTS (MeeGo Core System Testing) Triage || [http://meego.com/users/jarnoteivas Jarno Teivas], [http://meego.com/users/iekku Iekku Huttunen], Jeff Zheng
|}

=== MeeGo Bug Triage Meetings ===

* MeeGo Core Bug Triage
** Time: Every Monday at 07:00 [http://www.timeanddate.com/worldclock/converter.html UTC]
* MeeGo Handset Bug Triage
** Time: Every Wednesday at 07:00 [http://www.timeanddate.com/worldclock/converter.html UTC]
* MeeGo Netbook Bug Triage
** Time: Every Tuesday at 13:00 [http://www.timeanddate.com/worldclock/converter.html UTC]
* MeeGo IVI Bug Triage
** Time: TBD
* MeeGo Translation Bug Triage
** Time: TBD
* MeeGo SDK Bug Triage
** Time: Every Wednesday at 07:00 [http://www.timeanddate.com/worldclock/converter.html UTC]
* MCTS Bug Triage
** Time: Every Wednesday at 08:00 [http://www.timeanddate.com/worldclock/converter.html UTC]

Those meetings take place in the IRC channel'''#meego-meeting''' on [http://freenode.net irc.freenode.net].

* MeeGo Tablet Bug Triage
** Time: Every Thursday at 07:00 [http://www.timeanddate.com/worldclock/converter.html UTC]
** IRC channel '''#meego-meeting2''' on [http://freenode.net irc.freenode.net]

To discuss MeeGo bug triaging at any time feel free to visit the IRC channel '''#meego-bugs''' on [http://freenode.net irc.freenode.net].

* [[CoreBugTriageMinutesArchive|MeeGo Core Bug Triage Meeting Minutes Archive]]
* [[HandsetBugTriageMinutesArchive|MeeGo Handset Bug Triage Meeting Minutes Archive]]
* [[NetbookBugTriageMinutesArchive|MeeGo Netbook Bug Triage Meeting Minutes Archive]]
* [[SDKBugTriageMinutesArchive|MeeGo SDK Bug Triage Meeting Minutes Archive]]

=== Getting involved ===

Anyone can sign up for the triage team and start helping (see the [[Quality/Bugtriage_Guide|Triage Guide]] for information and steps). Just send an email to the [http://lists.meego.com/listinfo/meego-qa meego-qa mailing list] to get involved in. Thanks for your contribution to MeeGo!

== Other references ==
[[Quality/How_To_Report_Bugs|How to report MeeGo bugs?]]

[[Category:QA]]

SDK/Sysroot extension

2011-06-09T12:29:06Z

Edmondas:

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]] (Updated on 28th April)
* some PoF scripts on [https://www.gitorious.org/sysroot-extend Gitorious]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

'''Note:''' this feature is already implemented in [https://meego.gitorious.org/meego-developer-tools/madde/blobs/master/src/madlib/xrpm.pl madde].

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* ???

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

=== Mac OS X share in QEMU ===

Command to sync:
rsync -a -L --no-p --no-g --chmod=ugo=rwX --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

rsync speed ~900KB/s

Issues: rsync can't change ownership (chown) of copied files while using native Windows sharing service on Mac OS X, e.g.:
rsync: chown "/mnt/bin" failed: Permission denied (13)

I can't test it fully (using Qt-creator) because MeeGo SDK porting to Mac OS X is still in progress.

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\127.0.0.1\meego /user:root password

== Other notes ==
=== Linker options for qmake ===

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

=== Required directories for compilation ===
I found that these directories are required in Qt-Creator to compile application:

/lib
/usr/lib
/usr/include
/usr/share

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format (qemu version 0.12.5)
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

Note: qemu-img version (0.14.50) from git repository is much faster - the same conversation takes only ~1m30s

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

[[Category:SDK]]

Build Infrastructure/Packagers Developers/CLI Part 1

2011-05-24T13:24:00Z

Edmondas: /* OSC Tool Install */

= How to use the MeeGo / OBS Command Line Interface =

Once you've successfully used the the MeeGo / OBS Web Interface, we'll show you how you can check out packages from the Meego project for local modification and building.

== OSC Tool Install ==

You will need to install some tools in order to access and build packages. This includes the command line client 'osc', and packages that it relies on.

The easiest method is to use your distribution's package manager to pull the osc package and its requirements. Package repositories for many distributions are available [http://download.opensuse.org/repositories/openSUSE:/Tools/ here].

For example, with Fedora 12:
# cd /etc/yum.repos.d
# wget http://download.opensuse.org/repositories/openSUSE:/Tools/Fedora_12/openSUSE:Tools.repo
# wget http://download.opensuse.org/repositories/openSUSE:/Tools:/MeeGo/Fedora_12/openSUSE:Tools:MeeGo.repo
# yum install osc

The upstream [http://en.opensuse.org/openSUSE:Build_Service_Tutorial OBS Tutorial] describes more generalities about using the tools.

== Running OSC the first time ==

Prepare to run OSC.

Tip: To use a different source of packages and build server access,
you can easily create a wrapper script to do this for you:
$ cat > ~/bin/losc
#!/bin/bash
exec osc -Ahttps://api.meego.com "$@"
$ chmod +x ~/bin/losc

When you're using pub meego osc you should use:
exec osc -Ahttps://api'''.pub'''.meego.com "$@"
instead of
exec osc -Ahttps://api.meego.com "$@"

The first time, OSC will ask you some questions:

$ losc ls

Your user account / password are not configured yet.
You will be asked for them below, and they will be stored in
/home/james/.oscrc for future use.

Creating osc configuration file /home/james/.oscrc ...
Username: <login>
Password: <password>
done
*** certificate verify failed at depth 0
Subject: /O=build.linux.com/OU=Domain Control Validated/CN=build.linux.com
Issuer: /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07969287
Valid: May 14 21:46:46 2010 GMT - Apr 26 21:10:19 2012 GMT
Fingerprint(MD5): C13D91AB12008D2F9FD901A8FADDFC75
Fingerprint(SHA1): 144018EC455C20F395F18879DFD75E5B500B84F0
Reason: unable to get local issuer certificate
Reason: certificate not trusted
Reason: unable to verify the first certificate

The server certificate failed verification

Would you like to
0 - quit (default)
1 - continue anyways
2 - trust the server certificate permanently
9 - review the server certificate

Enter choice [0129]: 2

At this point the configuration is complete and osc completes the 'ls' command, listing available build targets. You should see your home project listed, ex: home:<login>

== Tip: Configuring OSC using $HOME/.oscrc ==

OSC may be customized using the file $HOME/.oscrc. Some hints:

To make it easier to run needed things as root, configure for sudo

In $HOME/.oscrc set:
su-wrapper = sudo
And then run 'visudo' and add the line:
<yourlogin> ALL=NOPASSWD: /usr/bin/build

If you tend to build for several different architectures/projects, it can be helpful to keep each build-root seperate, in $HOME/.oscrc, set:

build-root = /var/tmp/build-root/%(project)s-%(arch)s-%(package)s

== Summary and next steps ==

This guide gave an introduction on how to set up and initialize the 'osc' tool.
Next you will check out your project from the server and build it. Click [[Build_Infrastructure/Packagers_Developers/CLI_Part_2|here]] to open the 2nd part.

[[Category:Build Infrastructure]]

SDK/Docs/1.2/MeeGo SDK 1.2 Preview for Linux

2011-05-18T12:46:24Z

Edmondas: update installer version

== Download the MeeGo SDK ==
<ol>
<li>'''Download and install the SDK'''

{| class="wikitable"
|+ Download information
|-
! File !! File Size !! MD5SUM !! Installed Size
|-
| [http://s3.amazonaws.com/MeeGoSDK/installer/Linux/meego-sdk-installer-online-20110516 MeeGo SDK 1.2] || 26MB || 270feaadeab45da9bffe162bb792a919 || 5.2 GB
|}

$ chmod +x ./meego-sdk-installer-online
$ ./meego-sdk-installer-online --verbose

Note that --verbose is optional. 
SDK content details are described on the [[SDK/Docs/1.2/MeeGo SDK 1.2 Components|SDK Component Summary]] page.
</li>
<li>'''Select the components that will be installed. Mainly select the target for development. During downloading, once one of components failed to download due to the connection issue, just click 'Retry' button in the notification dialog to resume.'''</li>
[[File:Installer-select component.png]]
</ol>
== Install a MeeGo Target ==
A ''Target'' is used to build programs using the MeeGo toolchain and libraries (sysroot). Except you can install target in the process of SDK, you also can install target by launching Maintain MeeGo SDK.
'''MeeGo SDK 1.2 Preview > Maintain MeeGo SDk''' . Select targets from SDK targest and click next. It will install targets you selected 
[[File:Select target.png]]

 

'''Next Step:[[SDK/Docs/1.2/MeeGo_SDK_1.2_Preview_for_Linux/Configure_Qt_Creator_to_Develop_MeeGo_Applications]]'''

[[Category:SDK]]
[[Category:Meego-1.2]]
[[Category:Linux]]

File:Sysroot-extension3.pdf

2011-05-12T10:46:04Z

Edmondas: uploaded a new version of "File:Sysroot-extension3.pdf": testing pdf upload

SDK/Sysroot extension

2011-04-28T10:55:35Z

Edmondas: /* Goals */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]] (Updated on 28th April)
* some PoF scripts on [https://www.gitorious.org/sysroot-extend Gitorious]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

'''Note:''' this feature is already implemented in [https://meego.gitorious.org/meego-developer-tools/madde/blobs/master/src/madlib/xrpm.pl madde].

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* ???

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

=== Mac OS X share in QEMU ===

Command to sync:
rsync -a -L --no-p --no-g --chmod=ugo=rwX --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

rsync speed ~900KB/s

Issues: rsync can't change ownership (chown) of copied files while using native Windows sharing service on Mac OS X, e.g.:
rsync: chown "/mnt/bin" failed: Permission denied (13)

I can't test it fully (using Qt-creator) because MeeGo SDK porting to Mac OS X is still in progress.

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\127.0.0.1\meego /user:root password

== Other notes ==
=== Linker options for qmake ===

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

=== Required directories for compilation ===
I found that these directories are required in Qt-Creator to compile application:

/lib
/usr/lib
/usr/include
/usr/share

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format (qemu version 0.12.5)
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

Note: qemu-img version (0.14.50) from git repository is much faster - the same conversation takes only ~1m30s

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

[[Category:SDK]]

File:Sysroot-extension3.pdf

2011-04-28T10:54:43Z

Edmondas: uploaded a new version of "File:Sysroot-extension3.pdf": Updated presentation, added slide about Mac OS X and sync. idea

SDK/Sysroot extension

2011-04-28T09:42:16Z

Edmondas: /* Description */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]
* some PoF scripts on [https://www.gitorious.org/sysroot-extend Gitorious]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

'''Note:''' this feature is already implemented in [https://meego.gitorious.org/meego-developer-tools/madde/blobs/master/src/madlib/xrpm.pl madde].

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* ???

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

=== Mac OS X share in QEMU ===

Command to sync:
rsync -a -L --no-p --no-g --chmod=ugo=rwX --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

rsync speed ~900KB/s

Issues: rsync can't change ownership (chown) of copied files while using native Windows sharing service on Mac OS X, e.g.:
rsync: chown "/mnt/bin" failed: Permission denied (13)

I can't test it fully (using Qt-creator) because MeeGo SDK porting to Mac OS X is still in progress.

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\127.0.0.1\meego /user:root password

== Other notes ==
=== Linker options for qmake ===

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

=== Required directories for compilation ===
I found that these directories are required in Qt-Creator to compile application:

/lib
/usr/lib
/usr/include
/usr/share

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format (qemu version 0.12.5)
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

Note: qemu-img version (0.14.50) from git repository is much faster - the same conversation takes only ~1m30s

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

[[Category:SDK]]

SDK/Sysroot extension

2011-04-28T09:37:12Z

Edmondas: /* Description */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]
* some PoF scripts on [https://www.gitorious.org/sysroot-extend Gitorious]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

'''Note:''' this feature is already implemented.

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* ???

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

=== Mac OS X share in QEMU ===

Command to sync:
rsync -a -L --no-p --no-g --chmod=ugo=rwX --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

rsync speed ~900KB/s

Issues: rsync can't change ownership (chown) of copied files while using native Windows sharing service on Mac OS X, e.g.:
rsync: chown "/mnt/bin" failed: Permission denied (13)

I can't test it fully (using Qt-creator) because MeeGo SDK porting to Mac OS X is still in progress.

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\127.0.0.1\meego /user:root password

== Other notes ==
=== Linker options for qmake ===

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

=== Required directories for compilation ===
I found that these directories are required in Qt-Creator to compile application:

/lib
/usr/lib
/usr/include
/usr/share

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format (qemu version 0.12.5)
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

Note: qemu-img version (0.14.50) from git repository is much faster - the same conversation takes only ~1m30s

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

[[Category:SDK]]

File:Sysroot-extension3.pdf

2011-04-18T11:04:05Z

Edmondas:

SDK/Sysroot extension

2011-04-11T10:48:41Z

Edmondas: /* Mac OS X share in QEMU */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]
* some PoF scripts on [https://www.gitorious.org/sysroot-extend Gitorious]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* ???

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

=== Mac OS X share in QEMU ===

Command to sync:
rsync -a -L --no-p --no-g --chmod=ugo=rwX --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

rsync speed ~900KB/s

Issues: rsync can't change ownership (chown) of copied files while using native Windows sharing service on Mac OS X, e.g.:
rsync: chown "/mnt/bin" failed: Permission denied (13)

I can't test it fully (using Qt-creator) because MeeGo SDK porting to Mac OS X is still in progress.

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\127.0.0.1\meego /user:root password

== Other notes ==
=== Linker options for qmake ===

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

=== Required directories for compilation ===
I found that these directories are required in Qt-Creator to compile application:

/lib
/usr/lib
/usr/include
/usr/share

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format (qemu version 0.12.5)
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

Note: qemu-img version (0.14.50) from git repository is much faster - the same conversation takes only ~1m30s

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

SDK/Sysroot extension

2011-04-11T10:27:43Z

Edmondas: /* Mac OS X share in QEMU */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]
* some PoF scripts on [https://www.gitorious.org/sysroot-extend Gitorious]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* ???

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

=== Mac OS X share in QEMU ===

Command to sync:
rsync -a -L --no-p --no-g --chmod=ugo=rwX --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

rsync speed ~900KB/s

Issues: rsync can't change ownership (chown) of copied files while using native Windows sharing service on Mac OS X, e.g.:
rsync: chown "/mnt/bin" failed: Permission denied (13)

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\127.0.0.1\meego /user:root password

== Other notes ==
=== Linker options for qmake ===

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

=== Required directories for compilation ===
I found that these directories are required in Qt-Creator to compile application:

/lib
/usr/lib
/usr/include
/usr/share

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format (qemu version 0.12.5)
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

Note: qemu-img version (0.14.50) from git repository is much faster - the same conversation takes only ~1m30s

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

SDK/Sysroot extension

2011-04-11T10:16:45Z

Edmondas: /* Mac OS X share in QEMU */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]
* some PoF scripts on [https://www.gitorious.org/sysroot-extend Gitorious]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* ???

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

=== Mac OS X share in QEMU ===

Command to sync:
rsync -a -L --no-p --no-g --chmod=ugo=rwX --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

rsync speed ~900KB/s

Issues: TBD

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\127.0.0.1\meego /user:root password

== Other notes ==
=== Linker options for qmake ===

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

=== Required directories for compilation ===
I found that these directories are required in Qt-Creator to compile application:

/lib
/usr/lib
/usr/include
/usr/share

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format (qemu version 0.12.5)
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

Note: qemu-img version (0.14.50) from git repository is much faster - the same conversation takes only ~1m30s

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

SDK/Sysroot extension

2011-04-11T10:10:06Z

Edmondas: /* Windows share in QEMU */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]
* some PoF scripts on [https://www.gitorious.org/sysroot-extend Gitorious]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* ???

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

=== Mac OS X share in QEMU ===

Command to sync:
rsync -a -L --no-p --no-g --chmod=ugo=rwX --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

Issues: TBD

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\127.0.0.1\meego /user:root password

== Other notes ==
=== Linker options for qmake ===

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

=== Required directories for compilation ===
I found that these directories are required in Qt-Creator to compile application:

/lib
/usr/lib
/usr/include
/usr/share

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format (qemu version 0.12.5)
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

Note: qemu-img version (0.14.50) from git repository is much faster - the same conversation takes only ~1m30s

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

SDK/Sysroot extension

2011-04-11T08:05:23Z

Edmondas: /* Windows share in QEMU */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]
* some PoF scripts on [https://www.gitorious.org/sysroot-extend Gitorious]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* ???

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\127.0.0.1\meego /user:root password

== Other notes ==
=== Linker options for qmake ===

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

=== Required directories for compilation ===
I found that these directories are required in Qt-Creator to compile application:

/lib
/usr/lib
/usr/include
/usr/share

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format (qemu version 0.12.5)
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

Note: qemu-img version (0.14.50) from git repository is much faster - the same conversation takes only ~1m30s

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

SDK/Sysroot extension

2011-04-07T07:55:34Z

Edmondas: /* Windows share in QEMU */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]
* some PoF scripts on [https://www.gitorious.org/sysroot-extend Gitorious]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* ???

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\127.0.0.1\meego /user:root password

== Other notes ==
=== Linker options for qmake ===

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

=== Required directories for compilation ===
I found that these directories are required in Qt-Creator to compile application:

/lib
/usr/lib
/usr/include
/usr/share

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format (qemu version 0.12.5)
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

Note: qemu-img version (0.14.50) from git repository is much faster - the same conversation takes only ~1m30s

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

SDK/Sysroot extension

2011-04-07T07:55:01Z

Edmondas: /* QEMU share on Windows */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]
* some PoF scripts on [https://www.gitorious.org/sysroot-extend Gitorious]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* ???

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

TBD

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\127.0.0.1\meego /user:root password

== Other notes ==
=== Linker options for qmake ===

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

=== Required directories for compilation ===
I found that these directories are required in Qt-Creator to compile application:

/lib
/usr/lib
/usr/include
/usr/share

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format (qemu version 0.12.5)
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

Note: qemu-img version (0.14.50) from git repository is much faster - the same conversation takes only ~1m30s

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

SDK/Sysroot extension

2011-04-07T07:52:39Z

Edmondas: /* Goals */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]
* some PoF scripts on [https://www.gitorious.org/sysroot-extend Gitorious]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* ???

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

TBD

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\1.2.3.4\meego /user:root password

TBD

== Other notes ==
=== Linker options for qmake ===

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

=== Required directories for compilation ===
I found that these directories are required in Qt-Creator to compile application:

/lib
/usr/lib
/usr/include
/usr/share

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format (qemu version 0.12.5)
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

Note: qemu-img version (0.14.50) from git repository is much faster - the same conversation takes only ~1m30s

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

SDK/Sysroot extension

2011-04-07T07:39:51Z

Edmondas: /* Linker options for qmake */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* ???

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

TBD

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\1.2.3.4\meego /user:root password

TBD

== Other notes ==
=== Linker options for qmake ===

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

=== Required directories for compilation ===
I found that these directories are required in Qt-Creator to compile application:

/lib
/usr/lib
/usr/include
/usr/share

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format (qemu version 0.12.5)
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

Note: qemu-img version (0.14.50) from git repository is much faster - the same conversation takes only ~1m30s

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

SDK/Sysroot extension

2011-04-06T12:01:51Z

Edmondas: /* Other */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* ???

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

TBD

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\1.2.3.4\meego /user:root password

TBD

== Linker options for qmake ==

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format (qemu version 0.12.5)
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

Note: qemu-img version (0.14.50) from git repository is much faster - the same conversation takes only ~1m30s

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

SDK/Sysroot extension

2011-04-06T11:50:21Z

Edmondas: /* Cons */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* ???

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

TBD

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\1.2.3.4\meego /user:root password

TBD

== Linker options for qmake ==

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

SDK/Sysroot extension

2011-04-06T07:57:32Z

Edmondas: /* Goals */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentations (4-8 slides max) about my findings:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* it's not clear how used prefix could affect libraries later
* works only on Linux

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

TBD

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\1.2.3.4\meego /user:root password

TBD

== Linker options for qmake ==

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]

SDK/Sysroot extension

2011-04-06T07:56:53Z

Edmondas: /* Goals */

THIS STUDY IS IN PROGRESS 

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

== Introduction ==

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

== Goals ==
* analyse possible ways to extend the sysroot, write pros & cons for every option
* check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
* create short presentation 4-8 slides about my findings
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]

== Definitions ==

Main assets are:
* sysroot - tree structure which contains files/directories from real environment
* library - 3rd party API required to get some functionality
* environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:
* core - contains only the most important apps/libs
* full - full environment

Libraries can be in a form of:
* RPM package - some existing library in MeeGo repository
* source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):
* QEMU machine - virtual machine with running system, which has network and virtual image access
* Device - real device, which has network access
* host system - computer with supported os
* sysroot - existing file tree on host system
* OBS - remote building service

== Options ==

For RPM packages
# Extract content of RPM to existing sysroot (like mad-admin xdpkg)
# Install RPM in chrooted environment
# Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
# Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:
# Use compilation prefix (--prefix)
# Chroot to existing sysroot, build and install library
# Build and install library on QEMU machine or device, copy files to sysroot (create/update)
# Build and install library on QEMU machine or device, mount QEMU disk as sysroot
# Generate your own sysroot with installed library (local OBS)

Copying options:
# Recursive copying (cp -r)
# Using rsync via ssh protocol
# Using rsync via rsync protocol
# Copying from mounted raw image
# Copying from mounted system via sshfs

== RPM packages ==

=== Extract RPM to sysroot ===
==== Description ====

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot:
rpm2cpio LIBRARY.rpm | cpio -idmv

''Note:'' probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device:
zypper install --download-only <package_name>

Packages will be placed (on device/vm) in
/var/cache/zypp/packages/

==== Pros ====
# quite easy way to install single rpm package (library with no dependencies)

==== Cons ====
# requires additional work if package has dependencies
# some packages have pre/post-install scripts
# this option is good only for prebuild packages but not to install library from source tarball
# requires to register package in package system

=== Install RPM inside chroot ===
==== Description ====
Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

'''Note''': this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device):
# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory:
/var/cache/zypp/packages/

copy all files to sysroot, e.g.
/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment:
$ sudo mic-chroot <path_to_sysroot>

install package manually:
$ sudo rpm -i <package>

exit from chrooted environment:
$ exit

==== Pros ====
* easy way to install existing package (library) or package built using OBS service

==== Cons ====
* need to download packages from QEMU/device before chrooting to sysroot
* sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
* mic-chroot works only on Linux

=== Install in QEMU/device, copy files ===
==== Description ====
Use ssh protocol and rsync to copy files from QEMU to host system.

''Idea:'' Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

''Note:'' Perhaps it's possible to improve rsync performance by changing buffer size, e.g.:
$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

'''Note:''' rsync exists by default only on handset qemu image

==== Pros ====
# No need for building sysroots with OBS
# No need for additional error prone tools for updating the sysroots (just use target package management system)
# It will save developer's time, because library will be compiled only once on QEMU or device

==== Cons ====
# Additional time for developer initially because the developer needs to boot Qemu or device for that
# QEMU has some issues with ATI/Nvidia cards (handset only?)
# there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
# Some files could change during rsync copying, e.g.
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

== Source tarballs ==

=== Use compilation prefix ===
==== Description ====
Use madde cross compilation target and installation prefix.

An example:
$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
/usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way:
mad -t <target> bash --noprofile --norc

==== Pros ====
* easy way to build and install using madde toolchains

==== Cons ====
* it's not clear how used prefix could affect libraries later
* works only on Linux

=== Chroot, build & install ===
==== Description ====
Fetch library source-code and build it in chrooted environment.

An example:
$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

==== Pros ====
* no need to have runtime (QEMU/device)

==== Cons ====
* library might have dependencies on other libs that are not in sysroot
* requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
* mic-chroot works only on Linux

=== Build & install on QEMU/device, copy files ===

Same as for [[#Install_in_QEMU.2Fdevice.2C_copy_files|RPM]]

== Benchmarks ==

=== Installing rpms and rsync on running QEMU vm ===

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

* ''JSON Glib interface library'', packages: json-glib json-glib-devel json-glib-doc
* ''wxWidgets library'', packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
* ''Boost library'', packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
* ''Apache Portable Runtime library'', packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
* ''Universally unique identifier library'', packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

''Note:'' packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''RPMs count'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
|-
| Initial sync
| -
| 1414 mb
| -
| 164 s
| -
| 180 s
| -
| 165 s
| -
| 169 s (2m49s)
|-
| json-glib
| 11
| 1455 mb (+41)
| 60 s
| 6 s
| 113 s
| 6 s
| 77 s
| 6 s
| 83 s (1m23s)
| 6 s
|-
| wx
| 35
| 1537 mb (+82)
| 152 s
| 7 s
| 293 s
| 8 s
| 177 s
| 7 s
| 207s (3m27s)
| 7 s
|-
| boost
| 18
| 1758 mb (+221)
| 275 s
| 22 s
| 269 s
| 23 s
| 262 s
| 19 s
| 268 s (4m28s)
| 21 s
|-
| apr
| 13
| 1771 mb (+13)
| 52 s
| 3 s
| 59 s
| 3 s
| 67 s
| 4 s
| 59 s
| 3 s
|-
| uuid
| 6
| 1771 mb (<+1)
| 13 s
| 2 s
| 12 s
| 1 s
| 17 s
| 2 s
| 14 s
| 2 s
|}

''Note:'' rpm installation time depends on network connectivity

''Note:'' vm startup & shutdown time is not counted

=== Installation and rsync with vm start/stop ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 181 s
| -
| 170 s
| -
| 181 s
| -
| 177 s (2m57s)
| ~8.0
|-
| json-glib
| 1455 mb (+41)
| 80 s
| 10 s
| 50 s
| 10 s
| 80 s
| 10 s
| 70 s (1m10s)
| 10 s
| ~4.1
|-
| wx
| 1537 mb (+82)
| 181 s
| 11 s
| 116 s
| 14 s
| 181 s
| 11 s
| 159 s (2m39s)
| 12 s
| ~6.8
|-
| boost
| 1758 mb (+221)
| 219 s
| 31 s
| 141 s
| 27 s
| 219 s
| 31 s
| 193 s (3m13s)
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 50 s
| 6 s
| 38 s
| 7 s
| 50 s
| 6 s
| 46 s
| 6 s
| ~2.2
|-
| uuid
| 1771 mb (<+1)
| 15 s
| 5 s
| 11 s
| 5 s
| 15 s
| 5 s
| 13 s
| 5 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing rpms and rsync in qemu with -nographics ===

Results table for netbook vm (stopping and starting vm before each installation):
{| cellpadding="5" cellspacing="0" border="1"
|- align="center"
| '''Library'''
| '''Used space'''
| '''1 test: install time'''
| '''1 test: sync time'''
| '''2 test: install time'''
| '''2 test: sync time'''
| '''3 test: install time'''
| '''3 test: sync time'''
| '''Average install time'''
| '''Average sync time'''
| '''sync speed mb/s'''
|-
| Initial sync
| 1414 mb
| -
| 172 s
| -
| 202 s
| -
| 193 s
| -
| 189 s (3m9s)
| ~7.5
|-
| json-glib
| 1455 mb (+41)
| 64 s
| 13 s
| 54 s
| 12 s
| 62 s
| 13 s
| 60 s
| 13 s
| ~3
|-
| wx
| 1537 mb (+82)
| 125 s
| 11 s
| 139 s
| 17 s
| 141 s
| 10 s
| 135 s
| 13 s
| ~6.3
|-
| boost
| 1758 mb (+221)
| 127 s
| 28 s
| 170 s
| 29 s
| 160 s
| 29 s
| 152 s
| 29 s
| ~7.6
|-
| apr
| 1771 mb (+13)
| 36 s
| 6 s
| 55 s
| 7 s
| 41 s
| 7 s
| 44 s
| 7 s
| ~1.9
|-
| uuid
| 1771 mb
| 12 s
| 5 s
| 21 s
| 6 s
| 12 s
| 6 s
| 15 s
| 6 s
| -
|}

''Note:'' vm startup & shutdown time is not counted

=== Installing from source ===

Untested

=== rsync on Windows OS ===

There is rsync version for Windows (uses cygwin libs) - [http://www.itefix.no/i2/node/10650 cwrsync]

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin):

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
-a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real 20m43.933s
user 1m23.660s
sys 4m26.541s

du -h --max-depth=0 sysroot
1.6G sysroot/

sync speed is only '''~1.4 MB/s'''

''Question'': why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

=== ssh vs rsync protocol ===
==== Description ====
Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands:
rsync -a \
--exclude '/proc' \
--exclude '/dev' \
--exclude '/sys' \
rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time:

$ time ./rsync-rsyncd.sh
real '''4m56.249s'''
user 0m8.997s
sys 0m23.113s

1.7G ./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol.
rsync + ssh ~3m 20s
rsync + rsyncd ~4m 56s
diff ~1m 36s

==== Cons ====
# need to configure and start an additional service
# slower than copying via ssh (?!)

=== qemu disk mounting options ===
==== Description ====
Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):
# mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
# [http://www.acc.umu.se/~bosse/ filedisk] (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
# hdiutil (Mac OS X) - untested
# vdfuse - fuse module uses the VirtualBox access library - untested
# [http://libguestfs.org/ libguestfs] tools for accessing and modifying virtual machine disk images, untested

==== Pros ====
# loop device mounting gives good performance

==== Cons ====
# loop device mounting is available only on Linux

=== remote system mounting options ===
==== Description ====
There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:
# sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed '''~4.8MB/s'''
# [http://code.google.com/p/dokan/ dokan] user-mode file system library for Windows - really bad speed '''~270KB/s'''

==== Pros ====
# available on different platforms (operating systems)

==== Cons ====
# Bad performance comparing to loop mounting and rsyncing
# Bad performance on Windows using current Dokan implementation

== Questions & Answers ==
=== how symlinks are managed on Windows ===
* on MinGW "ln -s" simply copies the file (the same for hard link)
* About symlinks in Cygwin, from Wikipedia [[:wikipedia:Symbolic link#Cygwin symbolic links|article]]: <blockquote><cite>Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.</cite></blockquote>
* An example screenshot how Cygwin symlink looks in Windows Explorer:
[[File:Cygwin_symlink.png|border]]

Tools to create symlinks on Windows file system:
* [http://ss64.com/nt/mklink.html mklink] - supported in Windows 7 and Windows Vista
* [http://ss64.com/nt/mklink.html linkd] - supported from Windows 2000 (Windows Resource Kit Tools)

=== rsync on mingw32 ===
* Can't be easily ported, because of
** Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
** Differences in threads and sockets, missing POSIX functions on Windows
** Conclusion - rsync is designed for Unix like systems
* Possible actions:
** study how applications are ported to mingw using [http://www.mingw.org/node/48 msysDTK], [http://www.mingw.org/wiki/mingwPORT mingwPORT], [http://www.mingw.org/wiki/pthreads_library pthread library], [http://www.mingw.org/wiki/Use_the_thread_library thread library], [http://www.mingw.org/wiki/HOWTO HOWTOs], [http://www.mingw.org/wiki/Community_Supplied_Links other libraries]

==== other native implementations for Windows ====
* [http://rsyncwin32.sourceforge.net/ RsyncWin32]
** alpha version, there are bug reports about instability, last release in 2005
** based on rsync-2.6.3, current version 3.0.7
** implements POSIX functions, resolves issues with Windows threads and sockets
* [http://opensource.se/projects/rsync-bc/index.html rsync-bc]
** only allows to push data to a server
** based on rsync-2.5.6

=== alternatives to rsync ===

[http://www.cis.upenn.edu/~bcpierce/unison/ unison] - a file-synchronization tool for Unix and Windows
* file-synchronization tool for Unix and Windows
* uses the rsync algorithm
* communicates over the TCP/IP protocol
* written in the Objective Caml language ?!

[http://zsync.moria.org.uk/ zsync] - rsync over HTTP
* designed for file distribution, with one file on a server to be distributed to thousands of downloaders
* uses the same algorithm as rsync

[http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx robocopy] (standard feature of Windows Vista, 7 and Server 2008)
command-line directory replication command

TODO: try to use unison, check performance

=== samba on qemu and share mapping ===

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7).
1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help:
sc stop Browser
sc stop LanmanServer

need to disable service and restart system:
sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed:
sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports:
hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk
net use m: \\127.0.0.1\disk

map disk to sysroot? (Use [http://ss64.com/nt/mklink.html mklink]):
mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

=== qemu disk mapping and mounting on Windows ===
Idea: Try to mount QEMU image using [http://www.acc.umu.se/~bosse/ filedisk] and then map partition using [http://www.ext2fsd.com/ ext2 driver]

It's pretty simple to mount QEMU image:
filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:
* [http://www.ext2fsd.com/ Ext2Fsd] - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
* [http://www.chrysocome.net/explore2fs Explore2fs] - GUI explorer tool for accessing ext2 and ext3 filesystems.
** this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
* [http://ext2read.blogspot.com/ Ext2read] - an explorer like utility to explore ext2/ext3/ext4 files.
* [http://fs-driver.org/ Ext2 IFS] - provides full access to Linux Ext2 volumes.
** no support for Windows 7

=== qcow2 image mounting options ===
Tools to access qcow2 format images:
* qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
** Question: does it works on Windows, Mac OS X?
** Documentation: [http://blog.loftninjas.org/2008/10/27/mounting-kvm-qcow2-qemu-disk-images/ Mounting KVM qcow2 qemu disk images], [http://blogs.gnome.org/muelli/2010/03/mounting-qemu-qcow2-image-using-nbd/ Mounting QEmu qcow2 Image using NBD]
* [http://sourceforge.net/projects/fuseqemu/ fuseqemu] - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
* [http://libguestfs.org/ libguestfs] - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

=== Windows share in QEMU ===
Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing:
smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting:
mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync:
rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

TBD

=== QEMU share on Windows ===

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed)
export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools:
zypper install samba samba-client

Create new user and set password
smbpasswd -a root

Configuration file is in:
/etc/samba/smb.conf

Create share with access to filesystem:
[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server:
/etc/init.d/smb start

Testing (from cygwin or QEMU):
smbclient -U root -L 127.0.0.1

Map disk:
net use M: \\1.2.3.4\meego /user:root password

TBD

== Linker options for qmake ==

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library:
CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib:
CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

== Used tools ==

''time'' utility was used to measure performance, e.g.:

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime
.....9....8....7....6....5....4....3....2....1....0
Creating missing links... done.

real 2m54.520s
user 2m15.132s
sys 0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

== Used hardware ==

Description of hardware used in this study

Linux machine:
* Dell Latitude E6510
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
* RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
* HDD WD 250GB 7200RPM (encrypted fs)
* Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
* BogoMips 4787.89

Windows machine:
* HP EliteBook 8440p
* Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
* RAM 4GB, HDD 7200 rpm (encrypted fs)
* Windows 7 Enterprise (32-bit)
* System Base Score 4.5

== Other ==

QEMU raw image (3GB) conversion time to qcow2 format
time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real 25m54.360s
user 0m2.596s
sys 0m17.045s

Size after conversion - 1.8GB

== Conclusions ==

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

== Time Estimate / Schedule ==
At the moment there is no target milestone for this feature.

== Links ==
* [http://samba.anu.edu.au/rsync/documentation.html rsync documentation]
* [[SDK/Docs/1.1/Getting_started_with_the_MeeGo_SDK_for_Linux|MeeGo SDK on Linux]]
* [[SDK/MeeGo_SDK_for_Windows|MeeGo SDK for Windows]]
* [[SDK/Docs/1.1/Building a MeeGo chroot on Linux|Building a MeeGo chroot on Linux]]
* [http://talk.maemo.org/showthread.php?t=48536 Maemo talk - Madde: How to build/manage third-party]
* [http://forum.meego.com/showthread.php?t=2709 A little "How to work effictient with Meego SDK"]
* [[:wikipedia:Symbolic link|Symbolic link]]
* [http://wiki.alfresco.com/wiki/File_Server_Configuration#Native_SMB_collisions Native SMB collisions]
* [http://support.microsoft.com/kb/314053 TCP/IP and NBT configuration parameters for Windows XP]
* [http://leenux.org.uk/lpackham/smbclient/ Samba Client for Win32 (for cygwin)]
* [http://www.blisstonia.com/eolson/notes/smboverssh.php Sharing (tunneling) Samba/CIFS/SMB file systems over SSH]
* [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=4494 Building Qemu on Mac OS X on different architectures]