Meego Wiki
Views

SDK/Sysroot extension

From MeeGo wiki
< SDK(Difference between revisions)
Jump to: navigation, search
m (→Description)
m
 
(2 intermediate revisions not shown)
Line 1: Line 1:
-
<span style="color:#ff0000"> THIS STUDY IS IN PROGRESS </span>
+
<span style="color:#ff0000"> </span>
Author: Edmondas Girkantas
Author: Edmondas Girkantas
Line 17: Line 17:
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 1st presentation - [[Media:Sysroot-extension.pdf|QEMU+rsync]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
** 2nd presentation - [[Media:Sysroot-extension2.pdf|Windows and other sync. tools]]
-
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]]
+
** 3rd presentation - [[Media:Sysroot-extension3.pdf|Samba and shared folders]] (Updated on 28th April)
* some PoF scripts on [https://www.gitorious.org/sysroot-extend Gitorious]
* some PoF scripts on [https://www.gitorious.org/sysroot-extend Gitorious]
Line 69: Line 69:
==== Description ====
==== Description ====
-
'''Note:''' this feature is already implemented.
+
'''Note:''' this feature is already implemented in [https://meego.gitorious.org/meego-developer-tools/madde/blobs/master/src/madlib/xrpm.pl madde].
Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)
Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

Latest revision as of 12:29, 9 June 2011

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

Contents

Introduction

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

Goals

  • analyse possible ways to extend the sysroot, write pros & cons for every option
  • check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
  • create short presentations (4-8 slides max) about my findings:
  • some PoF scripts on Gitorious

Definitions

Main assets are:

  • sysroot - tree structure which contains files/directories from real environment
  • library - 3rd party API required to get some functionality
  • environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:

  • core - contains only the most important apps/libs
  • full - full environment

Libraries can be in a form of:

  • RPM package - some existing library in MeeGo repository
  • source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):

  • QEMU machine - virtual machine with running system, which has network and virtual image access
  • Device - real device, which has network access
  • host system - computer with supported os
  • sysroot - existing file tree on host system
  • OBS - remote building service

Options

For RPM packages

  1. Extract content of RPM to existing sysroot (like mad-admin xdpkg)
  2. Install RPM in chrooted environment
  3. Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
  4. Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:

  1. Use compilation prefix (--prefix)
  2. Chroot to existing sysroot, build and install library
  3. Build and install library on QEMU machine or device, copy files to sysroot (create/update)
  4. Build and install library on QEMU machine or device, mount QEMU disk as sysroot
  5. Generate your own sysroot with installed library (local OBS)

Copying options:

  1. Recursive copying (cp -r)
  2. Using rsync via ssh protocol
  3. Using rsync via rsync protocol
  4. Copying from mounted raw image
  5. Copying from mounted system via sshfs

RPM packages

Extract RPM to sysroot

Description

Note: this feature is already implemented in madde.

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot: 

rpm2cpio LIBRARY.rpm | cpio -idmv

Note: probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device: 

zypper install --download-only <package_name>

Packages will be placed (on device/vm) in 

/var/cache/zypp/packages/

Pros

  1. quite easy way to install single rpm package (library with no dependencies)

Cons

  1. requires additional work if package has dependencies
  2. some packages have pre/post-install scripts
  3. this option is good only for prebuild packages but not to install library from source tarball
  4. requires to register package in package system

Install RPM inside chroot

Description

Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

Note: this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device): 

# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory: 

/var/cache/zypp/packages/

copy all files to sysroot, e.g. 

/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment: 

$ sudo mic-chroot <path_to_sysroot>

install package manually: 

$ sudo rpm -i <package>

exit from chrooted environment: 

$ exit

Pros

  • easy way to install existing package (library) or package built using OBS service

Cons

  • need to download packages from QEMU/device before chrooting to sysroot
  • sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
  • mic-chroot works only on Linux

Install in QEMU/device, copy files

Description

Use ssh protocol and rsync to copy files from QEMU to host system.

Idea: Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

Note: Perhaps it's possible to improve rsync performance by changing buffer size, e.g.: 

$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

Note: rsync exists by default only on handset qemu image

Pros

  1. No need for building sysroots with OBS
  2. No need for additional error prone tools for updating the sysroots (just use target package management system)
  3. It will save developer's time, because library will be compiled only once on QEMU or device

Cons

  1. Additional time for developer initially because the developer needs to boot Qemu or device for that
  2. QEMU has some issues with ATI/Nvidia cards (handset only?)
  3. there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
  4. Some files could change during rsync copying, e.g. 
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

Source tarballs

Use compilation prefix

Description

Use madde cross compilation target and installation prefix.

An example: 

$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
    /usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way: 

mad -t <target> bash --noprofile --norc

Pros

  • easy way to build and install using madde toolchains

Cons

  •  ???

Chroot, build & install

Description

Fetch library source-code and build it in chrooted environment.

An example: 

$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

Pros

  • no need to have runtime (QEMU/device)

Cons

  • library might have dependencies on other libs that are not in sysroot
  • requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
  • mic-chroot works only on Linux

Build & install on QEMU/device, copy files

Same as for RPM

Benchmarks

Installing rpms and rsync on running QEMU vm

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

  • JSON Glib interface library, packages: json-glib json-glib-devel json-glib-doc
  • wxWidgets library, packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
  • Boost library, packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
  • Apache Portable Runtime library, packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
  • Universally unique identifier library, packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

Note: packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):

Library RPMs count Used space 1 test: install time 1 test: sync time 2 test: install time 2 test: sync time 3 test: install time 3 test: sync time Average install time Average sync time
Initial sync - 1414 mb - 164 s - 180 s - 165 s - 169 s (2m49s)
json-glib 11 1455 mb (+41) 60 s 6 s 113 s 6 s 77 s 6 s 83 s (1m23s) 6 s
wx 35 1537 mb (+82) 152 s 7 s 293 s 8 s 177 s 7 s 207s (3m27s) 7 s
boost 18 1758 mb (+221) 275 s 22 s 269 s 23 s 262 s 19 s 268 s (4m28s) 21 s
apr 13 1771 mb (+13) 52 s 3 s 59 s 3 s 67 s 4 s 59 s 3 s
uuid 6 1771 mb (<+1) 13 s 2 s 12 s 1 s 17 s 2 s 14 s 2 s

Note: rpm installation time depends on network connectivity

Note: vm startup & shutdown time is not counted

Installation and rsync with vm start/stop

Results table for netbook vm (stopping and starting vm before each installation):

Library Used space 1 test: install time 1 test: sync time 2 test: install time 2 test: sync time 3 test: install time 3 test: sync time Average install time Average sync time sync speed mb/s
Initial sync 1414 mb - 181 s - 170 s - 181 s - 177 s (2m57s) ~8.0
json-glib 1455 mb (+41) 80 s 10 s 50 s 10 s 80 s 10 s 70 s (1m10s) 10 s ~4.1
wx 1537 mb (+82) 181 s 11 s 116 s 14 s 181 s 11 s 159 s (2m39s) 12 s ~6.8
boost 1758 mb (+221) 219 s 31 s 141 s 27 s 219 s 31 s 193 s (3m13s) 29 s ~7.6
apr 1771 mb (+13) 50 s 6 s 38 s 7 s 50 s 6 s 46 s 6 s ~2.2
uuid 1771 mb (<+1) 15 s 5 s 11 s 5 s 15 s 5 s 13 s 5 s -

Note: vm startup & shutdown time is not counted

Installing rpms and rsync in qemu with -nographics

Results table for netbook vm (stopping and starting vm before each installation):

Library Used space 1 test: install time 1 test: sync time 2 test: install time 2 test: sync time 3 test: install time 3 test: sync time Average install time Average sync time sync speed mb/s
Initial sync 1414 mb - 172 s - 202 s - 193 s - 189 s (3m9s) ~7.5
json-glib 1455 mb (+41) 64 s 13 s 54 s 12 s 62 s 13 s 60 s 13 s ~3
wx 1537 mb (+82) 125 s 11 s 139 s 17 s 141 s 10 s 135 s 13 s ~6.3
boost 1758 mb (+221) 127 s 28 s 170 s 29 s 160 s 29 s 152 s 29 s ~7.6
apr 1771 mb (+13) 36 s 6 s 55 s 7 s 41 s 7 s 44 s 7 s ~1.9
uuid 1771 mb 12 s 5 s 21 s 6 s 12 s 6 s 15 s 6 s -

Note: vm startup & shutdown time is not counted

Installing from source

Untested

rsync on Windows OS

There is rsync version for Windows (uses cygwin libs) - cwrsync

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin): 

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
   -a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real   20m43.933s
user   1m23.660s
sys    4m26.541s

du -h --max-depth=0 sysroot
1.6G   sysroot/

sync speed is only ~1.4 MB/s

Question: why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

ssh vs rsync protocol

Description

Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands: 

rsync -a \
       --exclude '/proc' \
       --exclude '/dev' \
       --exclude '/sys' \
       rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time: 

$ time ./rsync-rsyncd.sh 
real	4m56.249s 
user	0m8.997s 
sys	0m23.113s 

1.7G	./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol. 

rsync + ssh     ~3m 20s
rsync + rsyncd  ~4m 56s
diff            ~1m 36s

Cons

  1. need to configure and start an additional service
  2. slower than copying via ssh (?!)

qemu disk mounting options

Description

Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):

  1. mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
  2. filedisk (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
  3. hdiutil (Mac OS X) - untested
  4. vdfuse - fuse module uses the VirtualBox access library - untested
  5. libguestfs tools for accessing and modifying virtual machine disk images, untested

Pros

  1. loop device mounting gives good performance

Cons

  1. loop device mounting is available only on Linux

remote system mounting options

Description

There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:

  1. sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed ~4.8MB/s
  2. dokan user-mode file system library for Windows - really bad speed ~270KB/s

Pros

  1. available on different platforms (operating systems)

Cons

  1. Bad performance comparing to loop mounting and rsyncing
  2. Bad performance on Windows using current Dokan implementation

Questions & Answers

how symlinks are managed on Windows

  • on MinGW "ln -s" simply copies the file (the same for hard link)
  • About symlinks in Cygwin, from Wikipedia article:
    Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.
  • An example screenshot how Cygwin symlink looks in Windows Explorer:

Cygwin symlink.png

Tools to create symlinks on Windows file system:

  • mklink - supported in Windows 7 and Windows Vista
  • linkd - supported from Windows 2000 (Windows Resource Kit Tools)

rsync on mingw32

  • Can't be easily ported, because of
    • Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
    • Differences in threads and sockets, missing POSIX functions on Windows
    • Conclusion - rsync is designed for Unix like systems
  • Possible actions:

other native implementations for Windows

  • RsyncWin32
    • alpha version, there are bug reports about instability, last release in 2005
    • based on rsync-2.6.3, current version 3.0.7
    • implements POSIX functions, resolves issues with Windows threads and sockets
  • rsync-bc
    • only allows to push data to a server
    • based on rsync-2.5.6

alternatives to rsync

unison - a file-synchronization tool for Unix and Windows

  • file-synchronization tool for Unix and Windows
  • uses the rsync algorithm
  • communicates over the TCP/IP protocol
  • written in the Objective Caml language ?!

zsync - rsync over HTTP

  • designed for file distribution, with one file on a server to be distributed to thousands of downloaders
  • uses the same algorithm as rsync

robocopy (standard feature of Windows Vista, 7 and Server 2008) command-line directory replication command

TODO: try to use unison, check performance

samba on qemu and share mapping

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

need to disable some services (on Windows 7). 1st issue - LanmanServer binds to 0.0.0.0:445, there is no way to bind to another interface

manual stopping will not help: 

sc stop Browser
sc stop LanmanServer

need to disable service and restart system: 

sc config LanmanServer start= disabled
shutdown -r -t 01

restore service if needed: 

sc config LanmanServer start= auto

start qemu with forwarded 139 & 445 ports: 

hostfwd=tcp:127.0.0.1:139-:139,hostfwd=tcp:127.0.0.1:445-:445

mount share to disk 

net use m: \\127.0.0.1\disk

map disk to sysroot? (Use mklink): 

mklink /D c:\MeeGoSDK_1.1\MADDE\sysroots\SYSROOT_NAME m:

qemu disk mapping and mounting on Windows

Idea: Try to mount QEMU image using filedisk and then map partition using ext2 driver

It's pretty simple to mount QEMU image: 

filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:

  • Ext2Fsd - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
  • Explore2fs - GUI explorer tool for accessing ext2 and ext3 filesystems.
    • this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
  • Ext2read - an explorer like utility to explore ext2/ext3/ext4 files.
  • Ext2 IFS - provides full access to Linux Ext2 volumes.
    • no support for Windows 7

qcow2 image mounting options

Tools to access qcow2 format images:

  • qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
  • fuseqemu - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
  • libguestfs - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

Windows share in QEMU

Idea: mount shared folder from Windows onto QEMU then rsync files

Command for testing: 

smbclient -U DOMAIN\\user \\\\10.0.2.2\\SHARE

Command for mounting: 

mount -t cifs -o username=DOMAIN\\user,password=PASSWORD //10.0.2.2/SERVICE MOUNT_POINT

Note: 10.0.2.2 is special IP address used in QEMU for networking

Command to sync: 

rsync -a -L --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

Note: symlinks transformed into referent file/dir, block or character special files are not copied (not required for sysroot)

rsync speed ~600KB/s

Mac OS X share in QEMU

Command to sync: 

rsync -a -L --no-p --no-g --chmod=ugo=rwX --exclude '/proc' --exclude '/dev' --exclude '/mnt' --exclude '/sys' / /MOUNT_POINT

rsync speed ~900KB/s

Issues: rsync can't change ownership (chown) of copied files while using native Windows sharing service on Mac OS X, e.g.: 

rsync: chown "/mnt/bin" failed: Permission denied (13)

I can't test it fully (using Qt-creator) because MeeGo SDK porting to Mac OS X is still in progress.

QEMU share on Windows

Idea: use samba to create Windows share in QEMU, then use QEMU port forwarding feature and map it on Windows

Required steps:

Set proxy server (if needed) 

export http_proxy=XXX.XXX.XXX.XXX:8080

Install Samba tools: 

zypper install samba samba-client

Create new user and set password 

smbpasswd -a root

Configuration file is in: 

/etc/samba/smb.conf

Create share with access to filesystem: 

[meego]
comment = Meego
path = /
writable = yes
write list = root

Start Samba server: 

/etc/init.d/smb start

Testing (from cygwin or QEMU): 

smbclient -U root -L 127.0.0.1

Map disk: 

net use M: \\127.0.0.1\meego /user:root password

Other notes

Linker options for qmake

To build application against some external non-qt library (uses pkg-config) you need to set additional qmake options in project file (project_name.pro):

For example for C uuid library: 

CONFIG += link_pkgconfig
PKGCONFIG += uuid

For C++ uuid lib: 

CONFIG += link_pkgconfig
PKGCONFIG += ossp-uuid
LIBS += -lossp-uuid++

Required directories for compilation

I found that these directories are required in Qt-Creator to compile application: 

/lib
/usr/lib
/usr/include
/usr/share

Used tools

time utility was used to measure performance, e.g.: 

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime 
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime 
.....9....8....7....6....5....4....3....2....1....0 
Creating missing links... done. 

real	2m54.520s 
user	2m15.132s 
sys	0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

Used hardware

Description of hardware used in this study

Linux machine:

  • Dell Latitude E6510
  • Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
  • RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
  • HDD WD 250GB 7200RPM (encrypted fs)
  • Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
  • BogoMips 4787.89

Windows machine:

  • HP EliteBook 8440p
  • Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
  • RAM 4GB, HDD 7200 rpm (encrypted fs)
  • Windows 7 Enterprise (32-bit)
  • System Base Score 4.5

Other

QEMU raw image (3GB) conversion time to qcow2 format (qemu version 0.12.5) 

time qemu-img convert -O qcow2 meego-netbook-ia32-qemu-1.1.20110110.1049-sda.raw meego.qcow2

real	25m54.360s
user	0m2.596s
sys	0m17.045s

Size after conversion - 1.8GB

Note: qemu-img version (0.14.50) from git repository is much faster - the same conversation takes only ~1m30s

Conclusions

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

Time Estimate / Schedule

At the moment there is no target milestone for this feature.

Links

Retrieved from "http://wiki.meego.com/SDK/Sysroot_extension"
Personal tools