Meego Wiki
Views

SDK/Sysroot extension

From MeeGo wiki
< SDK
Revision as of 10:32, 17 March 2011 by Edmondas (Talk | contribs)
Jump to: navigation, search

THIS STUDY IS IN PROGRESS

Author: Edmondas Girkantas

Interested parties: Veli Kaksonen

Contents

Introduction

The developer shall be able to extend the sysroot with additional libraries.

See http://bugs.meego.com/show_bug.cgi?id=9427

Goals

  • analyse possible ways to extend the sysroot, write pros & cons for every option
  • check option availability on Linux and Windows platforms; later on Mac OS X (Note: there is no MeeGo SDK installer for Mac OS X yet)
  • create short presentation 4-8 slides about my findings
    • 1st presentation - QEMU+rsync
    • 2nd presentation - Windows and other sync. tools

Definitions

Main assets are:

  • sysroot - tree structure which contains files/directories from real environment
  • library - 3rd party API required to get some functionality
  • environment - target system which is used to install additional library and create new sysroot (if needed)

Sysroots can be in a form of:

  • core - contains only the most important apps/libs
  • full - full environment

Libraries can be in a form of:

  • RPM package - some existing library in MeeGo repository
  • source tarball - some 3rd party library, not available in MeeGo

Possible environments (that can be used to build sysroot and install library):

  • QEMU machine - virtual machine with running system, which has network and virtual image access
  • Device - real device, which has network access
  • host system - computer with supported os
  • sysroot - existing file tree on host system
  • OBS - remote building service

Options

For RPM packages

  1. Extract content of RPM to existing sysroot (like mad-admin xdpkg)
  2. Install RPM in chrooted environment
  3. Install RPM into running QEMU machine or device, copy files to sysroot (create/update)
  4. Install RPM into running QEMU machine or device, mount QEMU disk as sysroot

For source tarballs:

  1. Use compilation prefix (--prefix)
  2. Chroot to existing sysroot, build and install library
  3. Build and install library on QEMU machine or device, copy files to sysroot (create/update)
  4. Build and install library on QEMU machine or device, mount QEMU disk as sysroot
  5. Generate your own sysroot with installed library (local OBS)

Copying options:

  1. Recursive copying (cp -r)
  2. Using rsync via ssh protocol
  3. Using rsync via rsync protocol
  4. Copying from mounted raw image
  5. Copying from mounted system via sshfs

RPM packages

Extract RPM to sysroot

Description

Implement extra rpm package (mad-admin xrpm) installation/removal interface (like existing xdpkg)

It is possible to download single rpm package and extract content to sysroot: 

rpm2cpio LIBRARY.rpm | cpio -idmv

Note: probably need to create symlinks from /usr/lib to /lib

If package has other dependencies then it is possible to use QEMU/device: 

zypper install --download-only <package_name>

Packages will be placed (on device/vm) in 

/var/cache/zypp/packages/

Pros

  1. quite easy way to install single rpm package (library with no dependencies)

Cons

  1. requires additional work if package has dependencies
  2. some packages have pre/post-install scripts
  3. this option is good only for prebuild packages but not to install library from source tarball
  4. requires to register package in package system
  5. currently works only on Linux

Install RPM inside chroot

Description

Download existing library (rpm) and put it to sysroot, then use mic-chroot to enter sysroot environment.

Note: this is only for meege-core-* sysroots, because zypper is available on other sysroots

Fetch prebuilt package (on MeeGo emulator/device): 

# zypper install --download-only <package_name>

The package and all dependencies will be stored (on Linux) in this directory: 

/var/cache/zypp/packages/

copy all files to sysroot, e.g. 

/usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs/

enter sysroot environment: 

$ sudo mic-chroot <path_to_sysroot>

install package manually: 

$ sudo rpm -i <package>

exit from chrooted environment: 

$ exit

Pros

  • easy way to install existing package (library) or package built using OBS service

Cons

  • need to download packages from QEMU/device before chrooting to sysroot
  • sysroot (meego-core-*) could have zypper by default, then it would be possible to extend sysroot without QEMU/device
  • mic-chroot works only on Linux

Install in QEMU/device, copy files

Description

Use ssh protocol and rsync to copy files from QEMU to host system.

Idea: Maybe we don't need to sync full filesystem, in most cases /usr/lib and /usr/include are enough

Note: Perhaps it's possible to improve rsync performance by changing buffer size, e.g.: 

$ rsync -a --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 ...

Note: rsync exists by default only on handset qemu image

Pros

  1. No need for building sysroots with OBS
  2. No need for additional error prone tools for updating the sysroots (just use target package management system)
  3. It will save developer's time, because library will be compiled only once on QEMU or device

Cons

  1. Additional time for developer initially because the developer needs to boot Qemu or device for that
  2. QEMU has some issues with ATI/Nvidia cards (handset only?)
  3. there is no native version of rsync for Windows, existing tools use cygwin, performance is bad
  4. Some files could change during rsync copying, e.g. 
file has vanished: "/home/meego/.cache/tracker/fulltext.db-journal"
file has vanished: "/home/meego/.cache/tracker/meta.db-journal"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1526) [generator=3.0.7]

Source tarballs

Use compilation prefix

Description

Use madde cross compilation target and installation prefix.

An example: 

$ wget http://sqlite.org/sqlite-autoconf-3070500.tar.gz
$ tar xvzf sqlite-autoconf-3070500.tar.gz
$ cd sqlite-autoconf-3070500/
$ mad -t meego-handset-ia32-1.1.2 ./configure --prefix \
    /usr/lib/madde/linux-i686/sysroots/meego-handset-ia32-madde-sysroot-1.1.20110106.1204-fs/
$ mad -t meego-handset-ia32-1.1.2 make
$ sudo mad -t meego-handset-ia32-1.1.2 make install

Other interesting way: 

mad -t <target> bash --noprofile --norc

Pros

  • easy way to build and install using madde toolchains

Cons

  • it's not clear how used prefix could affect libraries later
  • works only on Linux

Chroot, build & install

Description

Fetch library source-code and build it in chrooted environment.

An example: 

$ sudo mic-chroot /usr/lib/madde/linux-i686/sysroots/meego-core-ia32-madde-sysroot-1.1-fs
# cd /root
# curl -o sqlite-autoconf-3070500.tar.gz http://sqlite.org/sqlite-autoconf-3070500.tar.gz
# tar xvzf sqlite-autoconf-3070500.tar.gz
# cd sqlite-autoconf-3070500
# ./configure --prefix /usr
# make && make install
# exit

Pros

  • no need to have runtime (QEMU/device)

Cons

  • library might have dependencies on other libs that are not in sysroot
  • requires manual work or an additional build tool (like OBS) which will extract, configure (if needed), build and install library
  • mic-chroot works only on Linux

Build & install on QEMU/device, copy files

Same as for RPM

Benchmarks

Installing rpms and rsync on running QEMU vm

Testing plan - install 5 different libraries, measure installation and synchronisation time.

List of libraries used in this test:

  • JSON Glib interface library, packages: json-glib json-glib-devel json-glib-doc
  • wxWidgets library, packages: wxBase wxGTK wxGTK-devel wxGTK-gl wxGTK-media
  • Boost library, packages: boost boost-date-time boost-devel boost-doc boost-filesystem boost-graph boost-iostreams boost-math boost-program-options boost-python boost-regex boost-serialization boost-signals boost-static boost-system boost-test boost-thread boost-wave
  • Apache Portable Runtime library, packages: apr apr-devel apr-util apr-util-devel apr-util-ldap apr-util-sqlite
  • Universally unique identifier library, packages: uuid uuid-c++ uuid-c++-devel uuid-dce uuid-dce-devel uuid-devel

Note: packages are listed without dependencies

Results table for netbook vm (without stopping vm every time before installing new library, install/sync times is better because of os caching):

Library RPMs count Used space 1 test: install time 1 test: sync time 2 test: install time 2 test: sync time 3 test: install time 3 test: sync time Average install time Average sync time
Initial sync - 1414 mb - 164 s - 180 s - 165 s - 169 s (2m49s)
json-glib 11 1455 mb (+41) 60 s 6 s 113 s 6 s 77 s 6 s 83 s (1m23s) 6 s
wx 35 1537 mb (+82) 152 s 7 s 293 s 8 s 177 s 7 s 207s (3m27s) 7 s
boost 18 1758 mb (+221) 275 s 22 s 269 s 23 s 262 s 19 s 268 s (4m28s) 21 s
apr 13 1771 mb (+13) 52 s 3 s 59 s 3 s 67 s 4 s 59 s 3 s
uuid 6 1771 mb (<+1) 13 s 2 s 12 s 1 s 17 s 2 s 14 s 2 s

Note: rpm installation time depends on network connectivity

Note: vm startup & shutdown time is not counted

Installation and rsync with vm start/stop

Results table for netbook vm (stopping and starting vm before each installation):

Library Used space 1 test: install time 1 test: sync time 2 test: install time 2 test: sync time 3 test: install time 3 test: sync time Average install time Average sync time sync speed mb/s
Initial sync 1414 mb - 181 s - 170 s - 181 s - 177 s (2m57s) ~8.0
json-glib 1455 mb (+41) 80 s 10 s 50 s 10 s 80 s 10 s 70 s (1m10s) 10 s ~4.1
wx 1537 mb (+82) 181 s 11 s 116 s 14 s 181 s 11 s 159 s (2m39s) 12 s ~6.8
boost 1758 mb (+221) 219 s 31 s 141 s 27 s 219 s 31 s 193 s (3m13s) 29 s ~7.6
apr 1771 mb (+13) 50 s 6 s 38 s 7 s 50 s 6 s 46 s 6 s ~2.2
uuid 1771 mb (<+1) 15 s 5 s 11 s 5 s 15 s 5 s 13 s 5 s -

Note: vm startup & shutdown time is not counted

Installing rpms and rsync in qemu with -nographics

Results table for netbook vm (stopping and starting vm before each installation):

Library Used space 1 test: install time 1 test: sync time 2 test: install time 2 test: sync time 3 test: install time 3 test: sync time Average install time Average sync time sync speed mb/s
Initial sync 1414 mb - 172 s - 202 s - 193 s - 189 s (3m9s) ~7.5
json-glib 1455 mb (+41) 64 s 13 s 54 s 12 s 62 s 13 s 60 s 13 s ~3
wx 1537 mb (+82) 125 s 11 s 139 s 17 s 141 s 10 s 135 s 13 s ~6.3
boost 1758 mb (+221) 127 s 28 s 170 s 29 s 160 s 29 s 152 s 29 s ~7.6
apr 1771 mb (+13) 36 s 6 s 55 s 7 s 41 s 7 s 44 s 7 s ~1.9
uuid 1771 mb 12 s 5 s 21 s 6 s 12 s 6 s 15 s 6 s -

Note: vm startup & shutdown time is not counted

Installing from source

Untested

rsync on Windows OS

There is rsync version for Windows (uses cygwin libs) - cwrsync

Another option is to install full cygwin environment, which has rsync package

Performance of rsync on Windows OS is really bad (from cygwin): 

time rsync --exclude '/proc' --exclude '/dev' --exclude '/sys' \
   -a -e 'ssh -p 6666' root@127.0.0.1:/ sysroot
real   20m43.933s
user   1m23.660s
sys    4m26.541s

du -h --max-depth=0 sysroot
1.6G   sysroot/

sync speed is only ~1.4 MB/s

Question: why it is so slow? Possible answers: Antivirus / SSH protocol / Cygwin / Other?

ssh vs rsync protocol

Description

Use rsync (TCP/873) protocol (rsync --daemon) to copy files from QEMU to host system.

Used commands: 

rsync -a \
       --exclude '/proc' \
       --exclude '/dev' \
       --exclude '/sys' \
       rsync://root@127.0.0.1:1873/rsync sysroot
du -h --max-depth=0 ./sysroot/

Initial copy time: 

$ time ./rsync-rsyncd.sh 
real	4m56.249s 
user	0m8.997s 
sys	0m23.113s 

1.7G	./sysroot/

I found that initial copying of files using rsync protocol is slower than using ssh protocol. 

rsync + ssh     ~3m 20s
rsync + rsyncd  ~4m 56s
diff            ~1m 36s

Cons

  1. need to configure and start an additional service
  2. slower than copying via ssh (?!)

qemu disk mounting options

Description

Mount existing QEMU image (with installed library) then copy files (changes) to selected sysroot.

Options to install library - in a running QEMU machine or using chroot.

Tools for mounting QEMU images (ext3 is used):

  1. mount -o loop,offset=512 image.raw /mount_point (Linux) - good copying speed ~10MB/s on testing machine (
  2. filedisk (alternative for windows) - just for mounting raw image, then you need to use additional tool to read ext3 partition
  3. hdiutil (Mac OS X) - untested
  4. vdfuse - fuse module uses the VirtualBox access library - untested
  5. libguestfs tools for accessing and modifying virtual machine disk images, untested

Pros

  1. loop device mounting gives good performance

Cons

  1. loop device mounting is available only on Linux

remote system mounting options

Description

There are different tools to mount remote file system (e.g. using ssh protocol)

Tools for mounting remote file systems via ssh:

  1. sshfs (e.g sudo sshfs -d -p 6666 root@127.0.0.1:/ /mnt) - copying speed ~4.8MB/s
  2. dokan user-mode file system library for Windows - really bad speed ~270KB/s

Pros

  1. available on different platforms (operating systems)

Cons

  1. Bad performance comparing to loop mounting and rsyncing
  2. Bad performance on Windows using current Dokan implementation

Questions & Answers

how symlinks are managed on Windows

  • on MinGW "ln -s" simply copies the file (the same for hard link)
  • About symlinks in Cygwin, from Wikipedia article:
    Cygwin simulates POSIX-compliant symbolic links in the Microsoft Windows file system. It uses identical programming and user utility interfaces as Unix (see above), but creates Windows shortcuts (.lnk files) with additional information used by Cygwin at the time of symlink resolution. Cygwin symlinks are compliant with both Windows and POSIX standards. Some differences exist, however. Cygwin has no way to specify shortcut-related information - such as working directory or icon - as there is no place for such parameters in ln -s command. To create standard Microsoft .lnk files Cygwin provides the mkshortcut and readshortcut utilities.
  • An example screenshot how Cygwin symlink looks in Windows Explorer:

Cygwin symlink.png

Tools to create symlinks on Windows file system:

  • mklink - supported in Windows 7 and Windows Vista
  • linkd - supported from Windows 2000 (Windows Resource Kit Tools)

rsync on mingw32

  • Can't be easily ported, because of
    • Differences in Unix and Windows file system (symlinks, devices, paths ad etc)
    • Differences in threads and sockets, missing POSIX functions on Windows
    • Conclusion - rsync is designed for Unix like systems
  • Possible actions:

other native implementations for Windows

  • RsyncWin32
    • alpha version, there are bug reports about instability, last release in 2005
    • based on rsync-2.6.3, current version 3.0.7
    • implements POSIX functions, resolves issues with Windows threads and sockets
  • rsync-bc
    • only allows to push data to a server
    • based on rsync-2.5.6

alternatives to rsync

TBD

unison - a file-synchronization tool for Unix and Windows

zsync - rsync over HTTP

samba on qemu and share mapping

idea: install & configure samba on QEMU machine then try to mount this share and use as sysroot

TBD

qemu disk mapping and mounting on Windows

Idea: Try to mount QEMU image using filedisk and then map partition using ext2 driver

It's pretty simple to mount QEMU image: 

filedisk /mount 0 "FULL_PATH_TO_DISK_IMAGE" LETTER:

But then it's impossible to map this partition to drive, because Ext2 Volume Manager doesn't see it as a physical disk (volume).

List of tools for accessing Ext2 file systems on Windows:

  • Ext2Fsd - Open source ext3/4 file system driver for Windows (2K/XP/VISTA/WIN7).
  • Explore2fs - GUI explorer tool for accessing ext2 and ext3 filesystems.
    • this tool has feature to read image files, but it doesn't work (Couldn't find valid Superblock)
  • Ext2read - an explorer like utility to explore ext2/ext3/ext4 files.
  • Ext2 IFS - provides full access to Linux Ext2 volumes.
    • no support for Windows 7

qcow2 image mounting options

Tools to access qcow2 format images:

  • qemu-nbd - QEMU Disk Network Block Device Server (from QEMU project)
  • fuseqemu - Tool to access virtual machine images like vmdk, qcow, qcow2 or raw images (alpha version, only 1 release)
  • libguestfs - set of tools for accessing and modifying virtual machine (VM) disk images (Large, active development team)

TODO: check availability on different OS and test performance

Used tools

time utility was used to measure performance, e.g.: 

$ time sudo mad-admin create -e -f meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime 
Extracting QEMU runtime meego-handset-ia32-qemu-1.1.20101031.2201-sda-runtime 
.....9....8....7....6....5....4....3....2....1....0 
Creating missing links... done. 

real	2m54.520s 
user	2m15.132s 
sys	0m4.608s

rsync has --stats option, which gives information about copying speed and etc.

Used hardware

Description of hardware used in this study

Linux machine:

  • Dell Latitude E6510
  • Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (quad-core, clock: 533MHz)
  • RAM 4 GB @ 1067MHz DDR3 (3455MB seen by 32bit system)
  • HDD WD 250GB 7200RPM (encrypted fs)
  • Ubuntu 10.10 (ia32, 2.6.36-020636-generic)
  • BogoMips 4787.89

Windows machine:

  • HP EliteBook 8440p
  • Intel(R) Core(TM) i5 CPU M 520 @ 2.40GH (dual-core)
  • RAM 4GB, HDD 7200 rpm (encrypted fs)
  • Windows 7 Enterprise (32-bit)
  • System Base Score 4.5

Conclusions

IN PROGRESS

Because there are different types of libraries (RPM or source tarball), therefore real implementation of this sysroot extension could differ.

TBD

Time Estimate / Schedule

At the moment there is no target milestone for this feature.

Links

Retrieved from "http://wiki.meego.com/SDK/Sysroot_extension"
Personal tools