Release Infrastructure/devroot

Contents 1 story 2 use case 3 solution 3.1 devroot 3.1.1 init(conf, [path]) 3.1.2 update:([path]) 3.1.3 build 3.1.4 target-flash 3.1.5 target-update 3.1.6 target-test(test_package) 3.2 Implementation notes 3.2.1 Chroot setup, step 1 3.2.2 Chroot setup, step 2 3.2.3 Building 3.2.4 Publishing

story

As a developer I need to be able to

• quickly build, deploy, test my changes
• with possibility to work offline
• have similar build results than the releasing system so my test results are reliable
• be able to skip reverse dependency builds
• do all that in with a very simple tool

In order to speed up the development process, and be more productive

use case

case 1

1. modify one package
2. build it
3. update target device
4. run test cases

case 2

1. modify code across the stack (multiple pkgs)
2. recursively build all modified pkgs
3. update target device
4. run test cases

solution

these requirements can be met easily with manual work, there is no missing tools, the real problem is in simplifying and automating

devroot

a tool that wraps all the commands to run that process, automating many manual steps

1. init
2. update
3. build
4. target-flash
5. target-update
6. target-test

init(conf, [path])

takes as input the list of packages the developer will work on (FIXME: more details needed) create a chroot will all operations will run, it is necessary to isolate the build env to not get reliable results fetch all needed sources and extract them in the specified location e.g /home/user/chroot/usr/share/devtree/ chroot, will contain a bootstrap of the desired distro, all the build tools, and the build dependencies of the packages specified open items:

• where te fetch sources? src.rpm or git trees or obs projects?
• do we fetch dependencies? what about reverse dependencies?

update:([path])

update the chroot env, and the sources provide a merge possibility of the newer code and the modifications done locally

build

this should perform

• build (make)
• package
• publish

this is what rpmbuild does, but it would require that we archive the source tree and then rpmbuild would unpack it, also rpm would always flush the buildroot and start building from scratch again. this way is good for releasing but too slow for our purpose. rpmbuild has oprtions we can use to control all those and reach the desired process build in place (no archive) and keep old build data, so rebuilds are much faster (depending on how make detects changes) the build script of OBS should be used as it will produce a similar result as what happens on OBS this covers the build and package, publish is needed to make that package available to the target device this can be done through filesystem or http if build method is called from inside a certain pkg tree e.g /home/user/chroot/usr/share/devtree/pkg1, then it would only build this package if called from the devtree root /home/user/chroot/usr/share/devtree/ the it should detect whcih pkgs have modifications and only build those

target-flash

create an image using mic, and flash it to device, tools and details are device specific. we might skip this method

target-update

connect to device over usb-network and update the device through the published repo e.g. zypper ref;zypper up

target-test(test_package)

connect to device over usb-network and run the test cases provided. a good tool for that would be test-runner-lite part of OTS testing suite

Implementation notes

Chroot setup, step 1

We can use both osc or build to setup the initial chroot -- osc uses build itself, though uses OBS for dependency calculations. Both methods allow offline operation once the initial chroot got built.

For using build the best method would be having a complete offline copy of the distribution packages

For using osc we just download the packages needed. For a multiple package build a quick hack without changing too much in osc would be constructing and building a dummy package containing all the build dependencies of the packages we want to build.

Most likely it would be best to extend osc, though we currently risk having to do work again with osc2. As much of the functionality as possible should be done in osc-plugins.

Chroot setup, step 2

One problem we have with the rebuild-test cycle is speed. Doing a complete package rebuild is slow, so we need to keep the buildroot around. Additionally, we need to be able to tell build to use an existing buildroot. (As a side effect, this will easily allow us building a single package from a local git source tree, too, by just specifying the git source tree as buildroot)

TODO here:

• patch build to be able to use an existing buildroot (rpmbuild already can: --short-circuit with -bc and -bi)
• patch osc to use the additional build functionality. This is probably the only thing we should try to implement directly in osc, not in osc plugins
• put the buildroot in place; currently manual bind mount, will be replaced by a custom fuse filesystem

Another problem to solve here is unpacking the source packages; it is currently unclear how we'll find all source packages we need to unpack.

The sources will end up in a git repository, transparently masked by fuse. Patches will probably go into a separate branch (gbp-pq). This allows us to:

• easily remove patches from the buildroot
• easily (and quickly) clean the buildroot (just a git clean)
• easily determine which packages need to be rebuilt by looking at the repository state on multiple package builds
• keep a history of changes, and produce a patch with the changes

The functionality here can be implemented partly in the filesystem, partly in an osc plugin.

Building

• (for multiple packages) determine which packages to rebuild
• make sure the buildroot is available
• commit snapshot to git (ignoring files generated by build process -- we need to generate proper gitignore here)
• build with patched osc/build in an existing buildroot

Publishing

We can assume that the device for testing will be reachable by network (usually: usb network), so the easiest way to get the packets to the device would be by using zypper over http.

A simple httpd to expose a local directory is trivial to implement, so we just have to

• write the packages to some directory where they'll stay available (osc cleans old packages on build)
• remove old versions of the same package
• rebuild the catalog
• trigger the device to refresh the catalogs, and install/update the package