Meego Wiki
Views

Release Infrastructure/REVS

From MeeGo wiki
Jump to: navigation, search

Contents

REVS

So you want to get a grasp on what's going on when you release? Enter REVS – Release Engineering Visibility System (a software previously know as grasp).

REVS...

  • provides an authoritative package database handling Products, Platforms, Licenses, Bug/Feature Trackers, Roles, Packages etc
  • contains a release data-warehouse section for Releases, Baselines, Changelogs, Bugs, Images etc
  • has a flexible group hierarchy allowing data to be presented as it is needed
  • provides high speed access to all aspects of a release
  • allows management reports on bugs, features, changes, packages

Use cases (WIP)

Basic

  1. Import releases of baselines from a repository of packages.
  2. Compare the delta between baselines showing packages added, removed, unchanged and changed and their versions.
  3. List bugs that were mentioned in the changelogs of packages that have changed as links to their respective bug trackers.
  4. Mix baselines into products and be able to visualise and compare them as above.
  5. List all packages of a certain release, product or baseline and present them in a printable or machine readable format.

Advanced

  1. Define and display all kinds of grouping of packages by : category or arbitrary naming.
  2. Group and display packages by license and maintainer.
  3. Associate audit trail of where a package originated (source control url and tag), and how it was accepted into a baseline (Promotion requests)
  4. Store URLs of repositories and build configuration templates (kickstarts) that can be used to reproduce an image.


It integrates with, but isn't part of the BOSS workflow system.

Implementation consists of :

  • Data model
    • Master python models
      • sack : for collections of packages
      • dwh : for the release datawarehouse
      • nyc : License audit support (No You Can't)
  • Data IN
    • Rest API for updates
    • Data management via Django
    • Import from changes, rpms etc
    • BOSS continuous integration participant
  • Data OUT
    • Rest API for updates
    • Package set comparison reporting via Django

Data Model

Diagram

REVS Datamodel diagram

The model is produced using the linux 'dia' tool and the master version can be found on gitorious.

Full-resolution SVG

Overview and Terminology

The data model was written to support a vendor building a product on top of MeeGo.

It's not a purist model as it deals with the limitations of implementation choices.

SACK

SACK is the python/django component of REVS which has the sole purpose of providing the datamodel.

Package Group

Package Group is a abstractish datamodel used by most of the REVS datamodels, most notably Platform, Product, Baseline and Product Release. This allows a list-like model of the data along with providing hierarchy (flat and recursive groups). The instances of Package Group consist of Package Links.

Package Group supports hierarchies of other Package Groups.

Package Link

A Package Link represents things related to a package. It is used to allow grouping without replicating everything in the package. It has to have an underlying Package Name.

Package Links can derive to either pure Package, a Package Version, Installable (rpm) or Installable Version. See the later section for details about the each of the package classes.

Platform

A Platform is, at its basic form, a collection of packages. It represents a major distribution release and is used to group development releases, for example baselines.

The most basic example of a platform is MeeGo, which is a distribution.

Baseline

Baseline represents a point in time snapshot of the Package Versions associated by the Packages in a Platform. A noteworthy point is that Platform does not "contain" baselines, it links to instances of baselines.

Product

Product represents a collection of platforms that are used to run a distribution on a specific hardware. A subset of these packages usually form an iso or other device image.

An example of a product is MeeGo-N900, which targets N900 as its hardware and uses MeeGo as its distribution.

Product Release

A ProductRelease is a collection of snapshots/Baselines of the Packages of relevant Platforms which make an image for a device. Inherits from PkgGroup to allow Releases to be collected.

Package Name

Package Name is a separate entity from the Package Link. The primary purpose of it is to provide a linking between Package Name and Package Link (and its derivatives).

Package

Package is the pure entity of a package, but represented with the functionality of a Package Link.

Installable

Installable represents the conceptual installable binary blob(s) that can be produced by a Package.

Installable Version

This represents the actual installable binary blob(s) produced by a given PackageVersion and found in an Image. Note that sometimes software is only provided as an InstallableVersion, eg. as a binary RPM.

NYC

NYC is another component of REVS, its purpose is to handle Licenses within releases, to identify any and all possible miscompatibilities.

License

License is a Package Group, thus it can have packages and sublicenses. An example is LGPL, which (in MeeGo) has packages scim and gamin and its sublicenses are LGPLv2 and LGPLv3 and LGPLv2.1.

Download Source Code

REVS is currently available in gitorious: 

http://meego.gitorious.org/meego-infrastructure-tools/revs

git clone git@gitorious.org:meego-infrastructure-tools/revs.git

Installable packages (deb/rpm) can be found here : 

https://build.opensuse.org/package/show?package=revs&project=Maemo%3AMeeGo-Infra

Deploy to RPM

Our release process involves creating a suitable branch such as : release-w50

This is then used to build images for staging; any bug fixes are applied to the release branch as well as master.

To create an OBS buildable package from a REVS git tree, you can either use a [[1]] or do it manually.

Manual steps are:

1. Check out the git sources 

git clone http://git.gitorious.org/meego-infrastructure-tools/revs.git
cd revs

1. Create a tarball out of the tree to a OSC checkout: 

RELEASE=release-w50
VER=0.4
git archive --format=tar --prefix=revs-$VER/ $RELEASE | gzip > ${OSCCHECKOUT}/revs_$VER.tar.gz
cd ${OSCCHECKOUT}/

1. Commit the changes 

osc commit

To use the above script, you can just issue: 

./send_to_OBS.sh 0.4 ${OSCCHECKOUT}

Installing And Setting Up

Installing and setting up REVS is guided in the README file in the source code. It includes the following sections

  • Setting REVS Up
    • Prerequisites
    • Running REVS
  • Admin Interface and Trackers
    • Admin interface
  • Test Data
    • Adding Data

Installing using RPM

When installing REVS via RPM, ther are specific instructions for it. For the rest of the chapters, the installation OS is assumed OpenSUSE, version 11.2.

When installing on openSuse, the following extra repos are needed : 

 zypper ar http://download.opensuse.org/repositories/server:/http/openSUSE_11.2/server:http.repo
 zypper ar http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_11.2/devel:languages:python.repo

Start up by pulling in the REVS repository information and refresh the repositories: 

 zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo
 zypper ref

Then install the revs package: 

 zypper in revs

Lighttpd configuration

Create a file /etc/lighttpd/vhosts.d/revs.conf, with contents 

var.namebasedir = "/revs"
$HTTP["url"] =~ "^" + namebasedir {
        dir-listing.activate = "disable" 
}
alias.url += (
  namebasedir + "/static" => "/usr/share/revs/static/",
  namebasedir + "/media" => "/usr/lib/python2.6/site-packages/django/contrib/admin/media/",
)
url.redirect += (
  "^" + namebasedir + "$" => namebasedir + "/"
)
fastcgi.server += (
    "/revs.fcgi" => (
        "main" => (
            "socket" => "/var/run" + namebasedir + ".socket",
            "check-local" => "disable",
        )
    ),
)
url.rewrite-if-not-file += (
   "^(" + namebasedir + "/.*)$" => "/revs.fcgi/$1"
)

And uncomment the following line in /etc/lighttpd/lighttpd.conf: 

 include_shell "cat /etc/lighttpd/vhosts.d/*.conf"

Also make sure that modules alias, redirect, rewrite and fastcgi are enabled.

If you did a fresh install of the MySQL database, see the following section. Otherwise skip to database creation.

MYSQL Installation

Mysql database is needed by revs, but the setup is not too hard.

Issue the following commands: 

 rcmysql start
 mysql_secure_installation

And answer the questions asked by the installation script. Move on to the next section.

Database creation

REVS needs a functional mysql installation in order to work properly. The following script will do just that. 

mysql -u root -p
  DROP DATABASE IF EXISTS revs;
  CREATE DATABASE revs CHARACTER SET utf8;
  CREATE USER 'revs'@'localhost' IDENTIFIED BY 'revs';
  GRANT ALL ON revs.* TO 'revs'@'localhost';
  FLUSH PRIVILEGES;

Afterwards issue the following commands: 

django-admin.py syncdb --settings=revs.settings
django-admin.py migrate --settings=revs.settings

If you want to create a superuser for Django, you may do so in the previous script.

Starting up REVS

REVS can be started up with its own initscript. Before running it, make sure you have completed each of the above section completely. 

 /etc/init.d/lighttpd restart
 /etc/init.d/revs start

To automatically start REVS on boot do: 

chkconfig --add mysql lighttpd revs


After this, you can continue to importing data

Testing

REVS allows usage of Django test framework to run unit tests on its components. Example is provided by running 

run_tests.sh

in the source root directory. It reports if all tests passed.

REST

REVS has an API for controlling the models via HTTP. Currently Baseline, Package, Platform, Product, Product Release and Tracker models are modifiable and readable using our RESTful API.

Our REST implementation of choice is Django Piston, as it allows direct model handling using the Django ORM.

Currently the best way to use the API is using curl or anything that can GET / POST JSON. In curl this is achieved using 

-d @file.json

as the command line parameter to get data from a file. See the API section for details and examples for each model.

Also the responses are given in JSON, so one might need eg. edit-json to view or edit JSON files.

API

The API handlers all support either pure url-encoded data or JSON. Each model is capable of both but this document reviews only the JSON input and output.

Urls

The following URLS are supported: 

http://revs.host/revs/api/Package/<name> (Get a package name, list all versions) (Read/Write)
http://revs.host/revs/api/Package/<name>/<version> (fetches a Package by Version) (Read-only)
http://revs.host/revs/api/Product/ (shows a list of products) (Read-only)
http://revs.host/revs/api/Product/<name> (shows specific product) (Read/Write)
http://revs.host/revs/api/Baseline/ (shows a list of baselines) (Read-only)
http://revs.host/revs/api/Baseline/<name> (shows specific baseline) (Read/Write)
http://revs.host/revs/api/ProductRelease/ (shows a list of product releases) (Read-only)
http://revs.host/revs/api/ProductRelease/<name> (specific product release) (Read/Write)
http://revs.host/revs/api/Tracker/ (shows a list of trackers) (Read-only)
http://revs.host/revs/api/Tracker/<name> (specific tracker) (Read/Write)

Also there are a couple of functional urls: 

http://revs.host/revs/api/Package/<query>/search
http://revs.host/revs/api/ProductRelease/<query>/search

Baseline

JSON format for input and output 

  
{
    "platform": {
        "name": "MeeGo"
    },
    "name": "1.1.80.2.20101019.1",
    "date": "2010-12-12"
    "pkgs": [
        {
            "version": "0.15",
            "name": "spec-builder"
        },
    (many more packages)
}

Package

JSON format for input and output 

{
    "name": "bluez", 
    "platform": "MeeGo", 
    "latest_version": "4.76",
    "versions": [
        {
            "version": "4.70", 
            "package_system": "rpm", 
            "changelogentries": [
                {
                    "date": "Thu Sep 30 2010", 
                    "author": "Zhu Yanhai ", 
                    "email": "<yanhai.zhu@linux.intel.com>", 
                    "links": [], 
                    "log": "- Add powered.patch to make sure the initial power status aligned\n  with connman's requirement.\n"
                }, 
                {
                    "date": "Fri Aug 27 2010", 
                    "author": "Zhu Yanhai ", 
                    "email": "<yanhai.zhu@linux.intel.com>", 
                    "links": [], 
                    "log": "- Upgrade to 4.70\n"
                }
            ]
        },
     ... (many more versions)
}

Platform

JSON format for input and output 

{
        "name": "MeeGo"
}

Product

JSON format for input and output 

{
    "platforms": [
        {
            "name": "MeeGo"
        }
    ], 
    "name": "foobar", 
    "target_hardware": "n900"
}

Product Release

JSON format for input and output 

{
    "release_of": "test", 
    "baselines": [
        {
            "name": "1.1.80.3.20101026.1"
        }, 
        {
            "name": "1.1.80.2.20101019.1"
        }
    ], 
    "name": "test_weekly_w45",
    "date": "2010-17-12"
}

Tracker

JSON format for input and output 

{
        "name": "MeeGoBugzilla", 
        "key2url": "http://bugs.meego.com/show_bug.cgi?id=%key%", 
        "url": "http://bugs.meego.com/", 
        "link_class": "Bug", 
        "filter2url": "<a ref=\"http://bugs.meego.com/show_bug.cgi?id=\\g<key>\">\\g<1></a>", 
        "regexp": "(\\BMC#(?P<key>\\ )\\))"
        "platforms": [
            {
                "name": "MeeGo"
            }
         ]
}

Importing data to REVS

To parse changelogs directly from source rpms, use parse_release.py.

To feed parse release with .changes files, copy this script and run it against a MeeGo mirror directory to create tarballs of .changes files: 

./get-M-changes.sh /mnt/obs_repository/repository/MeeGo/1.0.80/1.1.80.4.20101102.1/

#!/bin/bash

prjdir=$1

[ -d $prjdir ] || {
  echo $prjdir is not a directory
  exit 1
}

prj=$(basename $prjdir)
dir=$(dirname $prjdir)

mkdir $prj
cd $prj

get_changes() {
        while read filename; do
            rpmname=$(rpm -qp --qf %{NAME} $filename)
            echo found $rpmname in $filename
            rpm -qp --changelog $filename > $rpmname.changes
        done
}

for area in core handset
do
    find $prjdir/$area -name *src.rpm* | get_changes
done
cd ..
tar cjf $prj.tar.bz $prj

And then bunzip and tar -x the files, then run parse release for each baseline and platform, like this script does: 

import() {
        parse_release.py --force \
            --platform=${platform} \
            --baseline=${base} \
            releases/${base}
    }
    platform=MeeGo
    base=1.1.80.2.20101019.1
    import
    base=1.1.80.3.20101026.1
    import
    base=1.1.80.4.20101102.1
    import

Importing Installables

REVS supports Installables, which represent binary blobs, like binary rpms.

An Installable collection can be imported from PackageName objects in the REVS database, and Installables versions can be found using the underlying Package object of the PackageName. Versions are in the for of PackageVersion.

The script in GIT does exactly that.

The Installables are essential in creating Image objects in REVS, since Images are just a collection of binaries, which have a functionality to boot and provide the functionality of the binaries.

REVS License

REVS is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

Retrieved from "http://wiki.meego.com/Release_Infrastructure/REVS"
Personal tools