Revision as of 15:08, 9 December 2010

REVS

So you want to get a grasp on what's going on when you release? Enter REVS – Release Engineering Visibility System (a software previously know as grasp).

REVS...

• provides an authoritative package database handling Products, Platforms, Licenses, Bug/Feature Trackers, Roles, Packages etc
• contains a release data-warehouse section for Releases, Baselines, Changelogs, Bugs, Images etc
• has a flexible group hierarchy allowing data to be presented as it is needed
• provides high speed access to all aspects of a release
• allows management reports on bugs, features, changes, packages

Use cases (WIP)

Basic

1. Import releases of baselines from a repository of packages.
2. Compare the delta between baselines showing packages added, removed, unchanged and changed and their versions.
3. List bugs that were mentioned in the changelogs of packages that have changed as links to their respective bug trackers.
4. Mix baselines into products and be able to visualise and compare them as above.
5. List all packages of a certain release, product or baseline and present them in a printable or machine readable format.

Advanced

1. Define and display all kinds of grouping of packages by : category or arbitrary naming.
2. Group and display packages by license and maintainer.
3. Associate audit trail of where a package originated (source control url and tag), and how it was accepted into a baseline (Promotion requests)
4. Store URLs of repositories and build configuration templates (kickstarts) that can be used to reproduce an image.

It integrates with, but isn't part of the BOSS workflow system.

Implementation consists of :

• Data model
  • Master python models
    • sack : for collections of packages
    • dwh : for the release datawarehouse
    • nyc : License audit support (No You Can't)
• Data IN
  • Rest API for updates
  • Data management via Django
  • Import from changes, rpms etc
  • BOSS continuous integration participant
• Data OUT
  • Rest API for updates
  • Package set comparison reporting via Django

Download Source Code

REVS is currently available in gitorious:

http://meego.gitorious.org/meego-infrastructure-tools/revs

git clone git@gitorious.org:meego-infrastructure-tools/revs.git

Installable packages (deb/rpm) can be found here :

https://build.opensuse.org/package/show?package=revs&project=Maemo%3AMeeGo-Infra

Deploy to RPM

Our release process involves creating a suitable branch such as : release-w50

This is then used to build images for staging; any bug fixes are applied to the release branch as well as master.

To create an OBS buildable package from a REVS git tree, you can either use a [[1]] or do it manually.

Manual steps are:

1. Check out the git sources

git clone http://git.gitorious.org/meego-infrastructure-tools/revs.git
cd revs

1. Create a tarball out of the tree to a OSC checkout:

RELEASE=release-w50
VER=0.4
git archive --format=tar --prefix=revs-$VER/ $RELEASE | gzip > ${OSCCHECKOUT}/revs-$VER.tar.gz
cd ${OSCCHECKOUT}/

1. Commit the changes

osc commit

To use the above script, you can just issue:

./send_to_OBS.sh 0.4 ${OSCCHECKOUT}

Installing And Setting Up

Installing and setting up REVS is guided in the README file in the source code. It includes the following sections

• Setting REVS Up
  • Prerequisites
  • Running REVS
• Admin Interface and Trackers
  • Admin interface
• Test Data
  • Adding Data

Installing using RPM

When installing REVS via RPM, ther are specific instructions for it. For the rest of the chapters, the installation OS is assumed OpenSUSE, version 11.2.

When installing on openSuse, the following extra repos are needed :

 zypper ar http://download.opensuse.org/repositories/server:/http/openSUSE_11.2/server:http.repo
 zypper ar http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_11.2/devel:languages:python.repo

Start up by pulling in the REVS repository information and refresh the repositories:

 zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra:/Testing/openSUSE_11.2-plus/Maemo:MeeGo-Infra:Testing.repo
 zypper ref

Then install the revs package:

 zypper in revs

Lighttpd configuration

Create a file /etc/lighttpd/vhosts.d/revs.conf, with contents

var.namebasedir = "/revs"
$HTTP["url"] =~ "^" + namebasedir {
        dir-listing.activate = "disable" 
}
alias.url += (
  namebasedir + "/static" => "/usr/share/revs/static/",
  namebasedir + "/media" => "/usr/lib/python2.6/site-packages/django/contrib/admin/media/",
)
url.redirect += (
  "^" + namebasedir + "$" => namebasedir + "/"
)
fastcgi.server += (
    "/revs.fcgi" => (
        "main" => (
            "socket" => "/var/run" + namebasedir + ".socket",
            "check-local" => "disable",
        )
    ),
)
url.rewrite-if-not-file += (
   "^(" + namebasedir + "/.*)$" => "/revs.fcgi/$1"
)

And uncomment the following line in /etc/lighttpd/lighttpd.conf:

 include_shell "cat /etc/lighttpd/vhosts.d/*.conf"

Also make sure that modules alias, redirect, rewrite and fastcgi are enabled.

If you did a fresh install of the MySQL database, see the following section. Otherwise skip to database creation.

MYSQL Installation

Mysql database is needed by revs, but the setup is not too hard.

Issue the following commands:

 rcmysql start
 mysql_secure_installation

And answer the questions asked by the installation script. Move on to the next section.

Database creation

REVS needs a functional mysql installation in order to work properly. The following script will do just that.

mysql -u root -p
  DROP DATABASE IF EXISTS revs;
  CREATE DATABASE revs CHARACTER SET utf8;
  CREATE USER 'revs'@'localhost' IDENTIFIED BY 'revs';
  GRANT ALL ON revs.* TO 'revs'@'localhost';
  FLUSH PRIVILEGES;

Afterwards issue the following command:

django-admin.py syncdb --settings=revs.settings

If you want to create a superuser for Django, you may do so in the previous script.

Starting up REVS

REVS can be started up with its own initscript. Before running it, make sure you have completed each of the above section completely.

 /etc/init.d/lighttpd restart
 /etc/init.d/revs start

To automatically start REVS on boot do:

chkconfig --add mysql lighttpd revs

After this, you can continue to importing data

Testing

REVS allows usage of Django test framework to run unit tests on its components. Example is provided by running

run_tests.sh

in the source root directory. It reports if all tests passed.

REST

REVS has an API for controlling the models via HTTP. Currently Baseline, Package, Platform, Product, Product Release and Tracker models are modifiable and readable using our RESTful API.

Our REST implementation of choice is Django Piston, as it allows direct model handling using the Django ORM.

Currently the best way to use the API is using curl or anything that can GET / POST JSON. In curl this is achieved using

-d @file.json

as the command line parameter to get data from a file. See the API section for details and examples for each model.

Also the responses are given in JSON, so one might need eg. edit-json to view or edit JSON files.

API

The API handlers all support either pure url-encoded data or JSON. Each model is capable of both but this document reviews only the JSON input and output.

Baseline

JSON format for input and output

  
{
    "platform": {
        "name": "MeeGo"
    },
    "name": "1.1.80.2.20101019.1",
    "date": "2010-12-12"
    "pkgs": [
        {
            "version": "0.15",
            "name": "spec-builder"
        },
    (many more packages)
}

Package

JSON format for input and output

{
    "name": "bluez", 
    "platform": "MeeGo", 
    "latest_version": "4.76",
    "versions": [
        {
            "version": "4.70", 
            "package_system": "rpm", 
            "changelogentries": [
                {
                    "date": "Thu Sep 30 2010", 
                    "author": "Zhu Yanhai ", 
                    "email": "<yanhai.zhu@linux.intel.com>", 
                    "links": [], 
                    "log": "- Add powered.patch to make sure the initial power status aligned\n  with connman's requirement.\n"
                }, 
                {
                    "date": "Fri Aug 27 2010", 
                    "author": "Zhu Yanhai ", 
                    "email": "<yanhai.zhu@linux.intel.com>", 
                    "links": [], 
                    "log": "- Upgrade to 4.70\n"
                }
            ]
        },
     ... (many more versions)
}

Platform

JSON format for input and output

{
        "name": "MeeGo"
}

Product

JSON format for input and output

{
    "platforms": [
        {
            "name": "MeeGo"
        }
    ], 
    "name": "foobar", 
    "target_hardware": "n900"
}

Product Release

JSON format for input and output

{
    "release_of": "test", 
    "baselines": [
        {
            "name": "1.1.80.3.20101026.1"
        }, 
        {
            "name": "1.1.80.2.20101019.1"
        }
    ], 
    "name": "test_weekly_w45"
}

Tracker

JSON format for input and output

{
        "name": "MeeGoBugzilla", 
        "key2url": "http://bugs.meego.com/show_bug.cgi?id=%key%", 
        "url": "http://bugs.meego.com/", 
        "link_class": "Bug", 
        "filter2url": "<a ref=\"http://bugs.meego.com/show_bug.cgi?id=\\g<key>\">\\g<1></a>", 
        "regexp": "(\\BMC#(?P<key>\\ )\\))"
        "platforms": [
            {
                "name": "MeeGo"
            }
         ]
}

Importing data to REVS

To parse changelogs directly from source rpms, use parse_release.py.

To feed parse release with .changes files, copy this script and run it against a MeeGo mirror directory to create tarballs of .changes files:

./get-M-changes.sh /mnt/obs_repository/repository/MeeGo/1.0.80/1.1.80.4.20101102.1/

#!/bin/bash

prjdir=$1

[ -d $prjdir ] || {
  echo $prjdir is not a directory
  exit 1
}

prj=$(basename $prjdir)
dir=$(dirname $prjdir)

mkdir $prj
cd $prj

get_changes() {
        while read filename; do
            rpmname=$(rpm -qp --qf %{NAME} $filename)
            echo found $rpmname in $filename
            rpm -qp --changelog $filename > $rpmname.changes
        done
}

for area in core handset
do
    find $prjdir/$area -name *src.rpm* | get_changes
done
cd ..
tar cjf $prj.tar.bz $prj

And then bunzip and tar -x the files, then run parse release for each baseline and platform, like this script does:

import() {
        parse_release.py --force \
            --platform=${platform} \
            --baseline=${base} \
            releases/${base}
    }
    platform=MeeGo
    base=1.1.80.2.20101019.1
    import
    base=1.1.80.3.20101026.1
    import
    base=1.1.80.4.20101102.1
    import

REVS License

REVS is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.