Views

Release Infrastructure/REVS

From MeeGo wiki

Revision as of 11:25, 15 June 2011 by Locusf (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

1 REVS

REVS

So you want to get a grasp on what's going on when you release? Enter REVS – Release Engineering Visibility System (a software previously know as grasp).

REVS...

provides an authoritative package database handling Products, Platforms, Licenses, Bug/Feature Trackers, Roles, Packages etc
contains a release data-warehouse section for Releases, Baselines, Changelogs, Bugs, Images etc
has a flexible group hierarchy allowing data to be presented as it is needed
provides high speed access to all aspects of a release
allows management reports on bugs, features, changes, packages

Use cases (WIP)

Basic

Import releases of baselines from a repository of packages.
Compare the delta between baselines showing packages added, removed, unchanged and changed and their versions.
List bugs that were mentioned in the changelogs of packages that have changed as links to their respective bug trackers.
Mix baselines into products and be able to visualise and compare them as above.
List all packages of a certain release, product or baseline and present them in a printable or machine readable format.

Advanced

Define and display all kinds of grouping of packages by : category or arbitrary naming.
Group and display packages by license and maintainer.
Associate audit trail of where a package originated (source control url and tag), and how it was accepted into a baseline (Promotion requests)
Store URLs of repositories and build configuration templates (kickstarts) that can be used to reproduce an image.

It integrates with, but isn't part of the BOSS workflow system.

Implementation consists of :

Data model
- Master python models
  - sack : for collections of packages
  - dwh : for the release datawarehouse
  - nyc : License audit support (No You Can't)
Data IN
- Rest API for updates
- Data management via Django
- Import from changes, rpms etc
- BOSS continuous integration participant
Data OUT
- Rest API for updates
- Package set comparison reporting via Django

Data Model

Diagram

The model is produced using the linux 'dia' tool and the master version can be found on gitorious.

Full-resolution SVG

Overview and Terminology

The data model was written to support a vendor building a product on top of MeeGo.

It's not a purist model as it deals with the limitations of implementation choices.

SACK

SACK is the python/django component of REVS which has the sole purpose of providing the datamodel.

Package Group

Package Group is a abstractish datamodel used by most of the REVS datamodels, most notably Platform, Product, Baseline and Product Release. This allows a list-like model of the data along with providing hierarchy (flat and recursive groups). The instances of Package Group consist of Package Links.

Package Group supports hierarchies of other Package Groups.

Package Link

A Package Link represents things related to a package. It is used to allow grouping without replicating everything in the package. It has to have an underlying Package Name.

Package Links can derive to either pure Package, a Package Version, Installable (rpm) or Installable Version. See the later section for details about the each of the package classes.

Platform

A Platform is, at its basic form, a collection of packages. It represents a major distribution release and is used to group development releases, for example baselines.

The most basic example of a platform is MeeGo, which is a distribution.

Baseline

Baseline represents a point in time snapshot of the Package Versions associated by the Packages in a Platform. A noteworthy point is that Platform does not "contain" baselines, it links to instances of baselines.

Product

Product represents a collection of platforms that are used to run a distribution on a specific hardware. A subset of these packages usually form an iso or other device image.

An example of a product is MeeGo-N900, which targets N900 as its hardware and uses MeeGo as its distribution.

Product Release

A ProductRelease is a collection of snapshots/Baselines of the Packages of relevant Platforms which make an image for a device. Inherits from PkgGroup to allow Releases to be collected.

Package Name

Package Name is a separate entity from the Package Link. The primary purpose of it is to provide a linking between Package Name and Package Link (and its derivatives).

Package

Package is the pure entity of a package, but represented with the functionality of a Package Link.

Installable

Installable represents the conceptual installable binary blob(s) that can be produced by a Package.

Installable Version

This represents the actual installable binary blob(s) produced by a given PackageVersion and found in an Image. Note that sometimes software is only provided as an InstallableVersion, eg. as a binary RPM.

NYC

NYC is another component of REVS, its purpose is to handle Licenses within releases, to identify any and all possible miscompatibilities.

License

License is a Package Group, thus it can have packages and sublicenses. An example is LGPL, which (in MeeGo) has packages scim and gamin and its sublicenses are LGPLv2 and LGPLv3 and LGPLv2.1.

Download Source Code

REVS is currently available in gitorious:

http://meego.gitorious.org/meego-infrastructure-tools/revs

git clone git@gitorious.org:meego-infrastructure-tools/revs.git

Installable packages (deb/rpm) can be found here :

https://build.opensuse.org/package/show?package=revs&project=Maemo%3AMeeGo-Infra

Release notes

The release notes reside in the release notes page.

Deployment process

When REVS is deployed, it goes through a release process iteration within the team. The process consists of creating a release branch from master branch, creating a release version tag when staging tests pass. After this, the sources are packaged using git-archive method, with HEAD as the current head of the release branch. To do your own packaging in eg. home project, just link repositories from the testing repository and branch your package, without following upstream release. If you do this, remember to reflect the TESTING_OBS_REPO variable to your setup.

 git status
 "On branch master"
 git checkout -b release-2011wXX (replace XX with week if any)
 git archive --format=tar --prefix=revs-X.X.X/ release-2011wXX|gzip > $(TESTING_OBS_REPO)/revs/revs_X.X.X.tar.gz (replace X.X.X with version you are testing)

The sources, along with any other necessary packaging information, are then uploaded to OBS for building. The project in which REVS lives for testing is Maemo:MeeGo-Infra:Testing.

(while in TESTING_OBS_REPO)
osc commit -m "Staging release"

The package may not build, if it doesn't, there is another GIT source iteration which shall include the fixes needed to get the package to build.

cd $(REVS_SRC_REPO)
(in release branch)
vim revs/.... (do whatever is needed to fix the OBS build)
git commit -m "Fixed xxx"
git archive --format=tar --prefix=revs-X.X.X/ release-2011wXX|gzip > cd $(TESTING_OBS_REPO)/revs/revs_X.X.X.tar.gz (replace X.X.X with version you are testing)
cd $(TESTING_OBS_REPO)/revs/
osc commit -m "Fixed xxx"

Before upgrading the staging server, one must destroy the staging database and import the production database. This is to create a certainty that the production database can be migrated successfully.

A simple mysqldump is a generally accepted way of doing this.

When the package builds successfully, it is installed to the staging server for further tests.

zypper ref
/etc/init.d/revs stop
zypper in revs

Follow any and all notes in the release notes page afterwards. This is because of possible migrations and data imports that need to take place in order for REVS to be functional.

(Do your release notes imports/migrations)
/etc/init.d/revs start

The staging server testing includes a rigorous testing of the interface and the admin interface. After the package is upgraded, data from production database shall be pulled into the staging server database.

Every new feature included in the release shall be tested and verified by the developers and an architect. If the developers or architect are not satisfied of the testing, the iteration will begin anew from the GIT sources.

The bugs arised during testing in staging are fixed and committed to the release branch, if the fixes concern only the current release. Otherwise the commits are made to the master branch. After this, the tag made earlier is updated to the current head of the release branch.

cd $(REVS_SRC_REPO)
(in release branch)
vim revs/.... (do whatever is needed to fix the OBS build)
git commit -m "Fixed xxx"
git archive --format=tar --prefix=revs-X.X.X/ release-2011wXX|gzip > cd $(TESTING_OBS_REPO)/revs/revs_X.X.X.tar.gz (replace X.X.X with version you are testing)
cd $(TESTING_OBS_REPO)/revs/
osc commit -m "Fixed xxx"

If the staging environment works and passes the tests, upgrade on production takes place. If it doesn't, an LVM rollback of staging is required and after that, the staging process starts again.

Installing And Setting Up

Installing and setting up REVS is guided in the README file in the source code. It includes the following sections

Setting REVS Up
- Prerequisites
- Running REVS
Admin Interface and Trackers
- Admin interface
Test Data
- Adding Data

Installing using RPM

When installing REVS via RPM, ther are specific instructions for it. For the rest of the chapters, the installation OS is assumed OpenSUSE, version 11.2.

When installing on openSuse, the following extra repos are needed :

 zypper ar http://download.opensuse.org/repositories/server:/http/openSUSE_11.2/server:http.repo
 zypper ar http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_11.2/devel:languages:python.repo

Start up by pulling in the REVS repository information and refresh the repositories:

 zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo
 zypper ref

Then install the revs package:

 zypper in revs

Lighttpd configuration

Create a file /etc/lighttpd/vhosts.d/revs.conf, with contents

var.namebasedir = "/revs"
$HTTP["url"] =~ "^" + namebasedir {
        dir-listing.activate = "disable" 
}
alias.url += (
  namebasedir + "/static" => "/usr/share/revs/static/",
  namebasedir + "/media" => "/usr/lib/python2.6/site-packages/django/contrib/admin/media/",
)
url.redirect += (
  "^" + namebasedir + "$" => namebasedir + "/"
)
fastcgi.server += (
    "/revs.fcgi" => (
        "main" => (
            "socket" => "/var/run" + namebasedir + ".socket",
            "check-local" => "disable",
        )
    ),
)
url.rewrite-if-not-file += (
   "^(" + namebasedir + "/.*)$" => "/revs.fcgi/$1"
)

And uncomment the following line in /etc/lighttpd/lighttpd.conf:

 include_shell "cat /etc/lighttpd/vhosts.d/*.conf"

Also make sure that modules alias, redirect, rewrite and fastcgi are enabled.

If you did a fresh install of the MySQL database, see the following section. Otherwise skip to database creation.

MYSQL Installation

Mysql database is needed by revs, but the setup is not too hard.

Issue the following commands:

 rcmysql start
 mysql_secure_installation

And answer the questions asked by the installation script. Move on to the next section.

Database creation

REVS needs a functional mysql installation in order to work properly. The following script will do just that.

mysql -u root -p
  DROP DATABASE IF EXISTS revs;
  CREATE DATABASE revs CHARACTER SET utf8;
  CREATE USER 'revs'@'localhost' IDENTIFIED BY 'revs';
  GRANT ALL ON revs.* TO 'revs'@'localhost';
  FLUSH PRIVILEGES;

Afterwards issue the following commands:

django-admin.py syncdb --settings=revs.settings
django-admin.py migrate --settings=revs.settings

If you want to create a superuser for Django, you may do so in the previous script.

Starting up REVS

REVS can be started up with its own initscript. Before running it, make sure you have completed each of the above section completely.

 /etc/init.d/lighttpd restart
 /etc/init.d/revs start

To automatically start REVS on boot do:

chkconfig --add mysql lighttpd revs

After this, you can continue to importing data

Install from Debian package

For starters, one can enable the Maemo:MeeGo-Infra repository by adding the following line to /etc/apt/sources.list:

deb http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/Debian_5.0/ /

After adding the repositories, you can install revs by:

apt-get update
apt-get install revs

Nginx installation

REVS in meego.com uses nginx, so here are the relevant config bits:

location ^~ /revs/static/ {
    alias /usr/share/revs/static/;
}
location /revs {
# host and port to fastcgi server
     fastcgi_pass unix:/var/run/revs.socket;
     fastcgi_param PATH_INFO $fastcgi_script_name;
     fastcgi_param REQUEST_METHOD $request_method;
     fastcgi_param QUERY_STRING $query_string;
     fastcgi_param CONTENT_TYPE $content_type;
     fastcgi_param CONTENT_LENGTH $content_length;
     fastcgi_pass_header Authorization;
     fastcgi_intercept_errors off;

}

Database installation

Follow the same instructions as in OpenSUSE installation guide, minus that django-admin.py is django-admin in Debian.

Testing

REVS allows usage of Django test framework to run unit tests on its components. Example is provided by running

run_tests.sh

in the source root directory. It reports if all tests passed.

REST

REVS has an API for controlling the models via HTTP. Currently Baseline, Package, Platform, Product, Product Release and Tracker models are modifiable and readable using our RESTful API.

Our REST implementation of choice is Django Piston, as it allows direct model handling using the Django ORM.

The scripts for reading and writing are available in the REVS source repository.

See the API section for details and examples for each model.

Also the responses are given in JSON, so one might need eg. edit-json to view or edit JSON files.

API

The API handlers all support either pure url-encoded data or JSON. Each model is capable of both but this document reviews only the JSON input and output.

Urls

The following URLS are supported:

 http://revs.host/revs/api/Package/<name> (Get a package name, list all versions) (Read/Write)
 http://revs.host/revs/api/Package/<name>/<version> (fetches a Package by Version) (Read-only)
 http://revs.host/revs/api/Product/ (shows a list of products) (Read-only)
 http://revs.host/revs/api/Product/<name> (shows specific product) (Read/Write)
 http://revs.host/revs/api/Baseline/ (shows a list of baselines) (Read-only)
 http://revs.host/revs/api/Baseline/?sort={date,name,platform} (shows a list of baselines, sorted by either date, name or platform) (Read-only)
 http://revs.host/revs/api/Baseline/<name> (shows specific baseline) (Read/Write)
 http://revs.host/revs/api/ProductRelease/ (shows a list of product releases) (Read-only)
 http://revs.host/revs/api/ProductRelease/?sort={'name', 'release_of', 'date'} (shows a list of product releases, sorted by either name, release of the product release or date) (Read-only)
 http://revs.host/revs/api/ProductRelease/<name> (specific product release) (Read/Write)
 http://revs.host/revs/api/Tracker/ (shows a list of trackers) (Read-only)
 http://revs.host/revs/api/Tracker/<name> (specific tracker) (Read/Write)

Also there are a couple of functional urls:

 http://revs.host/revs/api/Package/<query>/search
 http://revs.host/revs/api/ProductRelease/<query>/search