Meego Wiki
Views

Release Infrastructure/REVS

From MeeGo wiki
< Release Infrastructure(Difference between revisions)
Jump to: navigation, search
(Importing data to REVS)
(Undo revision 49814 by Jammiecockrell (talk) Spam)
 
(10 intermediate revisions not shown)
Line 1: Line 1:
= REVS =
= REVS =
-
So you want to get a ''grasp'' on what's going on when you release? Enter REVS – Release Engineering Visibility System (a software previously know as grasp).
+
== Deployment steps ==
-
REVS...
+
This instruction provides an overview of deployment process of revs from the
-
* provides an authoritative package database handling Products, Platforms, Licenses, Bug/Feature Trackers, Roles, Packages etc
+
sources fetched from gitorious. The result of the deployment is fully functional
-
* contains a release data-warehouse section for Releases, Baselines, Changelogs, Bugs, Images etc
+
tool ready to be used in production environment. Currently available for Debian only
-
* has a flexible group hierarchy allowing data to be presented as it is needed
+
-
* provides high speed access to all aspects of a release
+
-
* allows management reports on bugs, features, changes, packages
+
-
==Use cases (WIP)==
+
=== Install REVS from OBS repo (preferred) ===
-
===Basic===
+
-
# Import releases of baselines from a repository of packages.
+
-
# Compare the delta between baselines showing packages added, removed, unchanged and changed and their versions.
+
-
# List bugs that were mentioned in the changelogs of packages that have changed as links to their respective bug trackers.
+
-
# Mix baselines into products and be able to visualise and compare them as above.
+
-
# List all packages of a certain release, product or baseline and present them in a printable or machine readable format.
+
-
===Advanced===
+
Edit your sources.list.
-
# Define and display all kinds of grouping of packages by : category or arbitrary naming.
+
-
# Group and display packages by license and maintainer.
+
-
# Associate audit trail of where a package originated (source control url and tag), and how it was accepted into a baseline (Promotion requests)
+
-
# Store URLs of repositories and build configuration templates (kickstarts) that can be used to reproduce an image.
+
 +
<pre>
-
It integrates with, but isn't part of the [[BOSS]] workflow system.
+
    vim /etc/apt/sources.list
-
Implementation consists of :
+
</pre>
-
* Data model
+
-
** Master python models
+
-
*** sack : for collections of packages
+
-
*** dwh : for the release datawarehouse
+
-
*** nyc : License audit support (No You Can't)
+
-
* Data IN
+
-
** Rest API for updates
+
-
** Data management via Django
+
-
** Import from changes, rpms etc
+
-
** BOSS continuous integration participant
+
-
* Data OUT
+
-
** Rest API for updates
+
-
** Package set comparison reporting via Django
+
-
== Data Model ==
+
Add the following lines there:
-
=== Diagram ===
+
-
[[File:REVS.png|480px|REVS Datamodel diagram]]
+
-
The model is produced using the [http://projects.gnome.org/dia/ linux 'dia'] tool and the master version can be found [http://meego.gitorious.org/meego-infrastructure-tools/revs/blobs/raw/master/docs/sack.dia on gitorious].
+
<pre>
-
[[Media:REVS.svg|Full-resolution SVG]]
+
    deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0 ./
 +
    deb http://download.opensuse.org/repositories/openSUSE:/Tools/Debian_6.0/  /
-
=== Overview and Terminology ===
+
</pre>
-
The data model was written to support a vendor building a product on top of MeeGo.
+
-
It's not a purist model as it deals with the limitations of implementation choices.
+
Install LAMP (Linux Apache MySQL Python) configuration of REVS.
-
=== SACK ===
+
<pre>
-
SACK is the python/django component of REVS which has the sole purpose of providing the datamodel.
+
-
==== Package Group ====
+
    apt-get update
-
Package Group is a abstractish datamodel used by most of the REVS datamodels, most notably Platform, Product, Baseline and Product Release. This allows a list-like model of the data along with providing hierarchy (flat and recursive groups). The instances of Package Group consist of Package Links.
+
    apt-get install revs-lamp
-
Package Group supports hierarchies of other Package Groups.
+
</pre>
-
==== Package Link ====
+
If this is a "fresh" REVS install, you will be asked to type you '''root''' MySQL user
-
A Package Link represents things related to a package.  It is used to allow grouping without replicating everything in the package. It has to have an underlying Package Name.
+
password.
-
Package Links can derive to either pure Package, a Package Version, Installable (rpm) or Installable Version. See the later section for details about the each of the package classes.
+
After the installation is over you shall be able to access blank (with no data in it)
 +
instance of REVS via '''DOMAIN/revs''' url. You can also access admin site via
 +
'''DOMAIN/revs/admin''' url. Use '''admin'''/'''admin''' credentials.
-
==== Platform ====
+
=== Install REVS from GIT (if you want to play with sources) ===
-
A Platform is, at its basic form, a collection of packages. It represents a major distribution release and is used to group development releases, for example baselines.
+
-
The most basic example of a platform is MeeGo, which is a distribution.
+
Fetch package source from gitorious.
-
==== Baseline ====
+
<pre>
-
Baseline represents a point in time snapshot of the Package Versions associated by the Packages in a Platform. A noteworthy point is that Platform does not "contain" baselines, it links to instances of baselines.
+
-
==== Product ====
+
    git clone git://gitorious.org/meego-infrastructure-tools/revs.git
-
Product represents a collection of platforms that are used to run a distribution on a specific hardware. A subset of these packages usually form an iso or other device image.
+
</pre>
-
An example of a product is MeeGo-N900, which targets N900 as its hardware and uses MeeGo as its distribution.
+
Install all build dependencies.
-
==== Product Release ====
+
<pre>
-
A ProductRelease is a collection of snapshots/Baselines of the Packages of relevant Platforms which make an image for a device.
+
-
Inherits from PkgGroup to allow Releases to be collected.
+
-
===== Package Name =====
+
    sudo apt-get install yum python-django debhelper python-support python-setuptools python-sphinx python-sqlite
-
Package Name is a separate entity from the Package Link. The primary purpose of it is to provide a linking between Package Name and Package Link (and its derivatives).
+
</pre>
-
===== Package =====
+
Build the package.
-
Package is the pure entity of a package, but represented with the functionality of a Package Link.
+
-
===== Installable =====
+
-
Installable represents the conceptual installable binary blob(s) that can be produced by a Package.  
+
-
===== Installable Version =====
+
<pre>
-
This represents the actual installable binary blob(s) produced by a given PackageVersion and found in an Image. Note that sometimes software is only provided as an InstallableVersion, eg. as a binary RPM.
+
-
=== NYC ===
+
    cd revs
-
NYC is another component of REVS, its purpose is to handle Licenses within releases, to identify any and all possible miscompatibilities.
+
    dpkg-buildpackage -rfakeroot -uc -us -sa -tc -D
 +
    cd ..
-
==== License ====
+
</pre>
-
License is a Package Group, thus it can have packages and sublicenses. An example is LGPL, which (in MeeGo) has packages scim and gamin and its sublicenses are LGPLv2 and LGPLv3 and LGPLv2.1.
+
-
== Download Source Code ==
+
Install REVS and REVS' lamp configuration binary dependencies
-
REVS is currently available in gitorious:
+
<pre>
-
http://meego.gitorious.org/meego-infrastructure-tools/revs
+
    sudo apt-get install mysql-server apache2 libapache2-mod-wsgi python-mysqldb
-
git clone git@gitorious.org:meego-infrastructure-tools/revs.git
+
</pre>
-
Installable packages (deb/rpm) can be found here :
+
Install REVS together with LAMP configuration. Note that revs itself has to be
-
 
+
installed before LAMP configuration.
-
https://build.opensuse.org/package/show?package=revs&project=Maemo%3AMeeGo-Infra
+
-
 
+
-
== Release notes ==
+
-
The release notes reside in the [[Release_Infrastructure/REVS/ReleaseNotes|release notes page]].
+
-
== Deployment process ==
+
-
 
+
-
When REVS is deployed, it goes through a release process iteration within the team. The process consists of creating a release branch from master branch, creating a release version tag when staging tests pass. After this, the sources are packaged using git-archive method, with HEAD as the current head of the release branch. To do your own packaging in eg. home project, just link repositories from the testing repository and branch your package, without following upstream release. If you do this, remember to reflect the TESTING_OBS_REPO variable to your setup.
+
<pre>
<pre>
-
git status
 
-
"On branch master"
 
-
git checkout -b release-2011wXX (replace XX with week if any)
 
-
git archive --format=tar --prefix=revs-X.X.X/ release-2011wXX|gzip > $(TESTING_OBS_REPO)/revs/revs_X.X.X.tar.gz (replace X.X.X with version you are testing)
 
-
</pre>
 
-
The sources, along with any other necessary packaging information, are then uploaded to OBS for building. The project in which REVS lives for testing is Maemo:MeeGo-Infra:Testing.  
+
    sudo dpkg -i revs_*.deb
 +
    sudo dpkg -i revs-lamp_*.deb
-
<pre>
 
-
(while in TESTING_OBS_REPO)
 
-
osc commit -m "Staging release"
 
</pre>
</pre>
-
The package may not build, if it doesn't, there is another GIT source iteration which shall include the fixes needed to get the package to build.
 
-
<pre>
+
After installing REVS you may go to '''DOMAIN/revs''' to use the tool or to
-
cd $(REVS_SRC_REPO)
+
'''DOMAIN/revs/admin''' to access admin interface. The default admin user
-
(in release branch)
+
has username '''admin''' and password '''admin'''.
-
vim revs/.... (do whatever is needed to fix the OBS build)
+
-
git commit -m "Fixed xxx"
+
-
git archive --format=tar --prefix=revs-X.X.X/ release-2011wXX|gzip > cd $(TESTING_OBS_REPO)/revs/revs_X.X.X.tar.gz (replace X.X.X with version you are testing)
+
-
cd $(TESTING_OBS_REPO)/revs/
+
-
osc commit -m "Fixed xxx"
+
-
</pre>
+
-
Before upgrading the staging server, one must destroy the staging database and import the production database. This is to create a certainty that the production database can be migrated successfully.
+
=== Sync REVS with OBS instance ===
-
A simple mysqldump is a generally accepted way of doing this.
+
To sync with an OBS instance, use the admin UI to populate the projects. The names *MUST* be the sames as OBS and their repositories must be accessible to REVS.
-
When the package builds successfully, it is installed to the staging server for further tests.
+
After this is done, you can proceed by initially seed the database.
-
<pre>
+
==== Seeding database from initial base packages ====
-
zypper ref
+
-
/etc/init.d/revs stop
+
-
zypper in revs
+
-
</pre>
+
-
Follow any and all notes in the [[Release_Infrastructure/REVS/ReleaseNotes|release notes page]] afterwards. This is because of possible migrations and data imports that need to take place in order for REVS to be functional.
+
django-admin import_from_obs --settings=revs.settings
-
<pre>
+
This will create initial packages to a Platform and their versions to respective projects created in admin UI.
-
(Do your release notes imports/migrations)
+
-
/etc/init.d/revs start
+
-
</pre>
+
-
The staging server testing includes a rigorous testing of the interface and the admin interface. After the package is upgraded, data from production database shall be pulled into the staging server database.
+
==== Keeping REVS automatically updating ====
-
Every new feature included in the release shall be tested and verified by the developers and an architect. If the developers or architect are not satisfied of the testing, the iteration will begin anew from the GIT sources.  
+
Install revs-boss package so that you will get two participants: feed_revs and revs_image. The first is the package updater and second is an image importer, in case you are using IMG participant in the BOSS process. Remember to enable at least feed_revs.
-
The bugs arised during testing in staging are fixed and committed to the release branch, if the fixes concern only the current release. Otherwise the commits are made to the master branch. After this, the tag made earlier is updated to the current head of the release branch.
+
== Customization ==
-
<pre>
+
The goal of this article is to provide a comprehensive instruction on how to deploy
-
cd $(REVS_SRC_REPO)
+
several custom instances of REVS on the same machine.
-
(in release branch)
+
-
vim revs/.... (do whatever is needed to fix the OBS build)
+
-
git commit -m "Fixed xxx"
+
-
git archive --format=tar --prefix=revs-X.X.X/ release-2011wXX|gzip > cd $(TESTING_OBS_REPO)/revs/revs_X.X.X.tar.gz (replace X.X.X with version you are testing)
+
-
cd $(TESTING_OBS_REPO)/revs/
+
-
osc commit -m "Fixed xxx"
+
-
</pre>
+
-
If the staging environment works and passes the tests, upgrade on production takes place. If it doesn't, an LVM rollback of staging is required and after that, the staging process starts again.
+
=== Components ===
-
== Installing And Setting Up ==
+
There are basically three things that are supposed to differ in each deployed
 +
REVS instance.
-
Installing and setting up REVS is guided in the [http://meego.gitorious.org/meego-infrastructure-tools/revs/blobs/master/README README] file in the source code. It includes the following sections
+
# database
 +
# settings describing the database
 +
# namespace in REVS' root URL
-
* Setting REVS Up
+
There are also  things that are designed to be shared between REVS instances.
-
** Prerequisites
+
-
** Running REVS
+
-
* Admin Interface and Trackers
+
-
** Admin interface
+
-
* Test Data
+
-
** Adding Data
+
-
=== Installing using RPM ===
+
# templates
 +
# static files
 +
# python code
-
When installing REVS via RPM, ther are specific instructions for it. For the rest of the chapters, the installation OS is assumed OpenSUSE, version 11.2.
+
=== Configure custom REVS instance ===
-
When installing on openSuse, the following extra repos are needed :
+
In the instruction below it is assumed that you followed deployment instruction
-
 
+
and installed the default apache & mysql configuration. In case if you decided
-
  zypper ar http://download.opensuse.org/repositories/server:/http/openSUSE_11.2/server:http.repo
+
to install only revs and do web server and database configuration yourself, this
-
  zypper ar http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_11.2/devel:languages:python.repo
+
document is useless for you.
-
Start up by pulling in the REVS repository information and refresh the repositories:
+
In order not to abstract too much, lets configure a blank REVS instance in
 +
a namespace "ce". Below it is assumed that you invoke all commands as root.
-
  zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo
+
The first thing that you have to do is to copy the default REVS settings.py into
-
  zypper ref
+
something with a reasonable name.
-
Then install the revs package:
+
<pre>
-
  zypper in revs
+
    cp /etc/revs/settings.py /etc/revs/ce_settings.py
-
==== Lighttpd configuration ====
+
-
Create a file /etc/lighttpd/vhosts.d/revs.conf, with contents
 
-
<pre>
 
-
var.namebasedir = "/revs"
 
-
$HTTP["url"] =~ "^" + namebasedir {
 
-
        dir-listing.activate = "disable"
 
-
}
 
-
alias.url += (
 
-
  namebasedir + "/static" => "/usr/share/revs/static/",
 
-
  namebasedir + "/media" => "/usr/lib/python2.6/site-packages/django/contrib/admin/media/",
 
-
)
 
-
url.redirect += (
 
-
  "^" + namebasedir + "$" => namebasedir + "/"
 
-
)
 
-
fastcgi.server += (
 
-
    "/revs.fcgi" => (
 
-
        "main" => (
 
-
            "socket" => "/var/run" + namebasedir + ".socket",
 
-
            "check-local" => "disable",
 
-
        )
 
-
    ),
 
-
)
 
-
url.rewrite-if-not-file += (
 
-
  "^(" + namebasedir + "/.*)$" => "/revs.fcgi/$1"
 
-
)
 
</pre>
</pre>
-
And uncomment the following line in /etc/lighttpd/lighttpd.conf:
+
Open the brand new file with you favorite text editor. It currently has the
 +
following content.
-
  include_shell "cat /etc/lighttpd/vhosts.d/*.conf"
+
<pre>
-
Also make sure that modules '''alias''', '''redirect''', '''rewrite''' and '''fastcgi''' are enabled.
+
    from revs.basesettings import *
-
If you did a fresh install of the MySQL database, see the [[#MYSQL_Installation|following]] section. Otherwise skip to [[#Database_creation|database creation]].
+
    FORCE_SCRIPT_NAME = "/revs"
-
==== MYSQL Installation ====
+
    DATABASES = {
-
Mysql database is needed by revs, but the setup is not too hard.
+
        'default': {
 +
            'ENGINE': 'django.db.backends.mysql',
 +
            'NAME': 'revs',
 +
            'USER': 'revs',
 +
            'PASSWORD': 'revs',
 +
            'HOST': '',
 +
            'PORT': ''
 +
        }
 +
    }
-
Issue the following commands:
+
</pre>
-
  rcmysql start
+
There are two things to change. First, '''FORCE_SCRIPT_NAME'''. It has to be equal
-
  mysql_secure_installation
+
to the target namespace. It should start with slash and end without it. For more
 +
information concerning '''FORCE_SCRIPT_NAME''' check official [https://docs.djangoproject.com/en/dev/ref/settings/?from=olddocs#force-script-name Django Docs].
 +
Second, change database name so that new instance does not collide with the
 +
already existing one. After these two modifications your settings shall look
 +
like the ones below. Note, database user and password are not important thus for
 +
the sake of simplicity lets leave them as they are.
-
And answer the questions asked by the installation script. Move on to the next section.
+
<pre>
-
==== Database creation ====
+
    from revs.basesettings import *
-
REVS needs a functional mysql installation in order to work properly. The following script will do just that.
+
    FORCE_SCRIPT_NAME = "/ce"
 +
 
 +
    DATABASES = {
 +
        'default': {
 +
            'ENGINE': 'django.db.backends.mysql',
 +
            'NAME': 'ce',
 +
            'USER': 'revs',
 +
            'PASSWORD': 'revs',
 +
            'HOST': '',
 +
            'PORT': ''
 +
        }
 +
    }
-
<pre>
 
-
mysql -u root -p
 
-
  DROP DATABASE IF EXISTS revs;
 
-
  CREATE DATABASE revs CHARACTER SET utf8;
 
-
  CREATE USER 'revs'@'localhost' IDENTIFIED BY 'revs';
 
-
  GRANT ALL ON revs.* TO 'revs'@'localhost';
 
-
  FLUSH PRIVILEGES;
 
</pre>
</pre>
-
Afterwards issue the following commands:
+
After you changed you settings please link them to '''/usr/share/pyshared/revs/'''.
-
django-admin.py syncdb --settings=revs.settings
+
<pre>
-
django-admin.py migrate --settings=revs.settings
+
-
If you want to create a superuser for Django, you may do so in the previous script.
+
    ln -s /etc/revs/ce_settings.py /usr/share/pyshared/revs/
-
==== Starting up REVS ====
+
</pre>
-
REVS can be started up with its own initscript. Before running it, make sure you have completed each of the above section completely.
+
Additionally you have to link your settings to all python versions that support
-
  /etc/init.d/lighttpd restart
+
revs. It can be done by running '''link_settings.sh''' command which comes
-
  /etc/init.d/revs start
+
together with '''revs''' package.
-
To automatically start REVS on boot do:
+
<pre>
-
chkconfig --add mysql lighttpd revs
+
 +
    link_settings.sh
-
After this, you can continue to [[#Importing_data_to_REVS|importing data]]
+
</pre>
-
=== Install from Debian package ===
+
Now you need to create a database for you brand new REVS instance. Run MySQL
 +
client as a root user.
-
For starters, one can enable the Maemo:MeeGo-Infra repository by adding the following line to /etc/apt/sources.list:
 
<pre>
<pre>
-
deb http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/Debian_5.0/ /
 
-
</pre>
 
-
After adding the repositories, you can install revs by:
+
    mysql -u root -p
-
<pre>
 
-
apt-get update
 
-
apt-get install revs
 
</pre>
</pre>
-
==== Nginx installation ====
+
And execute the following SQL for '''ce''' database name specified in ce_settings.
-
REVS in meego.com uses nginx, so here are the relevant config bits:
+
 
<pre>
<pre>
-
location ^~ /revs/static/ {
 
-
    alias /usr/share/revs/static/;
 
-
}
 
-
location /revs {
 
-
# host and port to fastcgi server
 
-
    fastcgi_pass unix:/var/run/revs.socket;
 
-
    fastcgi_param PATH_INFO $fastcgi_script_name;
 
-
    fastcgi_param REQUEST_METHOD $request_method;
 
-
    fastcgi_param QUERY_STRING $query_string;
 
-
    fastcgi_param CONTENT_TYPE $content_type;
 
-
    fastcgi_param CONTENT_LENGTH $content_length;
 
-
    fastcgi_pass_header Authorization;
 
-
    fastcgi_intercept_errors off;
 
-
}
+
    CREATE DATABASE IF NOT EXISTS ce CHARACTER SET utf8;
 +
    GRANT ALL ON ce.* TO 'revs'@'localhost' IDENTIFIED BY 'revs';
 +
    FLUSH PRIVILEGES;
 +
 
</pre>
</pre>
-
==== Database installation ====
+
Now initialize database structure in this database.
-
Follow the same instructions as in OpenSUSE installation guide, minus that django-admin.py is django-admin in Debian.
+
-
== Testing ==
+
<pre>
-
REVS allows usage of Django test framework to run unit tests on its components. Example is provided by running
+
    python /usr/share/pyshared/revs/manage.py syncdb --settings=ce_settings --noinput
-
<pre>
 
-
run_tests.sh
 
</pre>
</pre>
-
in the source root directory. It reports if all tests passed.
+
The command shall additionally create '''admin''' user with '''admin''' password for
 +
you.
-
== REST ==
+
Now you need to configure the access to '''/ce''' namespace in browser.
-
REVS has an API for controlling the models via HTTP. Currently Baseline, Package, Platform, Product, Product Release and Tracker models are modifiable and readable using our RESTful API.
+
Lets copy the wsgi script responsible for handling REVS instances. Call a new
 +
script '''ce.wsgi''' for the sake of simplicity.
-
Our REST implementation of choice is [http://bitbucket.org/jespern/django-piston/wiki/Home Django Piston], as it allows direct model handling using the Django ORM.
+
<pre>
-
The scripts for [http://meego.gitorious.org/meego-infrastructure-tools/revs/blobs/raw/master/tmp/get.sh reading] and [http://meego.gitorious.org/meego-infrastructure-tools/revs/blobs/raw/master/tmp/prod.json writing] are available in the REVS source repository.
+
    cp /usr/share/revs/apache/revs.wsgi /usr/share/revs/apache/ce.wsgi
-
See the API section for details and examples for each model.
+
</pre>
 +
The script has the following code in it:
 +
<pre>
-
Also the responses are given in JSON, so one might need eg. [http://packages.debian.org/lenny/edit-json edit-json] to view or edit JSON files.
+
    import os
 +
    import sys
 +
 
 +
    os.environ['DJANGO_SETTINGS_MODULE'] = 'revs.settings'
 +
    import django.core.handlers.wsgi
 +
    application = django.core.handlers.wsgi.WSGIHandler()
-
=== API ===
 
-
The API handlers all support either pure url-encoded data or JSON. Each model is capable of both but this document reviews only the JSON input and output.
 
-
==== Urls ====
 
-
The following URLS are supported:
 
-
<pre>
 
-
http://revs.host/revs/api/Package/<name> (Get a package name, list all versions) (Read/Write)
 
-
http://revs.host/revs/api/Package/<name>/<version> (fetches a Package by Version) (Read-only)
 
-
http://revs.host/revs/api/Product/ (shows a list of products) (Read-only)
 
-
http://revs.host/revs/api/Product/<name> (shows specific product) (Read/Write)
 
-
http://revs.host/revs/api/Baseline/ (shows a list of baselines) (Read-only)
 
-
http://revs.host/revs/api/Baseline/?sort={date,name,platform} (shows a list of baselines, sorted by either date, name or platform) (Read-only)
 
-
http://revs.host/revs/api/Baseline/<name> (shows specific baseline) (Read/Write)
 
-
http://revs.host/revs/api/ProductRelease/ (shows a list of product releases) (Read-only)
 
-
http://revs.host/revs/api/ProductRelease/?sort={'name', 'release_of', 'date'} (shows a list of product releases, sorted by either name, release of the product release or date) (Read-only)
 
-
http://revs.host/revs/api/ProductRelease/<name> (specific product release) (Read/Write)
 
-
http://revs.host/revs/api/Tracker/ (shows a list of trackers) (Read-only)
 
-
http://revs.host/revs/api/Tracker/<name> (specific tracker) (Read/Write)
 
</pre>
</pre>
-
Also there are a couple of functional urls:
+
Change it to look like:
<pre>
<pre>
-
http://revs.host/revs/api/Package/<query>/search
 
-
http://revs.host/revs/api/ProductRelease/<query>/search
 
-
</pre>
 
-
----
 
-
==== Baseline ====
+
     import os
-
'''JSON format for input and output'''
+
     import sys
-
<pre> 
+
-
{
+
-
     "platform": {
+
-
        "name": "MeeGo"
+
-
     },
+
-
    "name": "1.1.80.2.20101019.1",
+
-
    "date": "2010-12-12"
+
-
    "pkgs": [
+
-
        {
+
-
            "version": "0.15",
+
-
            "name": "spec-builder"
+
-
        },
+
-
    (many more packages)
+
-
}
+
-
</pre>
+
-
----
+
-
==== Package ====
+
    os.environ['DJANGO_SETTINGS_MODULE'] = 'revs.ce_settings'
 +
    import django.core.handlers.wsgi
 +
    application = django.core.handlers.wsgi.WSGIHandler()
-
'''JSON format for input and output'''
 
-
<pre>
 
-
{
 
-
    "name": "bluez",
 
-
    "platform": "MeeGo",
 
-
    "latest_version": "4.76",
 
-
    "versions": [
 
-
        {
 
-
            "version": "4.70",
 
-
            "package_system": "rpm",
 
-
            "changelogentries": [
 
-
                {
 
-
                    "date": "Thu Sep 30 2010",
 
-
                    "author": "Zhu Yanhai ",
 
-
                    "email": "<yanhai.zhu@linux.intel.com>",
 
-
                    "links": [],
 
-
                    "log": "- Add powered.patch to make sure the initial power status aligned\n  with connman's requirement.\n"
 
-
                },
 
-
                {
 
-
                    "date": "Fri Aug 27 2010",
 
-
                    "author": "Zhu Yanhai ",
 
-
                    "email": "<yanhai.zhu@linux.intel.com>",
 
-
                    "links": [],
 
-
                    "log": "- Upgrade to 4.70\n"
 
-
                }
 
-
            ]
 
-
        },
 
-
    ... (many more versions)
 
-
}
 
</pre>
</pre>
-
----
 
-
==== Platform ====
+
Copy the default apache config.
-
'''JSON format for input and output'''
 
<pre>
<pre>
-
{
 
-
        "name": "MeeGo"
 
-
}
 
-
</pre>
 
-
----
 
-
==== Product ====
 
-
'''JSON format for input and output'''
 
-
<pre>
 
-
{
 
-
    "platforms": [
 
-
        {
 
-
            "name": "MeeGo"
 
-
        }
 
-
    ],
 
-
    "name": "foobar",
 
-
    "target_hardware": "n900"
 
-
}
 
-
</pre>
 
-
----
 
-
==== Product Release ====
+
     cp /etc/apache2/sites-available/revs /etc/apache2/sites-available/ce
-
'''JSON format for input and output'''
+
-
<pre>
+
-
{
+
-
     "release_of": "test",
+
-
    "baselines": [
+
-
        {
+
-
            "name": "1.1.80.3.20101026.1"
+
-
        },
+
-
        {
+
-
            "name": "1.1.80.2.20101019.1"
+
-
        }
+
-
    ],
+
-
    "name": "test_weekly_w45",
+
-
    "date": "2010-17-12"
+
-
}
+
-
</pre>
+
-
----
+
-
==== Tracker ====
 
-
'''JSON format for input and output'''
 
-
<pre>
 
-
{
 
-
        "name": "MeeGoBugzilla",
 
-
        "key2url": "http://bugs.meego.com/show_bug.cgi?id=%key%",
 
-
        "url": "http://bugs.meego.com/",
 
-
        "link_class": "Bug",
 
-
        "filter2url": "<a ref=\"http://bugs.meego.com/show_bug.cgi?id=\\g<key>\">\\g<1></a>",
 
-
        "regexp": "(\\BMC#(?P<key>\\ )\\))"
 
-
        "platforms": [
 
-
            {
 
-
                "name": "MeeGo"
 
-
            }
 
-
        ]
 
-
}
 
</pre>
</pre>
-
----
 
-
== Importing data to REVS ==
+
Open the new file with some text editor. It is supposed to have the following
 +
content:
-
REVS uses 2 main scripts to import data from RPM repositories, [https://meego.gitorious.org/meego-infrastructure-tools/revs/blobs/ext_revived/bin/parse_primary.py parse_primary] and [https://meego.gitorious.org/meego-infrastructure-tools/revs/blobs/ext_revived/bin/parse_other.py parse_other].
+
<pre>
-
Each of these scripts provide different functionality, eg. parse_primary parses the ##hash##-primary.xml.gz and creates proper package, package version, license and many more REVS objects from this information. The script parse_other has the only functionality of collecting changelog information from ##hash##-other.xml.gz and linking this changelog information to package versions (and creating new ones if needed).
+
    # Serve django views
 +
    WSGIScriptAlias /revs /usr/share/revs/apache/revs.wsgi
 +
    WSGIPassAuthorization On
-
=== Usage ===
+
</pre>
-
==== parse_primary.py ====
+
Replace '''/revs''' with '''/ce''' (correlates with the namespace) and '''revs.wsgi'''
 +
with '''ce.wsgi'''. Eventually the file is supposed to look like the one below.
-
parse_primary.py -f --baseline|-b <baseline> --platform|-p <platform> <URL_TO_primary.xml.gz>
+
<pre>
-
;-f
+
    # Serve django views
-
: Forces the creation of a baseline or a platform
+
    WSGIScriptAlias /ce /usr/share/revs/apache/ce.wsgi
-
;-b|--baseline
+
    WSGIPassAuthorization On
-
: Baseline name to use for updating/creation, use -f to create
+
-
;-p|--platform
+
-
: Platform name to use, use -f to create
+
-
==== parse_other.py ====
+
</pre>
-
parse_other.py -f -b|--baseline <URL_TO_other.xml.gz>
+
Enable the site and reload apache web server
-
;-f
+
<pre>
-
: Force creation/update of baseline and its changelogs
+
-
;-b|--baseline
+
-
; Baseline to use for changelogs
+
-
== REVS License ==
+
    a2ensite ce
 +
    /etc/init.d/apache2 reload
-
REVS is free software: you can redistribute it and/or modify
+
</pre>
-
it under the terms of the GNU General Public License as published by
+
-
the Free Software Foundation, either version 2 of the License, or
+
-
(at your option) any later version.
+
-
 
+
-
This program is distributed in the hope that it will be useful,
+
-
but WITHOUT ANY WARRANTY; without even the implied warranty of
+
-
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+
-
GNU General Public License for more details.
+
-
You should have received a copy of the GNU General Public License
+
Go to '''DOMAIN_NAME/ce''' in your browser and enjoy.
-
along with this program.  If not, see <http://www.gnu.org/licenses/>.
+

Latest revision as of 10:30, 12 April 2012

Contents

REVS

Deployment steps

This instruction provides an overview of deployment process of revs from the sources fetched from gitorious. The result of the deployment is fully functional tool ready to be used in production environment. Currently available for Debian only

Install REVS from OBS repo (preferred)

Edit your sources.list.


    vim /etc/apt/sources.list

Add the following lines there:


    deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0 ./
    deb http://download.opensuse.org/repositories/openSUSE:/Tools/Debian_6.0/  /

Install LAMP (Linux Apache MySQL Python) configuration of REVS.


    apt-get update
    apt-get install revs-lamp

If this is a "fresh" REVS install, you will be asked to type you root MySQL user password.

After the installation is over you shall be able to access blank (with no data in it) instance of REVS via DOMAIN/revs url. You can also access admin site via DOMAIN/revs/admin url. Use admin/admin credentials.

Install REVS from GIT (if you want to play with sources)

Fetch package source from gitorious.


    git clone git://gitorious.org/meego-infrastructure-tools/revs.git

Install all build dependencies.


    sudo apt-get install yum python-django debhelper python-support python-setuptools python-sphinx python-sqlite

Build the package.


    cd revs
    dpkg-buildpackage -rfakeroot -uc -us -sa -tc -D
    cd ..

Install REVS and REVS' lamp configuration binary dependencies


    sudo apt-get install mysql-server apache2 libapache2-mod-wsgi python-mysqldb

Install REVS together with LAMP configuration. Note that revs itself has to be installed before LAMP configuration.


    sudo dpkg -i revs_*.deb
    sudo dpkg -i revs-lamp_*.deb

After installing REVS you may go to DOMAIN/revs to use the tool or to DOMAIN/revs/admin to access admin interface. The default admin user has username admin and password admin.

Sync REVS with OBS instance

To sync with an OBS instance, use the admin UI to populate the projects. The names *MUST* be the sames as OBS and their repositories must be accessible to REVS.

After this is done, you can proceed by initially seed the database.

Seeding database from initial base packages

django-admin import_from_obs --settings=revs.settings

This will create initial packages to a Platform and their versions to respective projects created in admin UI.

Keeping REVS automatically updating

Install revs-boss package so that you will get two participants: feed_revs and revs_image. The first is the package updater and second is an image importer, in case you are using IMG participant in the BOSS process. Remember to enable at least feed_revs.

Customization

The goal of this article is to provide a comprehensive instruction on how to deploy several custom instances of REVS on the same machine.

Components

There are basically three things that are supposed to differ in each deployed REVS instance.

  1. database
  2. settings describing the database
  3. namespace in REVS' root URL

There are also things that are designed to be shared between REVS instances.

  1. templates
  2. static files
  3. python code

Configure custom REVS instance

In the instruction below it is assumed that you followed deployment instruction and installed the default apache & mysql configuration. In case if you decided to install only revs and do web server and database configuration yourself, this document is useless for you.

In order not to abstract too much, lets configure a blank REVS instance in a namespace "ce". Below it is assumed that you invoke all commands as root.

The first thing that you have to do is to copy the default REVS settings.py into something with a reasonable name.


    cp /etc/revs/settings.py /etc/revs/ce_settings.py

Open the brand new file with you favorite text editor. It currently has the following content.


    from revs.basesettings import *

    FORCE_SCRIPT_NAME = "/revs"

    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'NAME': 'revs',
            'USER': 'revs',
            'PASSWORD': 'revs',
            'HOST': '',
            'PORT': ''
        }
    }

There are two things to change. First, FORCE_SCRIPT_NAME. It has to be equal to the target namespace. It should start with slash and end without it. For more information concerning FORCE_SCRIPT_NAME check official Django Docs. Second, change database name so that new instance does not collide with the already existing one. After these two modifications your settings shall look like the ones below. Note, database user and password are not important thus for the sake of simplicity lets leave them as they are.


    from revs.basesettings import *

    FORCE_SCRIPT_NAME = "/ce"

    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'NAME': 'ce',
            'USER': 'revs',
            'PASSWORD': 'revs',
            'HOST': '',
            'PORT': ''
        }
    }

After you changed you settings please link them to /usr/share/pyshared/revs/.


    ln -s /etc/revs/ce_settings.py /usr/share/pyshared/revs/

Additionally you have to link your settings to all python versions that support revs. It can be done by running link_settings.sh command which comes together with revs package.


    link_settings.sh

Now you need to create a database for you brand new REVS instance. Run MySQL client as a root user.


    mysql -u root -p

And execute the following SQL for ce database name specified in ce_settings.


    CREATE DATABASE IF NOT EXISTS ce CHARACTER SET utf8;
    GRANT ALL ON ce.* TO 'revs'@'localhost' IDENTIFIED BY 'revs';
    FLUSH PRIVILEGES;

Now initialize database structure in this database.


    python /usr/share/pyshared/revs/manage.py syncdb --settings=ce_settings --noinput

The command shall additionally create admin user with admin password for you.

Now you need to configure the access to /ce namespace in browser.

Lets copy the wsgi script responsible for handling REVS instances. Call a new script ce.wsgi for the sake of simplicity.


    cp /usr/share/revs/apache/revs.wsgi /usr/share/revs/apache/ce.wsgi

The script has the following code in it:


    import os
    import sys

    os.environ['DJANGO_SETTINGS_MODULE'] = 'revs.settings'
    import django.core.handlers.wsgi
    application = django.core.handlers.wsgi.WSGIHandler()

Change it to look like:


    import os
    import sys

    os.environ['DJANGO_SETTINGS_MODULE'] = 'revs.ce_settings'
    import django.core.handlers.wsgi
    application = django.core.handlers.wsgi.WSGIHandler()

Copy the default apache config.


    cp /etc/apache2/sites-available/revs /etc/apache2/sites-available/ce

Open the new file with some text editor. It is supposed to have the following content:


    # Serve django views
    WSGIScriptAlias /revs /usr/share/revs/apache/revs.wsgi
    WSGIPassAuthorization On

Replace /revs with /ce (correlates with the namespace) and revs.wsgi with ce.wsgi. Eventually the file is supposed to look like the one below.


    # Serve django views
    WSGIScriptAlias /ce /usr/share/revs/apache/ce.wsgi
    WSGIPassAuthorization On

Enable the site and reload apache web server


    a2ensite ce
    /etc/init.d/apache2 reload

Go to DOMAIN_NAME/ce in your browser and enjoy.

Personal tools