MeeGo wiki - User contributions [en]

Getting started with OBS

2011-12-06T15:39:22Z

Iamer: /* Adding repositories */

The MeeGo Open Build System (OBS) seems rather complicated, this page will attempt to make it nice and easy to get started. See also the Mer project's documentation on [http://wiki.maemo.org/Mer/Build/Application_Building application building] with OBS.

It's a work in progress, feel free to add and modify!

=Prerequisites=

* You need a meego.com login to be able to access the OBS
* Install osc (for example, <code>apt-get install osc</code> may work out-of-the-box on recent versions of Ubuntu & Debian, other repositories are available on [http://download.opensuse.org/repositories/openSUSE:/Tools/ openSUSE tools])
* As an alternative to osc you may use the web interface to upload your files

=Preparations=

OBS is based around ''projects'', ''packages'' and ''repositories''. '''Packages''' have a set of files (source code, packaging meta-data, etc.) which are compiled by build servers against particular distributions. Each distribution results in your code being available in a '''repository'''.

==Setting up your home project==

Your home project is like a home directory in which you can create sub-projects, repositories, etc.

* Visit the [https://build.pub.meego.com/ OBS page]
* Follow the extra [https://build.pub.meego.com/user/register_user registration instructions]
* Log in and click on 'create home project' (down at the bottom of the page iirc)

==Creating a project==
A project is required before anything can be built.

* Go to the "Projects" tab for the project in which you wish to create it
* Select "Add a new package"
* Enter a UNIX-style (all lower case) "Name"
* Enter a title-case, with appropriate spacing, "Title"
* Enter a brief "Description"
* Click "Save Changes"

==Setting up a sub-project (optional)==
A sub-project allows you to group related activities together.

* In the "Overview" tab for your home project, select "Actions > Create subproject".

==Adding repositories==

* Go to the "Repositories" tab for the project in question
* Click "Add Repository"

As Harmattan OBS support is still in development, you can [http://forum.meego.com/showpost.php?p=25005&postcount=29 add it through the advanced interface].

Before trying to issue submit requests to projects that are under automated integration (ex. CE:* projects on community OBS), make sure you have created the correct set of repositories needed.

For each repository in target project, there should be a repository in source project which builds only against the target repository and have at least the same architectures as the target repository they build against.

This can be done easily using [https://github.com/MeeGoIntegration/mint-utils/blob/master/addrepos.py the addrepos osc plugin ], drop it in your ~/.osc-plugins and run the command :

osc -A https://api.pub.meego.com addrepos MYPROJECT TARGETPROJECT

It will open an editor with the repositories appended to the bottom of your prj meta.

=Adding files to a package and building them=

==Needed files==

===MeeGo===
For MeeGo you need a .spec file and a tarball containing your source code, etc.

===Harmattan===
For Harmattan you need a <code>.dsc</code> file, source tarball and <code>.debian.tar.gz</code> (this allows one to keep the debian directory out of the source). The source is extracted using <code>dpkg-source -x ....dsc</code>, so restrictions on file formats apply (e.g. <code>mypkg_1.0.0.tar.gz</code>, <code>mypkg_1.0.0.dsc</code>, <code>mypkg_1.0.0.debian.tar.gz</code>)

Alternatively, you may just use the <code>.dsc</code> together with a <code>.tar.gz</code> file containing both, the sources and the debian directory.

====Use an additional tool====
The steps below are encapsulated in a follow-up to [http://mud-builder.garage.maemo.org/ mud-builder], called <code>mud2</code>. The [http://forum.meego.com/showpost.php?p=24047&postcount=15 tool only has a dependency on Perl] and performs all the steps below, hoping to streamline the process of taking Qt Creator projects and producing MeeGo, Harmattan and Fremantle OBS-compatible files.

====Set up QtCreator to produce files====
'''First''', you need to modify the rules file (<code>qtc_packaging/debian_harmattan/rules</code>, which is then copied to <code>debian/rules</code>):
* Uncomment the lines marked with <code># Uncomment this line for use without Qt Creator</code>.
* In the <code>clean</code> section, comment the line <code>$(MAKE) clean</code>

'''Second''', to get the files needed for building you may add a custom build step:
* Projects -> Harmattan -> Build -> Build Steps -> Add Build Step
* Enter Command: <code>C:\QtSDK\Madde\wbin\mad.cmd</code> or in Linux use <code>/opt/QtSDK/Madde/bin/mad</code>
* Leave Working directory unchanged: <code>%{buildDir}</code>
* Enter Command arguments: <code>-t harmattan-platform-api dpkg-buildpackage -sa -S -uc -us -Imoc -Iobj -Ircc -Iui -I.svn -I*.deb -I*.changes -Iqtc_packaging -IMakefile -I*.pro.user -I<packagename></code>
Notice that most of the command arguments remove unneccessary files from the tar package. They are not mandatory but reduce file size.
* Build it
You will get the resulting <code><packagename>-<version>.tar.gz</code> and <code><packagename>-<version>.dsc</code> (together with a non-needed <code><packagename>-<version>.source.changes</code>) in the directory above the source directory.

'''Take care''':
QtCreator uses the <code>debian</code> directory for packaging. However, it only copies the content of <code>qtc_packaging/debian_harmattan</code> into <code>debian</code> when you deploy the package (thus press run or debug). If you change something in <code>qtc_packaging/debian_harmattan</code> and only run a build without run/debug, you must manually copy the changes to <code>debian</code> before.

'''Security framework relevant applications''':
The <code>rules</code> file must include a step to add the <code><package name>.aegis</code> manifest into the built debian archive. For this, add <code>aegis-deb-add -control debian/<package name>/DEBIAN/control .. debian/<package name>.aegis=_aegis</code> at the end (after <code>dh_builddeb</code>). Needed build dependency in the <code>control</code> file is <code>aegis-builder (>= 1.6)</code>.

'''If you are using Windows or having build problems with changelog or rules file:'''
It seams that QTCreator for Windows sometimes creates the changelog and rules file with windows lineendings.
Which causes the OBS to fail. (a <code>rules</code> file with wrong lineendings causes the <code>clean</code> step to fail.

==Start building==

===From the terminal===
* Checkout (like SVN) your home project, which will contain all your sub-projects and packages
osc -A https://api.pub.meego.com co home:[username]

* Add files to your local copy
osc add [filename]

* Commit the files
osc commit

At this point the OBS will try to build your project remotely. You can also build locally by doing <code>osc build</code>

If you add new packages later on or commit changes from another computer you can update your local copy by doing
osc update

===Using the webinterface===
* Add the correct files as listed above and they will be automatically built for you

==Debugging==
What to do if packages don't build in OBS.

===Harmattan===
It is advised to first try if a package builds in the Scratchbox based SDK before submitting it to OBS. The Scratchbox environment should be reasonably similar to OBS and lower rebuild time enables faster debugging of packaging issues.

Also, some packages from recent distributions might complain about out of date build dependencies, mostly concerning debhelper and cdbs. As debhelper and cdb are generally not easy to replace with newer versions, it is possible to either use older packages or edit the '''debian/control''' and '''debian/control.in''' and downgrade the dependencies.

===OBS local build===
It is possible to build packages locally, using the osc tool. Also unlike during the online build, it is possible to enter the build chroot and inspect its state. Also, local builds do not need to wait in the online build queue and may be faster (depending on local hardware).

* install osc
* install [http://download.opensuse.org/repositories/openSUSE:/Tools/ OBS build tools]
* install quemu
* check out a project with some packages
<pre>
osc co home:name:project
</pre>
* enter a package folder folder
<pre>
cd /home:name:project/package
</pre>
* initiate local build
<pre>
osc build
</pre>
====COBS+Harmattan====
* to do a local build with the Harmattan target from COBS:
<pre>
osc build MeeGo_1.2_Harmattan_Maemo.org_MeeGo_1.2_Harmattan_standard armv7el
</pre>

====Possible pitfalls====

'''Problem:'''
Harmattan packages might fail to download (404) during a local build.

'''Solution:'''
Just wait, after all of the packages have failed it will use an alternate download method that will work.

'''Problem:'''
<pre>
failed to run command `/sbin/ldconfig': No such file or directory
</pre>
and
<pre>
chroot: failed to run command `dpkg': No such file or directory
</pre>
'''Solution:'''There are some Qemu packages missing in you system (most probably qemu-user-static or an equivalent).

On Ubuntu 11.04, these packages should be installed:
<pre>
qemu-user-static qemu-common qemu-kvm qemu
</pre>

'''NOTE:''' Looks like even just ''qemu-user-static'' package suffices for builds to run to completion.

=Installing packages=
== Into your MeeGo system ==
{{main|Zypper#Adding_a_repository}}
* Once packages are built, you can include them in your MeeGo installation.
* The general method is:
zypper ar -f <nowiki>http://repo.pub.meego.com/home:/username/repository/</nowiki> title
zypper in <package>
** Where:
** ''username'' is your user
** ''repository'' is the repository branch name
** ''title'' is the name zypper will use for the repository

== Into Harmattan ==
# Find the repository address:
## Go to the appropriate project's ''Overview'' page (e.g. your home project, or sub-project)
## Click on the MeeGo 1.2 Harmattan repository on the ''Build Status''
## Copy the link for the ''Download Repository''
* Create <code>/etc/apt/sources.list.d/project.list</code> (where ''project'' is a memorable name, e.g. your username for your home project), containing the download link, prefixed with '''<code>deb</code>''' and suffixed by '''<code>./</code>'''. For example:
deb <nowiki>http://repo.pub.meego.com/home:/jaffa/Harmattan/</nowiki> ./
* Run:
apt-get update
apt-get install package
:...where ''package'' is, of course, your package name.

Updates to the installed applications will appear at ''Settings > Applications > Manage Applications > Updates''

== Into Maemo 5 ==
Enable Extras-devel.

Getting started with OBS

2011-11-22T06:26:11Z

Iamer: /* Adding repositories */

The MeeGo Open Build System (OBS) seems rather complicated, this page will attempt to make it nice and easy to get started. See also the Mer project's documentation on [http://wiki.maemo.org/Mer/Build/Application_Building application building] with OBS.

It's a work in progress, feel free to add and modify!

=Prerequisites=

* You need a meego.com login to be able to access the OBS
* Install osc (for example, <code>apt-get install osc</code> may work out-of-the-box on recent versions of Ubuntu & Debian, other repositories are available on [http://download.opensuse.org/repositories/openSUSE:/Tools/ openSUSE tools])
* As an alternative to osc you may use the web interface to upload your files

=Preparations=

OBS is based around ''projects'', ''packages'' and ''repositories''. '''Packages''' have a set of files (source code, packaging meta-data, etc.) which are compiled by build servers against particular distributions. Each distribution results in your code being available in a '''repository'''.

==Setting up your home project==

Your home project is like a home directory in which you can create sub-projects, repositories, etc.

* Visit the [https://build.pub.meego.com/ OBS page]
* Follow the extra [https://build.pub.meego.com/user/register_user registration instructions]
* Log in and click on 'create home project' (down at the bottom of the page iirc)

==Creating a project==
A project is required before anything can be built.

* Go to the "Projects" tab for the project in which you wish to create it
* Select "Add a new package"
* Enter a UNIX-style (all lower case) "Name"
* Enter a title-case, with appropriate spacing, "Title"
* Enter a brief "Description"
* Click "Save Changes"

==Setting up a sub-project (optional)==
A sub-project allows you to group related activities together.

* In the "Overview" tab for your home project, select "Actions > Create subproject".

==Adding repositories==

* Go to the "Repositories" tab for the project in question
* Click "Add Repository"

As Harmattan OBS support is still in development, you can [http://forum.meego.com/showpost.php?p=25005&postcount=29 add it through the advanced interface].

Before trying to issue submit requests to projects that are under automated integration (ex. CE:* projects on community OBS), make sure you have created the correct set of repositories needed.

For each repository in target project, there should be a repository in source project which builds only against the target repository and have at least the same architectures as the target repository they build against.

This can be done easily using [https://github.com/iamer/useful-scripts/blob/master/addrepos.py the addrepos osc plugin ], drop it in your ~/.osc-plugins and run the command :

osc -A https://api.pub.meego.com addrepos MYPROJECT TARGETPROJECT

It will open an editor with the repositories appended to the bottom of your prj meta.

=Adding files to a package and building them=

==Needed files==

===MeeGo===
For MeeGo you need a .spec file and a tarball containing your source code, etc.

===Harmattan===
For Harmattan you need a <code>.dsc</code> file, source tarball and <code>.debian.tar.gz</code> (this allows one to keep the debian directory out of the source). The source is extracted using <code>dpkg-source -x ....dsc</code>, so restrictions on file formats apply (e.g. <code>mypkg_1.0.0.tar.gz</code>, <code>mypkg_1.0.0.dsc</code>, <code>mypkg_1.0.0.debian.tar.gz</code>)

Alternatively, you may just use the <code>.dsc</code> together with a <code>.tar.gz</code> file containing both, the sources and the debian directory.

====Use an additional tool====
The steps below are encapsulated in a follow-up to [http://mud-builder.garage.maemo.org/ mud-builder], called <code>mud2</code>. The [http://forum.meego.com/showpost.php?p=24047&postcount=15 tool only has a dependency on Perl] and performs all the steps below, hoping to streamline the process of taking Qt Creator projects and producing MeeGo, Harmattan and Fremantle OBS-compatible files.

====Set up QtCreator to produce files====
'''First''', you need to modify the rules file (<code>qtc_packaging/debian_harmattan/rules</code>, which is then copied to <code>debian/rules</code>):
* Uncomment the lines marked with <code># Uncomment this line for use without Qt Creator</code>.
* In the <code>clean</code> section, comment the line <code>$(MAKE) clean</code>

'''Second''', to get the files needed for building you may add a custom build step:
* Projects -> Harmattan -> Build -> Build Steps -> Add Build Step
* Enter Command: <code>C:\QtSDK\Madde\wbin\mad.cmd</code> or in Linux use <code>/opt/QtSDK/Madde/bin/mad</code>
* Leave Working directory unchanged: <code>%{buildDir}</code>
* Enter Command arguments: <code>-t harmattan-platform-api dpkg-buildpackage -sa -S -uc -us -Imoc -Iobj -Ircc -Iui -I.svn -I*.deb -I*.changes -Iqtc_packaging -IMakefile -I*.pro.user -I<packagename></code>
Notice that most of the command arguments remove unneccessary files from the tar package. They are not mandatory but reduce file size.
* Build it
You will get the resulting <code><packagename>-<version>.tar.gz</code> and <code><packagename>-<version>.dsc</code> (together with a non-needed <code><packagename>-<version>.source.changes</code>) in the directory above the source directory.

'''Take care''':
QtCreator uses the <code>debian</code> directory for packaging. However, it only copies the content of <code>qtc_packaging/debian_harmattan</code> into <code>debian</code> when you deploy the package (thus press run or debug). If you change something in <code>qtc_packaging/debian_harmattan</code> and only run a build without run/debug, you must manually copy the changes to <code>debian</code> before.

'''Security framework relevant applications''':
The <code>rules</code> file must include a step to add the <code><package name>.aegis</code> manifest into the built debian archive. For this, add <code>aegis-deb-add -control debian/<package name>/DEBIAN/control .. debian/<package name>.aegis=_aegis</code> at the end (after <code>dh_builddeb</code>). Needed build dependency in the <code>control</code> file is <code>aegis-builder (>= 1.6)</code>.

'''If you are using Windows or having build problems with changelog or rules file:'''
It seams that QTCreator for Windows sometimes creates the changelog and rules file with windows lineendings.
Which causes the OBS to fail. (a <code>rules</code> file with wrong lineendings causes the <code>clean</code> step to fail.

==Start building==

===From the terminal===
* Checkout (like SVN) your home project, which will contain all your sub-projects and packages
osc -A https://api.pub.meego.com co home:[username]

* Add files to your local copy
osc add [filename]

* Commit the files
osc commit

At this point the OBS will try to build your project remotely. You can also build locally by doing <code>osc build</code>

If you add new packages later on or commit changes from another computer you can update your local copy by doing
osc update

===Using the webinterface===
* Add the correct files as listed above and they will be automatically built for you

==Debugging==
What to do if packages don't build in OBS.

===Harmattan===
It is advised to first try if a package builds in the Scratchbox based SDK before submitting it to OBS. The Scratchbox environment should be reasonably similar to OBS and lower rebuild time enables faster debugging of packaging issues.

Also, some packages from recent distributions might complain about out of date build dependencies, mostly concerning debhelper and cdbs. As debhelper and cdb are generally not easy to replace with newer versions, it is possible to either use older packages or edit the '''debian/control''' and '''debian/control.in''' and downgrade the dependencies.

===OBS local build===
It is possible to build packages locally, using the osc tool. Also unlike during the online build, it is possible to enter the build chroot and inspect its state. Also, local builds do not need to wait in the online build queue and may be faster (depending on local hardware).

* install osc
* install [http://download.opensuse.org/repositories/openSUSE:/Tools/ OBS build tools]
* install quemu
* check out a project with some packages
<pre>
osc co home:name:project
</pre>
* enter a package folder folder
<pre>
cd /home:name:project/package
</pre>
* initiate local build
<pre>
osc build
</pre>
====COBS+Harmattan====
* to do a local build with the Harmattan target from COBS:
<pre>
osc build MeeGo_1.2_Harmattan_Maemo.org_MeeGo_1.2_Harmattan_standard armv7el
</pre>

====Possible pitfalls====

'''Problem:'''
Harmattan packages might fail to download (404) during a local build.

'''Solution:'''
Just wait, after all of the packages have failed it will use an alternate download method that will work.

'''Problem:'''
<pre>
failed to run command `/sbin/ldconfig': No such file or directory
</pre>
and
<pre>
chroot: failed to run command `dpkg': No such file or directory
</pre>
'''Solution:'''There are some Qemu packages missing in you system (most probably qemu-user-static or an equivalent).

On Ubuntu 11.04, these packages should be installed:
<pre>
qemu-user-static qemu-common qemu-kvm qemu
</pre>

'''NOTE:''' Looks like even just ''qemu-user-static'' package suffices for builds to run to completion.

=Installing packages=
== Into your MeeGo system ==
{{main|Zypper#Adding_a_repository}}
* Once packages are built, you can include them in your MeeGo installation.
* The general method is:
zypper ar -f <nowiki>http://repo.pub.meego.com/home:/username/repository/</nowiki> title
zypper in <package>
** Where:
** ''username'' is your user
** ''repository'' is the repository branch name
** ''title'' is the name zypper will use for the repository

== Into Harmattan ==
# Find the repository address:
## Go to the appropriate project's ''Overview'' page (e.g. your home project, or sub-project)
## Click on the MeeGo 1.2 Harmattan repository on the ''Build Status''
## Copy the link for the ''Download Repository''
* Create <code>/etc/apt/sources.list.d/project.list</code> (where ''project'' is a memorable name, e.g. your username for your home project), containing the download link, prefixed with '''<code>deb</code>''' and suffixed by '''<code>./</code>'''. For example:
deb <nowiki>http://repo.pub.meego.com/home:/jaffa/Harmattan/</nowiki> ./
* Run:
apt-get update
apt-get install package
:...where ''package'' is, of course, your package name.

Updates to the installed applications will appear at ''Settings > Applications > Manage Applications > Updates''

== Into Maemo 5 ==
Enable Extras-devel.

Getting started with OBS

2011-11-22T06:18:47Z

Iamer: /* Adding repositories */

The MeeGo Open Build System (OBS) seems rather complicated, this page will attempt to make it nice and easy to get started. See also the Mer project's documentation on [http://wiki.maemo.org/Mer/Build/Application_Building application building] with OBS.

It's a work in progress, feel free to add and modify!

=Prerequisites=

* You need a meego.com login to be able to access the OBS
* Install osc (for example, <code>apt-get install osc</code> may work out-of-the-box on recent versions of Ubuntu & Debian, other repositories are available on [http://download.opensuse.org/repositories/openSUSE:/Tools/ openSUSE tools])
* As an alternative to osc you may use the web interface to upload your files

=Preparations=

OBS is based around ''projects'', ''packages'' and ''repositories''. '''Packages''' have a set of files (source code, packaging meta-data, etc.) which are compiled by build servers against particular distributions. Each distribution results in your code being available in a '''repository'''.

==Setting up your home project==

Your home project is like a home directory in which you can create sub-projects, repositories, etc.

* Visit the [https://build.pub.meego.com/ OBS page]
* Follow the extra [https://build.pub.meego.com/user/register_user registration instructions]
* Log in and click on 'create home project' (down at the bottom of the page iirc)

==Creating a project==
A project is required before anything can be built.

* Go to the "Projects" tab for the project in which you wish to create it
* Select "Add a new package"
* Enter a UNIX-style (all lower case) "Name"
* Enter a title-case, with appropriate spacing, "Title"
* Enter a brief "Description"
* Click "Save Changes"

==Setting up a sub-project (optional)==
A sub-project allows you to group related activities together.

* In the "Overview" tab for your home project, select "Actions > Create subproject".

==Adding repositories==

* Go to the "Repositories" tab for the project in question
* Click "Add Repository"

As Harmattan OBS support is still in development, you can [http://forum.meego.com/showpost.php?p=25005&postcount=29 add it through the advanced interface].

Before trying to issue submit requests to projects that are under automated integration (ex. CE:* projects on community OBS), make sure you have created the correct set of repositories needed.

For each repository in target project, there should be a repository in source project which builds only against the target repository and have at least the same architectures as the target repository they build against.

This can be done easily using the attached osc plugin, drop it in your ~/.osc-plugins and run the command :

osc -A https://api.pub.meego.com addrepos MYPROJECT TARGETPROJECT

It will open an editor with the repositories appended to the bottom of your prj meta.

=Adding files to a package and building them=

==Needed files==

===MeeGo===
For MeeGo you need a .spec file and a tarball containing your source code, etc.

===Harmattan===
For Harmattan you need a <code>.dsc</code> file, source tarball and <code>.debian.tar.gz</code> (this allows one to keep the debian directory out of the source). The source is extracted using <code>dpkg-source -x ....dsc</code>, so restrictions on file formats apply (e.g. <code>mypkg_1.0.0.tar.gz</code>, <code>mypkg_1.0.0.dsc</code>, <code>mypkg_1.0.0.debian.tar.gz</code>)

Alternatively, you may just use the <code>.dsc</code> together with a <code>.tar.gz</code> file containing both, the sources and the debian directory.

====Use an additional tool====
The steps below are encapsulated in a follow-up to [http://mud-builder.garage.maemo.org/ mud-builder], called <code>mud2</code>. The [http://forum.meego.com/showpost.php?p=24047&postcount=15 tool only has a dependency on Perl] and performs all the steps below, hoping to streamline the process of taking Qt Creator projects and producing MeeGo, Harmattan and Fremantle OBS-compatible files.

====Set up QtCreator to produce files====
'''First''', you need to modify the rules file (<code>qtc_packaging/debian_harmattan/rules</code>, which is then copied to <code>debian/rules</code>):
* Uncomment the lines marked with <code># Uncomment this line for use without Qt Creator</code>.
* In the <code>clean</code> section, comment the line <code>$(MAKE) clean</code>

'''Second''', to get the files needed for building you may add a custom build step:
* Projects -> Harmattan -> Build -> Build Steps -> Add Build Step
* Enter Command: <code>C:\QtSDK\Madde\wbin\mad.cmd</code> or in Linux use <code>/opt/QtSDK/Madde/bin/mad</code>
* Leave Working directory unchanged: <code>%{buildDir}</code>
* Enter Command arguments: <code>-t harmattan-platform-api dpkg-buildpackage -sa -S -uc -us -Imoc -Iobj -Ircc -Iui -I.svn -I*.deb -I*.changes -Iqtc_packaging -IMakefile -I*.pro.user -I<packagename></code>
Notice that most of the command arguments remove unneccessary files from the tar package. They are not mandatory but reduce file size.
* Build it
You will get the resulting <code><packagename>-<version>.tar.gz</code> and <code><packagename>-<version>.dsc</code> (together with a non-needed <code><packagename>-<version>.source.changes</code>) in the directory above the source directory.

'''Take care''':
QtCreator uses the <code>debian</code> directory for packaging. However, it only copies the content of <code>qtc_packaging/debian_harmattan</code> into <code>debian</code> when you deploy the package (thus press run or debug). If you change something in <code>qtc_packaging/debian_harmattan</code> and only run a build without run/debug, you must manually copy the changes to <code>debian</code> before.

'''Security framework relevant applications''':
The <code>rules</code> file must include a step to add the <code><package name>.aegis</code> manifest into the built debian archive. For this, add <code>aegis-deb-add -control debian/<package name>/DEBIAN/control .. debian/<package name>.aegis=_aegis</code> at the end (after <code>dh_builddeb</code>). Needed build dependency in the <code>control</code> file is <code>aegis-builder (>= 1.6)</code>.

'''If you are using Windows or having build problems with changelog or rules file:'''
It seams that QTCreator for Windows sometimes creates the changelog and rules file with windows lineendings.
Which causes the OBS to fail. (a <code>rules</code> file with wrong lineendings causes the <code>clean</code> step to fail.

==Start building==

===From the terminal===
* Checkout (like SVN) your home project, which will contain all your sub-projects and packages
osc -A https://api.pub.meego.com co home:[username]

* Add files to your local copy
osc add [filename]

* Commit the files
osc commit

At this point the OBS will try to build your project remotely. You can also build locally by doing <code>osc build</code>

If you add new packages later on or commit changes from another computer you can update your local copy by doing
osc update

===Using the webinterface===
* Add the correct files as listed above and they will be automatically built for you

==Debugging==
What to do if packages don't build in OBS.

===Harmattan===
It is advised to first try if a package builds in the Scratchbox based SDK before submitting it to OBS. The Scratchbox environment should be reasonably similar to OBS and lower rebuild time enables faster debugging of packaging issues.

Also, some packages from recent distributions might complain about out of date build dependencies, mostly concerning debhelper and cdbs. As debhelper and cdb are generally not easy to replace with newer versions, it is possible to either use older packages or edit the '''debian/control''' and '''debian/control.in''' and downgrade the dependencies.

===OBS local build===
It is possible to build packages locally, using the osc tool. Also unlike during the online build, it is possible to enter the build chroot and inspect its state. Also, local builds do not need to wait in the online build queue and may be faster (depending on local hardware).

* install osc
* install [http://download.opensuse.org/repositories/openSUSE:/Tools/ OBS build tools]
* install quemu
* check out a project with some packages
<pre>
osc co home:name:project
</pre>
* enter a package folder folder
<pre>
cd /home:name:project/package
</pre>
* initiate local build
<pre>
osc build
</pre>
====COBS+Harmattan====
* to do a local build with the Harmattan target from COBS:
<pre>
osc build MeeGo_1.2_Harmattan_Maemo.org_MeeGo_1.2_Harmattan_standard armv7el
</pre>

====Possible pitfalls====

'''Problem:'''
Harmattan packages might fail to download (404) during a local build.

'''Solution:'''
Just wait, after all of the packages have failed it will use an alternate download method that will work.

'''Problem:'''
<pre>
failed to run command `/sbin/ldconfig': No such file or directory
</pre>
and
<pre>
chroot: failed to run command `dpkg': No such file or directory
</pre>
'''Solution:'''There are some Qemu packages missing in you system (most probably qemu-user-static or an equivalent).

On Ubuntu 11.04, these packages should be installed:
<pre>
qemu-user-static qemu-common qemu-kvm qemu
</pre>

'''NOTE:''' Looks like even just ''qemu-user-static'' package suffices for builds to run to completion.

=Installing packages=
== Into your MeeGo system ==
{{main|Zypper#Adding_a_repository}}
* Once packages are built, you can include them in your MeeGo installation.
* The general method is:
zypper ar -f <nowiki>http://repo.pub.meego.com/home:/username/repository/</nowiki> title
zypper in <package>
** Where:
** ''username'' is your user
** ''repository'' is the repository branch name
** ''title'' is the name zypper will use for the repository

== Into Harmattan ==
# Find the repository address:
## Go to the appropriate project's ''Overview'' page (e.g. your home project, or sub-project)
## Click on the MeeGo 1.2 Harmattan repository on the ''Build Status''
## Copy the link for the ''Download Repository''
* Create <code>/etc/apt/sources.list.d/project.list</code> (where ''project'' is a memorable name, e.g. your username for your home project), containing the download link, prefixed with '''<code>deb</code>''' and suffixed by '''<code>./</code>'''. For example:
deb <nowiki>http://repo.pub.meego.com/home:/jaffa/Harmattan/</nowiki> ./
* Run:
apt-get update
apt-get install package
:...where ''package'' is, of course, your package name.

Updates to the installed applications will appear at ''Settings > Applications > Manage Applications > Updates''

== Into Maemo 5 ==
Enable Extras-devel.

Release Infrastructure/BOSS/Installation

2011-08-25T14:42:05Z

Iamer: /* Configuration */

= Overview =

Whilst BOSS is deployed in production, this packaged version is still
in beta-testing and is subject to change. In particular the
repositories are not yet finalised (eg installing from :RC).

However BOSS and SkyNET both install and run at a basic level on a
clean Debian 6.0 or OpenSuse 11.4 installation.

= Installation =

== Preparation ==

* Proxy : On OpenSUSE you need to set the proxy in /etc/sysconfig/proxy

== BOSS ==
=== Overview ===
BOSS provides the ruote worker and the main AMQP server.

This component is the main process dispatcher; no work happens
here but all process steps are initiated from here.

Currently BOSS uses the Ruote filesystem storage system and the
worker and viewer need to run on the same system. This will change
when BOSS moves to Redis.

The stable and testing projects are at the community OBS, the state
of the packages can be monitored at :

Testing : https://build.pub.meego.com/project/monitor?project=Project:MINT:Testing

Release Candidate : https://build.pub.meego.com/project/monitor?project=Project:MINT:RC

Stable : (No release yet) https://build.pub.meego.com/project/monitor?project=Project:MINT

=== SUSE ===

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo
zypper ref
zypper in boss

(new version of BOSS on opensuse 11.4)
zypper ar https://build.pub.meego.com/project/monitor?project=Project:MINT:RC
zypper ref
zypper in boss

This will install Rabbit MQ and the boss worker daemon.

To run boss:

rcboss start / stop

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss

This will install Rabbit MQ and the boss worker daemon.

Then, as usual
/etc/init.d/boss {start|stop|restart|force-reload|log}

Default settings are found in /etc/default/boss

=== Configuration ===

BOSS is preconfigured with username boss and password boss on the boss vhost
(AMQP will lose vhosts soon)

BOSS stores workitem and process data in /var/spool/boss/

=== SkyNET installation for OpenSuse 11.4 ===

In order to be able to work with BOSS, please install SkyNET now:

zypper in boss-skynet

from the repository

https://build.pub.meego.com/project/monitor?project=Project:MINT:RC

== BOSS-Viewer ==

This must be installed on the boss server when using the FS storage.

The default port is 9292, point your browser at http://127.0.0.1:9292/_ruote/ (if you are running on the same machine)

#FIXME: Make sure you go to /_ruote/ otherwise you will get a big scary error.

=== SUSE ===

zypper ar https://build.pub.meego.com/project/monitor?project=Project:MINT:RC
zypper ref
zypper in boss-viewer

This will install boss-viewer.

To run boss-viewer:

rcboss-viewer start / stop

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss-viewer

This will install the boss viewer.

Then, as usual
/etc/init.d/boss {start|stop|restart|force-reload|log}

== BOSS OBS Plugin ==
The OBS plugin is designed to launch processes when a build/publish event occurs on the OBS

=== Installation ===
The plugin is installed on the OBS backend which runs the schedulers

openSUSE 11.2
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.2/Project:MINT:RC.repo
zypper ref
zypper in boss-obs-plugin

opensuse 11.4
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.4/devel:languages:perl.repo
zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss-obs-plugin

=== Configuration ===

'''You *MUST* have a participant registered as "obs_event" (provided by the robogrator launcher package https://meego.gitorious.org/meego-infrastructure-tools/boss-launcher-robogrator ) running before enabling this plugin or ruote will silently fill up with hundreds/thousands/millions of stalled processes.'''

Edit the file:
/usr/lib/obs/server/BSConfig.pm
and define/add:
our $notification_plugin = "notify_boss";
our $BOSS_host="boss"; # hostname for server running BOSS AMQP
our $BOSS_user="boss"; # AMQP username
our $BOSS_passwd="boss"; # AMQP password (cleartext)

You may also uncomment and set to 1:
our $multiaction_notify_support = 1;
(This allows osc requests containing multiple actions to be handled properly).

The plugin itself is installed in :
/usr/lib/obs/server/plugins/notify_boss.pm

The schedulers will need to be restarted to take effect. Note there is currently an issue that
the plugin will block if the AMQP server is not available - this will halt the OBS.

= Getting started and Testing the Deployment =

Once you have installed BOSS and SkyNET you are ready to create and run some processes.

== Watch BOSS ==
On the BOSS machine:

/etc/init.d/boss log &

You'll see messages about the engine being alive and participant
registration. Further debug output is possible here.

== Setup a participant ==

On the skynet machine we need to setup some code as a participant:

skynet make_participant -n check -p /usr/share/doc/python-boss-skynet/example-check-participant
skynet make_participant -n notify -p /usr/share/doc/python-boss-skynet/example-notify-participant

This creates a daemontools/participant directory structure with a
symbolic link to the code for these participants.

Now we need to ensure they start to run under daemontools (and restart if
there's a problem):

skynet enable check
skynet enable notify

To watch the log output:

skynet log check &
skynet log notify &

Now we need to tell boss that there is something listening on these
queues and available for use in processes:

skynet register -n check
skynet register -n notify

The skynet command also supports:
* stop : shutdown the participant and don't respawn.
* reload : stop and start once
* register : can also specify the amqp queue to use if it's not the name

== Launch a process ==

With a participant ready to do some work, we're ready to launch a process:

/usr/share/boss-skynet/skynet_launch

The log output should show

2011-03-29 00:49:21.186558500 checking for /tmp/success
2011-03-29 00:49:21.186849500 Failed to read /tmp/success
2011-03-29 00:49:21.708706500 Nothing to say

Now do:

echo Hi there > /tmp/success

And relaunch:

/usr/share/boss-skynet/skynet_launch

gives
2011-03-29 00:49:31.659986500 checking for /tmp/success
2011-03-29 00:49:31.660440500 Read /tmp/success
2011-03-29 00:49:31.878978500 Send email saying : Hi there
2011-03-29 00:49:31.878980500

Of course a process can be launched from anywhere on the network using
any 'trigger' and participants also run on any machine.

== Debugging ==

= Next steps =

Do something useful with your installation using the [[../Standard workflow|standard workflow]] and the [[../Participants|participants and launchers]]

Release Infrastructure/BOSS/Standard workflow

2011-06-16T13:55:04Z

Iamer: /* Robogrator Process Launcer */

BOSS comes with a sample "standard workflow" which consists of:
* a plugin to the OBS to intercept the [[Release Infrastructure/BOSS/OBS Event List|event notifications]]
* the robogrator process launcher
* a [[Release_Infrastructure/BOSS/Participants|coherent set of participants]]

This is not a mandatory workflow but was developed at Nokia and is designed to be useful for MeeGo vendors

=== Installation and source ===

Package : https://build.pub.meego.com/package/show?package=boss-launcher-robogrator&project=Project:MINT:Testing

GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-launcher-robogrator
NOTE: skynet branch for now

=== Robogrator Process Launcer ===

Robogrator is the robotic package integrator. It reacts to the events occuring on the OBS and looks at the data inside to identify which project-specific process to launch.

NOTE: robogrator is just a name , the real name of the participant is obs_event so register the skynet participant by invoking the command :

skynet register -n obs_event

It uses the data sent by the obs-boss-plugin to create a fresh process.

All processes have the fields:
* project : the project used to drive the process. Usually the project the package is in but for requests type events this is the target project.
* ev : the (undocumented) OBS Notify object parameters

For the actual process, robogrator uses the namespace/project/event-label to find a specific ruote process.

For the standard workflow the first participant is the standard_workflow_handler

It is configured in /etc/skynet/robogrator.conf

=== standard_workflow_handler ===

This participant serves two purposes:
* tracks packages that are rebuilt when a repo is published
* set some standard fields used by other participants

* REPO_PUBLISHED
** repository
** packages (packages successfully built in this build/publish cycle)
* BUILD_UNCHANGED, BUILD_SUCCESS, BUILD_FAIL
** package
* SRCSRV_CREATE_PACKAGE, SRCSRV_DELETE_PACKAGE
** package
* SRCSRV_VERSION_CHANGE
** package
** rev
** version
* SRCSRV_REQUEST_*
** actions
** who
** rid
** description
** when

Release Infrastructure/BOSS/Participants

2011-05-16T10:53:52Z

Iamer: /* Bugzilla */

List of participants ported / implemented for BOSS SkyNET :

== OBS prechecks ==

GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-prechecks

Package : https://build.pub.meego.com/package/show?package=boss-participant-prechecks&project=Project:MINT:Testing

Configuration : The file at /etc/oscrc needs to be edited so that these participants can communicate with the OBS instance.

=== check_already_testing ===
Compares the checksum of the packages being submitted to packages of the same
name possibly in the Testing project. If the checksum matches it sets STATUS
= FAILED

This is for over-eager developers - or if two people in a team submit
close together.

=== check_no_changes ===
Compares the checksum of the packages being submitted to packages of the same
name possibly in the Target project. If the checksum matches it sets STATUS =
FAILED

Developers sometimes submit without doing a proper update - this
catches that situation.

=== check_multiple_destinations ===
Checks if the request tries to submit packages to multiple projects at the same
time and sets STATUS = FAILED if so

=== check_package_is_complete ===
Checks if each of the packages being submitted contains at least the following
files :
* Source tarball : *.tar.gz *.tar.bz2 or *.tgz
* Changes file : *.changes
* Spec file : *.spec
and sets STATUS = FAILED if not

Nb the presence of the .changes file is important for the changelog
participants to operate correctly

=== check_package_built_at_source ===

Prerequisite : check_has_valid_repo

Checks if the packages being submitted are built successfully against the
designated target repository for the architectures of interest and sets STATUS = FAILED if not

=== check_spec ===
Checks if the spec file of each of the packages being submitted is valid. Currently
the only validity check applied is that it shouldn't contain the %changelog tag
and sets STATUS = FAILED if it does

OBS is responsible for inserting the .changes file contents into the spec file.

=== get_submitter_email ===
Gets the request submitter email from OBS and makes sure it is not an empty string
sets STATUS = FAILED otherwise.

=== check_submitter_is_maintainer ===
Checks the request submitter is actually a maintainer in the source project from
which the request is origination, sets STATUS = FAILED otherwise.

=== check_has_valid_repo ===
Finds a repository in the source project that builds ONLY against a certain target
project / repo , sets STATUS = FAILED if it does not find one.

=== check_has_relevant_changelog ===

Prerequisite : get_relevant_changelog

Checks that each package in the request actions has a non empty relevant change entries field.

Use this if you want to enforce that new entries be added to the .changes file before allowing a request to be accepted.

=== check_is_from_devel :regexp ===

Checks that each package in the request originates from a project the matches the regexp provided as a parameter.

== Resolving Requests ==

Package : [https://build.pub.meego.com/package/show?package=boss-participant-resolverequest&project=Project:MINT boss-participant-resolverequest]

=== change_request_state ===

Used to accept or decline a submit request.

Parameters:
* :action (accept/reject) : Determines what to do with the request

Fields:
* 'msg' : The contents of the 'msg' array in the workitem is used as the action message

Config:
* See [[#OBS / BuildService Participants]]

=== build_trial ===

=== revert_trial ===

== Changelog ==

GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-getchangelog

* branch skynet

=== get_changelog ===

Gets the .changes file from the project/package and puts it into the
'changelog' field. Note that for SRCSRV_REQUEST_*, project is the
_target_ and may not return the expected log.
Thus get_relevant_changelog may be more relevant for that case.

Dependencies:
* /etc/skynet/

=== get_relevant_changelog ===

Gets the .changes file for the source project/package/revision and
does a diff against the target project/package putting the 'added'
lines into a 'relevant_changelog' field for each package in the
request's actions array.

This is particularly useful for acting on external links such as bug#
or feature# mentioned in the new changelog lines.

The complexity of a diff provides for situations where multiple
changelog entries are made in one area of a project before the package
is finally accepted.

== Bugzilla ==

Interacts with bugzilla to:
* add comments to bugs
* change bug status

Links :
* GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-bugzilla
* Package : https://build.pub.meego.com/package/show?package=boss-participant-bugzilla&project=Project:MINT:Testing
* Configuration : The file at /etc/skynet/bugzilla.conf needs to be edited so that the participant can communicate with the Bugzilla instance. The file has lots of comments to explain the settings.

Prerequisites :
* Bugzilla with REST api installed (https://wiki.mozilla.org/Bugzilla:REST_API)
* get_relevant_changelog

Takes the following parameters (can be different depending on the target bugzilla):
* status : UNCONFIRMED/NEW/ASSIGNED/REOPENED/RESOLVED/VERIFIED/CLOSED
* resolution : FIXED/INVALID/WONTFIX/DUPLICATE/WORKSFORME/MOVED
For the comment :
* comment : Comment text for the bug
or
* template : Cheetah template file passed the workitem hash

== Documentation ==

== Image Creation ==

== Notification ==

GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-notify

NOTE: skynet branch for now

Package : https://build.pub.meego.com/package/show?package=boss-participant-notify&project=Project:MINT:Testing

=== notify ===
Email notification based on templates.

=== notify_irc :msg, :irc_channel ===
#TODO IRC notification to an IRC bot.

== OBS / BuildService Participants ==

Several participants interact with the OBS using the buildservice library.

They use the skynet configuration system to look for an 'oscrc' value
in the [obs] section.
eg
<code>
[obs]
oscrc = /etc/skynet/default.oscrc
</code>

This is a normal oscrc file (mode 600) and should contain [apiurl] sections with
an aliases section that includes the namespace value set by the OBS in
the obsEvent (see BSConfig.pm)
eg:
<code>
[https://api.pub.meego.com]
aliases=OBS
user=<user>
passx=<passx>
</code>

Release Infrastructure/BOSS/Participants

2011-05-16T10:52:47Z

Iamer: /* Bugzilla */

List of participants ported / implemented for BOSS SkyNET :

== OBS prechecks ==

GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-prechecks

Package : https://build.pub.meego.com/package/show?package=boss-participant-prechecks&project=Project:MINT:Testing

Configuration : The file at /etc/oscrc needs to be edited so that these participants can communicate with the OBS instance.

=== check_already_testing ===
Compares the checksum of the packages being submitted to packages of the same
name possibly in the Testing project. If the checksum matches it sets STATUS
= FAILED

This is for over-eager developers - or if two people in a team submit
close together.

=== check_no_changes ===
Compares the checksum of the packages being submitted to packages of the same
name possibly in the Target project. If the checksum matches it sets STATUS =
FAILED

Developers sometimes submit without doing a proper update - this
catches that situation.

=== check_multiple_destinations ===
Checks if the request tries to submit packages to multiple projects at the same
time and sets STATUS = FAILED if so

=== check_package_is_complete ===
Checks if each of the packages being submitted contains at least the following
files :
* Source tarball : *.tar.gz *.tar.bz2 or *.tgz
* Changes file : *.changes
* Spec file : *.spec
and sets STATUS = FAILED if not

Nb the presence of the .changes file is important for the changelog
participants to operate correctly

=== check_package_built_at_source ===

Prerequisite : check_has_valid_repo

Checks if the packages being submitted are built successfully against the
designated target repository for the architectures of interest and sets STATUS = FAILED if not

=== check_spec ===
Checks if the spec file of each of the packages being submitted is valid. Currently
the only validity check applied is that it shouldn't contain the %changelog tag
and sets STATUS = FAILED if it does

OBS is responsible for inserting the .changes file contents into the spec file.

=== get_submitter_email ===
Gets the request submitter email from OBS and makes sure it is not an empty string
sets STATUS = FAILED otherwise.

=== check_submitter_is_maintainer ===
Checks the request submitter is actually a maintainer in the source project from
which the request is origination, sets STATUS = FAILED otherwise.

=== check_has_valid_repo ===
Finds a repository in the source project that builds ONLY against a certain target
project / repo , sets STATUS = FAILED if it does not find one.

=== check_has_relevant_changelog ===

Prerequisite : get_relevant_changelog

Checks that each package in the request actions has a non empty relevant change entries field.

Use this if you want to enforce that new entries be added to the .changes file before allowing a request to be accepted.

=== check_is_from_devel :regexp ===

Checks that each package in the request originates from a project the matches the regexp provided as a parameter.

== Resolving Requests ==

Package : [https://build.pub.meego.com/package/show?package=boss-participant-resolverequest&project=Project:MINT boss-participant-resolverequest]

=== change_request_state ===

Used to accept or decline a submit request.

Parameters:
* :action (accept/reject) : Determines what to do with the request

Fields:
* 'msg' : The contents of the 'msg' array in the workitem is used as the action message

Config:
* See [[#OBS / BuildService Participants]]

=== build_trial ===

=== revert_trial ===

== Changelog ==

GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-getchangelog

* branch skynet

=== get_changelog ===

Gets the .changes file from the project/package and puts it into the
'changelog' field. Note that for SRCSRV_REQUEST_*, project is the
_target_ and may not return the expected log.
Thus get_relevant_changelog may be more relevant for that case.

Dependencies:
* /etc/skynet/

=== get_relevant_changelog ===

Gets the .changes file for the source project/package/revision and
does a diff against the target project/package putting the 'added'
lines into a 'relevant_changelog' field for each package in the
request's actions array.

This is particularly useful for acting on external links such as bug#
or feature# mentioned in the new changelog lines.

The complexity of a diff provides for situations where multiple
changelog entries are made in one area of a project before the package
is finally accepted.

== Bugzilla ==

Interacts with bugzilla to:
* add comments to bugs
* change bug status

Links :
* GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-bugzilla
* Package : https://build.pub.meego.com/package/show?package=boss-participant-bugzilla&project=Project:MINT:Testing
* Configuration : The file at /etc/skynet/bugzilla.conf needs to be edited so that the participant can communicate with the Bugzilla instance. The file has lots of comments to explain the settings.

Prerequisites :
* Bugzilla with REST api installed (https://wiki.mozilla.org/Bugzilla:REST_API)
* get_relevant_changelog

Takes the following parameters (can be different depending on the target bugzilla):
* status UNCONFIRMED/NEW/ASSIGNED/REOPENED/RESOLVED/VERIFIED/CLOSED
* resolution FIXED/INVALID/WONTFIX/DUPLICATE/WORKSFORME/MOVED
For the comment :
* comment Comment text for the bug
or
* template Cheetah template file passed the workitem hash

== Documentation ==

== Image Creation ==

== Notification ==

GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-notify

NOTE: skynet branch for now

Package : https://build.pub.meego.com/package/show?package=boss-participant-notify&project=Project:MINT:Testing

=== notify ===
Email notification based on templates.

=== notify_irc :msg, :irc_channel ===
#TODO IRC notification to an IRC bot.

== OBS / BuildService Participants ==

Several participants interact with the OBS using the buildservice library.

They use the skynet configuration system to look for an 'oscrc' value
in the [obs] section.
eg
<code>
[obs]
oscrc = /etc/skynet/default.oscrc
</code>

This is a normal oscrc file (mode 600) and should contain [apiurl] sections with
an aliases section that includes the namespace value set by the OBS in
the obsEvent (see BSConfig.pm)
eg:
<code>
[https://api.pub.meego.com]
aliases=OBS
user=<user>
passx=<passx>
</code>

Release Infrastructure/BOSS/Participants

2011-05-16T10:51:46Z

Iamer: /* Bugzilla */

List of participants ported / implemented for BOSS SkyNET :

== OBS prechecks ==

GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-prechecks

Package : https://build.pub.meego.com/package/show?package=boss-participant-prechecks&project=Project:MINT:Testing

Configuration : The file at /etc/oscrc needs to be edited so that these participants can communicate with the OBS instance.

=== check_already_testing ===
Compares the checksum of the packages being submitted to packages of the same
name possibly in the Testing project. If the checksum matches it sets STATUS
= FAILED

This is for over-eager developers - or if two people in a team submit
close together.

=== check_no_changes ===
Compares the checksum of the packages being submitted to packages of the same
name possibly in the Target project. If the checksum matches it sets STATUS =
FAILED

Developers sometimes submit without doing a proper update - this
catches that situation.

=== check_multiple_destinations ===
Checks if the request tries to submit packages to multiple projects at the same
time and sets STATUS = FAILED if so

=== check_package_is_complete ===
Checks if each of the packages being submitted contains at least the following
files :
* Source tarball : *.tar.gz *.tar.bz2 or *.tgz
* Changes file : *.changes
* Spec file : *.spec
and sets STATUS = FAILED if not

Nb the presence of the .changes file is important for the changelog
participants to operate correctly

=== check_package_built_at_source ===

Prerequisite : check_has_valid_repo

Checks if the packages being submitted are built successfully against the
designated target repository for the architectures of interest and sets STATUS = FAILED if not

=== check_spec ===
Checks if the spec file of each of the packages being submitted is valid. Currently
the only validity check applied is that it shouldn't contain the %changelog tag
and sets STATUS = FAILED if it does

OBS is responsible for inserting the .changes file contents into the spec file.

=== get_submitter_email ===
Gets the request submitter email from OBS and makes sure it is not an empty string
sets STATUS = FAILED otherwise.

=== check_submitter_is_maintainer ===
Checks the request submitter is actually a maintainer in the source project from
which the request is origination, sets STATUS = FAILED otherwise.

=== check_has_valid_repo ===
Finds a repository in the source project that builds ONLY against a certain target
project / repo , sets STATUS = FAILED if it does not find one.

=== check_has_relevant_changelog ===

Prerequisite : get_relevant_changelog

Checks that each package in the request actions has a non empty relevant change entries field.

Use this if you want to enforce that new entries be added to the .changes file before allowing a request to be accepted.

=== check_is_from_devel :regexp ===

Checks that each package in the request originates from a project the matches the regexp provided as a parameter.

== Resolving Requests ==

Package : [https://build.pub.meego.com/package/show?package=boss-participant-resolverequest&project=Project:MINT boss-participant-resolverequest]

=== change_request_state ===

Used to accept or decline a submit request.

Parameters:
* :action (accept/reject) : Determines what to do with the request

Fields:
* 'msg' : The contents of the 'msg' array in the workitem is used as the action message

Config:
* See [[#OBS / BuildService Participants]]

=== build_trial ===

=== revert_trial ===

== Changelog ==

GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-getchangelog

* branch skynet

=== get_changelog ===

Gets the .changes file from the project/package and puts it into the
'changelog' field. Note that for SRCSRV_REQUEST_*, project is the
_target_ and may not return the expected log.
Thus get_relevant_changelog may be more relevant for that case.

Dependencies:
* /etc/skynet/

=== get_relevant_changelog ===

Gets the .changes file for the source project/package/revision and
does a diff against the target project/package putting the 'added'
lines into a 'relevant_changelog' field for each package in the
request's actions array.

This is particularly useful for acting on external links such as bug#
or feature# mentioned in the new changelog lines.

The complexity of a diff provides for situations where multiple
changelog entries are made in one area of a project before the package
is finally accepted.

== Bugzilla ==

Interacts with bugzilla to:
* add comments to bugs
* change bug status

* GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-bugzilla

* Package : https://build.pub.meego.com/package/show?package=boss-participant-bugzilla&project=Project:MINT:Testing

* Configuration : The file at /etc/skynet/bugzilla.conf needs to be edited so that the participant can communicate with the Bugzilla instance. The file has lots of comments to explain the settings.

Prerequisites :
* Bugzilla with REST api installed (https://wiki.mozilla.org/Bugzilla:REST_API)
* get_relevant_changelog

Takes the following parameters (can be different depending on the target bugzilla):
* status UNCONFIRMED/NEW/ASSIGNED/REOPENED/RESOLVED/VERIFIED/CLOSED
* resolution FIXED/INVALID/WONTFIX/DUPLICATE/WORKSFORME/MOVED

For the comment :

* comment Comment text for the bug
or
* template Cheetah template file passed the workitem hash

== Documentation ==

== Image Creation ==

== Notification ==

GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-notify

NOTE: skynet branch for now

Package : https://build.pub.meego.com/package/show?package=boss-participant-notify&project=Project:MINT:Testing

=== notify ===
Email notification based on templates.

=== notify_irc :msg, :irc_channel ===
#TODO IRC notification to an IRC bot.

== OBS / BuildService Participants ==

Several participants interact with the OBS using the buildservice library.

They use the skynet configuration system to look for an 'oscrc' value
in the [obs] section.
eg
<code>
[obs]
oscrc = /etc/skynet/default.oscrc
</code>

This is a normal oscrc file (mode 600) and should contain [apiurl] sections with
an aliases section that includes the namespace value set by the OBS in
the obsEvent (see BSConfig.pm)
eg:
<code>
[https://api.pub.meego.com]
aliases=OBS
user=<user>
passx=<passx>
</code>

Release Infrastructure/BOSS/Participants

2011-05-16T10:50:44Z

Iamer: Document bugzilla participant

List of participants ported / implemented for BOSS SkyNET :

== OBS prechecks ==

GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-prechecks

Package : https://build.pub.meego.com/package/show?package=boss-participant-prechecks&project=Project:MINT:Testing

Configuration : The file at /etc/oscrc needs to be edited so that these participants can communicate with the OBS instance.

=== check_already_testing ===
Compares the checksum of the packages being submitted to packages of the same
name possibly in the Testing project. If the checksum matches it sets STATUS
= FAILED

This is for over-eager developers - or if two people in a team submit
close together.

=== check_no_changes ===
Compares the checksum of the packages being submitted to packages of the same
name possibly in the Target project. If the checksum matches it sets STATUS =
FAILED

Developers sometimes submit without doing a proper update - this
catches that situation.

=== check_multiple_destinations ===
Checks if the request tries to submit packages to multiple projects at the same
time and sets STATUS = FAILED if so

=== check_package_is_complete ===
Checks if each of the packages being submitted contains at least the following
files :
* Source tarball : *.tar.gz *.tar.bz2 or *.tgz
* Changes file : *.changes
* Spec file : *.spec
and sets STATUS = FAILED if not

Nb the presence of the .changes file is important for the changelog
participants to operate correctly

=== check_package_built_at_source ===

Prerequisite : check_has_valid_repo

Checks if the packages being submitted are built successfully against the
designated target repository for the architectures of interest and sets STATUS = FAILED if not

=== check_spec ===
Checks if the spec file of each of the packages being submitted is valid. Currently
the only validity check applied is that it shouldn't contain the %changelog tag
and sets STATUS = FAILED if it does

OBS is responsible for inserting the .changes file contents into the spec file.

=== get_submitter_email ===
Gets the request submitter email from OBS and makes sure it is not an empty string
sets STATUS = FAILED otherwise.

=== check_submitter_is_maintainer ===
Checks the request submitter is actually a maintainer in the source project from
which the request is origination, sets STATUS = FAILED otherwise.

=== check_has_valid_repo ===
Finds a repository in the source project that builds ONLY against a certain target
project / repo , sets STATUS = FAILED if it does not find one.

=== check_has_relevant_changelog ===

Prerequisite : get_relevant_changelog

Checks that each package in the request actions has a non empty relevant change entries field.

Use this if you want to enforce that new entries be added to the .changes file before allowing a request to be accepted.

=== check_is_from_devel :regexp ===

Checks that each package in the request originates from a project the matches the regexp provided as a parameter.

== Resolving Requests ==

Package : [https://build.pub.meego.com/package/show?package=boss-participant-resolverequest&project=Project:MINT boss-participant-resolverequest]

=== change_request_state ===

Used to accept or decline a submit request.

Parameters:
* :action (accept/reject) : Determines what to do with the request

Fields:
* 'msg' : The contents of the 'msg' array in the workitem is used as the action message

Config:
* See [[#OBS / BuildService Participants]]

=== build_trial ===

=== revert_trial ===

== Changelog ==

GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-getchangelog

* branch skynet

=== get_changelog ===

Gets the .changes file from the project/package and puts it into the
'changelog' field. Note that for SRCSRV_REQUEST_*, project is the
_target_ and may not return the expected log.
Thus get_relevant_changelog may be more relevant for that case.

Dependencies:
* /etc/skynet/

=== get_relevant_changelog ===

Gets the .changes file for the source project/package/revision and
does a diff against the target project/package putting the 'added'
lines into a 'relevant_changelog' field for each package in the
request's actions array.

This is particularly useful for acting on external links such as bug#
or feature# mentioned in the new changelog lines.

The complexity of a diff provides for situations where multiple
changelog entries are made in one area of a project before the package
is finally accepted.

== Bugzilla ==
GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-bugzilla

Package : https://build.pub.meego.com/package/show?package=boss-participant-bugzilla&project=Project:MINT:Testing

Configuration : The file at /etc/skynet/bugzilla.conf needs to be edited so that the participant can communicate with the Bugzilla instance. The file has lots of comments to explain the settings.

Prerequisites :
* Bugzilla with REST api installed (https://wiki.mozilla.org/Bugzilla:REST_API)
* get_relevant_changelog

Interacts with bugzilla to:
* add comments to bugs
* change bug status

Takes the following parameters (can be different depending on the target bugzilla):
* status UNCONFIRMED/NEW/ASSIGNED/REOPENED/RESOLVED/VERIFIED/CLOSED
* resolution FIXED/INVALID/WONTFIX/DUPLICATE/WORKSFORME/MOVED
* comment Comment text for the bug
or
* template Cheetah template file passed the workitem hash

== Documentation ==

== Image Creation ==

== Notification ==

GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-notify

NOTE: skynet branch for now

Package : https://build.pub.meego.com/package/show?package=boss-participant-notify&project=Project:MINT:Testing

=== notify ===
Email notification based on templates.

=== notify_irc :msg, :irc_channel ===
#TODO IRC notification to an IRC bot.

== OBS / BuildService Participants ==

Several participants interact with the OBS using the buildservice library.

They use the skynet configuration system to look for an 'oscrc' value
in the [obs] section.
eg
<code>
[obs]
oscrc = /etc/skynet/default.oscrc
</code>

This is a normal oscrc file (mode 600) and should contain [apiurl] sections with
an aliases section that includes the namespace value set by the OBS in
the obsEvent (see BSConfig.pm)
eg:
<code>
[https://api.pub.meego.com]
aliases=OBS
user=<user>
passx=<passx>
</code>

ARM/N900/DeveloperEdition/BOSS

2011-05-11T21:02:47Z

Iamer: /* BOSS Checks */

= Automating DE processes =

The DE workflow is (roughly) :

# Developer branches package from P:DE:Trunk
# Developer submits package to P:DE:Trunk:Testing
# Boss runs checks on package
# Boss creates branch of P:DE:Trunk
# Boss submits package(s) to new branch

Later:
# Boss rejects package
or
# Boss notifies maintainers that checks passed

Then:
# Package is accepted to P:DE:Trunk by a maintainer
# bz is updated

== BOSS Checks ==

BOSS runs the following checks (and stops as soon as one fails)

* [http://wiki.meego.com/Release_Infrastructure/BOSS/Participants#check_submitter_is_maintainer check_submitter_is_maintainer]
* [http://wiki.meego.com/Release_Infrastructure/BOSS/Participants#check_package_built_at_source check_package_built_at_source]
* [http://wiki.meego.com/Release_Infrastructure/BOSS/Participants#check_package_is_complete check_package_is_complete]
* [http://wiki.meego.com/Release_Infrastructure/BOSS/Participants#check_spec check_spec]

To be done:
* changelog in proper format
* tarball not changed within same package version
* executing spectacle doesn't change anything if .yaml present
* content of source tar ball should not change without its changing.
* last changelog version matches spec file version

= Bug Lifecycle =

The DE bug lifecycle is (roughly):

* 'open' -> FIXED -> RELEASED -> VERIFIED

# Developer starts working on the bug/feature and sets to ASSIGNED
# Developer submits package to P:DE:Trunk:Testing and documents changelog and/or commit properly (fixes bmc #1234;) - if changelog doesn't have a valid bug/feature reference the request MUST fail but we do not document the failure in the bug report
# Bugzilla records the commit message with link to the repo and timestamp
# Package is accepted to P:DE:Trunk (manual for now....)
# Bugzilla records acceptance with comment and bug status is automatically changed to FIXED if bug is still in OPEN status (New, assigned, needinfo, reopened, waiting)
# If bug is already in FIXED resolution, only acceptance comment is written in the bug report
# If bug is not in OPEN status (New, assigned, needinfo, reopened, waiting) or resolved FIXED (eg. duplicate, wontfix, etc... or verified or closed) the request MUST fail but we do not document the failure in the bug report

Later:
# Release is triggered
# Bug status is automatically changed to RELEASED, a comment with a link to the image is created and the Target Build value is set to match appropriate release

Later still:
# QA review the release and for each bug marked RELEASED, either change to VERIFIED or re-open

Note that eventually, both FIXED and RELEASED statuses would require specific credentials to be modified.

ARM/N900/DeveloperEdition/BOSS

2011-05-11T20:55:37Z

Iamer: /* BOSS Checks */

= Automating DE processes =

The DE workflow is (roughly) :

# Developer branches package from P:DE:Trunk
# Developer submits package to P:DE:Trunk:Testing
# Boss runs checks on package
# Boss creates branch of P:DE:Trunk
# Boss submits package(s) to new branch

Later:
# Boss rejects package
or
# Boss notifies maintainers that checks passed

Then:
# Package is accepted to P:DE:Trunk by a maintainer
# bz is updated

== BOSS Checks ==

BOSS runs the following checks (and stops as soon as one fails)

* [http://wiki.meego.com/Release_Infrastructure/BOSS/Participants#check_if_submitter_is_maintainer check_if_submitter_is_maintainer]
* [http://wiki.meego.com/Release_Infrastructure/BOSS/Participants#check_package_built_at_source check_package_built_at_source]
* [http://wiki.meego.com/Release_Infrastructure/BOSS/Participants#check_package_is_complete check_package_is_complete]
* [http://wiki.meego.com/Release_Infrastructure/BOSS/Participants#check_spec check_spec]

To be done:
* changelog in proper format
* tarball not changed within same package version
* executing spectacle doesn't change anything if .yaml present
* content of source tar ball should not change without its changing.
* last changelog version matches spec file version

= Bug Lifecycle =

The DE bug lifecycle is (roughly):

* 'open' -> FIXED -> RELEASED -> VERIFIED

# Developer starts working on the bug/feature and sets to ASSIGNED
# Developer submits package to P:DE:Trunk:Testing and documents changelog and/or commit properly (fixes bmc #1234;) - if changelog doesn't have a valid bug/feature reference the request MUST fail but we do not document the failure in the bug report
# Bugzilla records the commit message with link to the repo and timestamp
# Package is accepted to P:DE:Trunk (manual for now....)
# Bugzilla records acceptance with comment and bug status is automatically changed to FIXED if bug is still in OPEN status (New, assigned, needinfo, reopened, waiting)
# If bug is already in FIXED resolution, only acceptance comment is written in the bug report
# If bug is not in OPEN status (New, assigned, needinfo, reopened, waiting) or resolved FIXED (eg. duplicate, wontfix, etc... or verified or closed) the request MUST fail but we do not document the failure in the bug report

Later:
# Release is triggered
# Bug status is automatically changed to RELEASED, a comment with a link to the image is created and the Target Build value is set to match appropriate release

Later still:
# QA review the release and for each bug marked RELEASED, either change to VERIFIED or re-open

Note that eventually, both FIXED and RELEASED statuses would require specific credentials to be modified.

Release Infrastructure/BOSS/Participants

2011-05-01T21:07:49Z

Iamer: /* get_changelog */

Release Infrastructure/BOSS/Participants

2011-05-01T21:06:38Z

Iamer: /* get_relevant_changelog */

Release Infrastructure/BOSS/Participants

2011-04-29T10:32:04Z

Iamer:

Release Infrastructure/BOSS/Participants

2011-04-27T06:58:49Z

Iamer:

Release Infrastructure/BOSS/Participants

2011-04-27T06:41:41Z

Iamer:

Release Infrastructure/BOSS/Participants

2011-04-26T14:32:14Z

Iamer: /* submitter_maintainer -> check_submitter_is_maintainer */

Release Infrastructure/BOSS/Participants

2011-04-26T14:32:03Z

Iamer: /* target_repo -> check_has_valid_repo */

Release Infrastructure/BOSS/Participants

2011-04-26T14:31:35Z

Iamer: /* package_successful -> check_package_built_at_source */

Release Infrastructure/BOSS/Participants

2011-04-26T14:30:59Z

Iamer: /* package_complete -> check_package_is_complete */

Release Infrastructure/BOSS/Participants

2011-04-26T14:30:40Z

Iamer:

Release Infrastructure/BOSS/Installation

2011-04-26T14:01:02Z

Iamer: /* Next steps */

= Overview =

Whilst BOSS is deployed in production, this packaged version is still
in beta-testing and is subject to change. In particular the
repositories are not yet finalised (eg installing from :RC).

However BOSS and SkyNET both install and run at a basic level on a
clean Debian 6.0 or OpenSuse 11.4 installation.

= Installation =

== Preparation ==

* Proxy : On OpenSUSE you need to set the proxy in /etc/sysconfig/proxy

== BOSS ==
=== Overview ===
BOSS provides the ruote worker and the main AMQP server.

This component is the main process dispatcher; no work happens
here but all process steps are initiated from here.

Currently BOSS uses the Ruote filesystem storage system and the
worker and viewer need to run on the same system. This will change
when BOSS moves to Redis.

The stable and testing projects are at the community OBS, the state
of the packages can be monitored at :

Testing : https://build.pub.meego.com/project/monitor?project=Project:MINT:Testing

Release Candidate : (No release yet) https://build.pub.meego.com/project/monitor?project=Project:MINT:RC

Stable : (No release yet) https://build.pub.meego.com/project/monitor?project=Project:MINT

=== SUSE ===

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo
zypper ref
zypper in boss

(new version of BOSS on opensuse 11.4)
zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss

This will install Rabbit MQ and the boss worker daemon.

To run boss:

rcboss start / stop

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss

This will install Rabbit MQ and the boss worker daemon.

Then, as usual
/etc/init.d/boss {start|stop|restart|force-reload|log}

Default settings are found in /etc/default/boss

=== Configuration ===

BOSS is preconfigured with username boss and password boss on the boss vhost
(AMQP will lose vhosts soon)

BOSS stores workitem and process data in /var/spool/boss/

== BOSS-Viewer ==

This must be installed on the boss server when using the FS storage.

The default port is 9292, point your browser at http://127.0.0.1:9292/_ruote/ (if you are running on the same machine)

#FIXME: Make sure you go to /_ruote/ otherwise you will get a big scary error.

=== SUSE ===

zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss-viewer

This will install boss-viewer.

To run boss-viewer:

rcboss-viewer start / stop

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss-viewer

This will install the boss viewer.

Then, as usual
/etc/init.d/boss {start|stop|restart|force-reload|log}

== SkyNET ==
=== Overview ===

SkyNET manages long-running BOSS Participants.

It provides:

* daemonisation
* start/stop control
* consistent logging
* user-owned participants

=== SUSE ===

zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss-skynet

If you installed a minimal OpenSUSE you might get a dependency error and as shown here proceed with choice 1 :

Problem: boss-skynet-0.1-5.1.noarch requires /usr/bin/python, but this requirement cannot be provided
uninstallable providers: python-base-2.7-8.4.i586[openSUSE-11.4-11.4-0]
python-base-2.7-8.4.x86_64[openSUSE-11.4-11.4-0]
Solution 1: deinstallation of patterns-openSUSE-minimal_base-11.4-6.9.1.x86_64
Solution 2: do not install boss-skynet-0.1-5.1.noarch
Solution 3: break boss-skynet by ignoring some of its dependencies
Choose from above solutions by number or cancel [1/2/3/c] (c): 1
.
.
.

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
EOF

apt-get install --no-install-recommends boss-skynet

=== Configuration ===
The file /etc/skynet/boss.conf is used by all participants

It specifies the boss instance used by default and the credentials
required. Individual participants can override this.

The /etc/skynet/skynet.conf file specifies where SkyNET stores the
running participants and where daemontools monitors them
(/var/lib/SkyNET/services/ and /var/lib/SkyNET/store/ by default)

== BOSS OBS Plugin ==
The OBS plugin is designed to launch processes when a build/publish event occurs on the OBS

=== Installation ===
The plugin is installed on the OBS backend which runs the schedulers

openSUSE 11.2
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.2/Project:MINT:RC.repo
zypper ref
zypper in boss-obs-plugin

opensuse 11.4
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss-obs-plugin

=== Configuration ===

'''You *MUST* have a participant registered as "obs_event" running before enabling this plugin or ruote will silently fill up with hundreds/thousands/millions of stalled processes.'''

Edit the file:
/usr/lib/obs/server/BSConfig.pm
and define/add:
our $notification_plugin = "notify_boss";
our $BOSS_host="boss"; # hostname for server running BOSS AMQP
our $BOSS_user="boss"; # AMQP username
our $BOSS_passwd="boss"; # AMQP password (cleartext)

The plugin itself is installed in :
/usr/lib/obs/server/plugins/notify_boss.pm

The schedulers will need to be restarted to take effect. Note there is currently an issue that
the plugin will block if the AMQP server is not available - this will halt the OBS.

= Getting started =

Once you have installed BOSS and SkyNET you are ready to create and run some processes.

== Watch BOSS ==
On the BOSS machine:

/etc/init.d/boss log &

You'll see messages about the engine being alive and participant
registration. Further debug output is possible here.

== Setup a participant ==

On the skynet machine we need to setup some code as a participant:

skynet make_participant -n check -p /usr/share/doc/python-boss-skynet/example-check-participant
skynet make_participant -n notify -p /usr/share/doc/python-boss-skynet/example-notify-participant

This creates a daemontools/participant directory structure with a
symbolic link to the code for these participants.

Now we need to ensure they start to run under daemontools (and restart if
there's a problem):

skynet enable check
skynet enable notify

To watch the log output:

skynet log check &
skynet log notify &

Now we need to tell boss that there is something listening on these
queues and available for use in processes:

skynet register -n check
skynet register -n notify

The skynet command also supports:
* stop : shutdown the participant and don't respawn.
* reload : stop and start once
* register : can also specify the amqp queue to use if it's not the name

== Launch a process ==

With a participant ready to do some work, we're ready to launch a process:

/usr/share/boss-skynet/skynet_launch

The log output should show

2011-03-29 00:49:21.186558500 checking for /tmp/success
2011-03-29 00:49:21.186849500 Failed to read /tmp/success
2011-03-29 00:49:21.708706500 Nothing to say

Now do:

echo Hi there > /tmp/success

And relaunch:

/usr/share/boss-skynet/skynet_launch

gives
2011-03-29 00:49:31.659986500 checking for /tmp/success
2011-03-29 00:49:31.660440500 Read /tmp/success
2011-03-29 00:49:31.878978500 Send email saying : Hi there
2011-03-29 00:49:31.878980500

Of course a process can be launched from anywhere on the network using
any 'trigger' and participants also run on any machine.

== Debugging ==

= Next steps =

Do something useful with your installation using already available participants and launchers : http://wiki.meego.com/Release_Infrastructure/BOSS/Participants

Release Infrastructure/BOSS/Installation

2011-04-26T13:57:52Z

Iamer:

= Overview =

Whilst BOSS is deployed in production, this packaged version is still
in beta-testing and is subject to change. In particular the
repositories are not yet finalised (eg installing from :RC).

However BOSS and SkyNET both install and run at a basic level on a
clean Debian 6.0 or OpenSuse 11.4 installation.

= Installation =

== Preparation ==

* Proxy : On OpenSUSE you need to set the proxy in /etc/sysconfig/proxy

== BOSS ==
=== Overview ===
BOSS provides the ruote worker and the main AMQP server.

This component is the main process dispatcher; no work happens
here but all process steps are initiated from here.

Currently BOSS uses the Ruote filesystem storage system and the
worker and viewer need to run on the same system. This will change
when BOSS moves to Redis.

The stable and testing projects are at the community OBS, the state
of the packages can be monitored at :

Testing : https://build.pub.meego.com/project/monitor?project=Project:MINT:Testing

Release Candidate : (No release yet) https://build.pub.meego.com/project/monitor?project=Project:MINT:RC

Stable : (No release yet) https://build.pub.meego.com/project/monitor?project=Project:MINT

=== SUSE ===

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo
zypper ref
zypper in boss

(new version of BOSS on opensuse 11.4)
zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss

This will install Rabbit MQ and the boss worker daemon.

To run boss:

rcboss start / stop

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss

This will install Rabbit MQ and the boss worker daemon.

Then, as usual
/etc/init.d/boss {start|stop|restart|force-reload|log}

Default settings are found in /etc/default/boss

=== Configuration ===

BOSS is preconfigured with username boss and password boss on the boss vhost
(AMQP will lose vhosts soon)

BOSS stores workitem and process data in /var/spool/boss/

== BOSS-Viewer ==

This must be installed on the boss server when using the FS storage.

The default port is 9292, point your browser at http://127.0.0.1:9292/_ruote/ (if you are running on the same machine)

#FIXME: Make sure you go to /_ruote/ otherwise you will get a big scary error.

=== SUSE ===

zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss-viewer

This will install boss-viewer.

To run boss-viewer:

rcboss-viewer start / stop

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss-viewer

This will install the boss viewer.

Then, as usual
/etc/init.d/boss {start|stop|restart|force-reload|log}

== SkyNET ==
=== Overview ===

SkyNET manages long-running BOSS Participants.

It provides:

* daemonisation
* start/stop control
* consistent logging
* user-owned participants

=== SUSE ===

zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss-skynet

If you installed a minimal OpenSUSE you might get a dependency error and as shown here proceed with choice 1 :

Problem: boss-skynet-0.1-5.1.noarch requires /usr/bin/python, but this requirement cannot be provided
uninstallable providers: python-base-2.7-8.4.i586[openSUSE-11.4-11.4-0]
python-base-2.7-8.4.x86_64[openSUSE-11.4-11.4-0]
Solution 1: deinstallation of patterns-openSUSE-minimal_base-11.4-6.9.1.x86_64
Solution 2: do not install boss-skynet-0.1-5.1.noarch
Solution 3: break boss-skynet by ignoring some of its dependencies
Choose from above solutions by number or cancel [1/2/3/c] (c): 1
.
.
.

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
EOF

apt-get install --no-install-recommends boss-skynet

=== Configuration ===
The file /etc/skynet/boss.conf is used by all participants

It specifies the boss instance used by default and the credentials
required. Individual participants can override this.

The /etc/skynet/skynet.conf file specifies where SkyNET stores the
running participants and where daemontools monitors them
(/var/lib/SkyNET/services/ and /var/lib/SkyNET/store/ by default)

== BOSS OBS Plugin ==
The OBS plugin is designed to launch processes when a build/publish event occurs on the OBS

=== Installation ===
The plugin is installed on the OBS backend which runs the schedulers

openSUSE 11.2
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.2/Project:MINT:RC.repo
zypper ref
zypper in boss-obs-plugin

opensuse 11.4
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss-obs-plugin

=== Configuration ===

'''You *MUST* have a participant registered as "obs_event" running before enabling this plugin or ruote will silently fill up with hundreds/thousands/millions of stalled processes.'''

Edit the file:
/usr/lib/obs/server/BSConfig.pm
and define/add:
our $notification_plugin = "notify_boss";
our $BOSS_host="boss"; # hostname for server running BOSS AMQP
our $BOSS_user="boss"; # AMQP username
our $BOSS_passwd="boss"; # AMQP password (cleartext)

The plugin itself is installed in :
/usr/lib/obs/server/plugins/notify_boss.pm

The schedulers will need to be restarted to take effect. Note there is currently an issue that
the plugin will block if the AMQP server is not available - this will halt the OBS.

= Getting started =

Once you have installed BOSS and SkyNET you are ready to create and run some processes.

== Watch BOSS ==
On the BOSS machine:

/etc/init.d/boss log &

You'll see messages about the engine being alive and participant
registration. Further debug output is possible here.

== Setup a participant ==

On the skynet machine we need to setup some code as a participant:

skynet make_participant -n check -p /usr/share/doc/python-boss-skynet/example-check-participant
skynet make_participant -n notify -p /usr/share/doc/python-boss-skynet/example-notify-participant

This creates a daemontools/participant directory structure with a
symbolic link to the code for these participants.

Now we need to ensure they start to run under daemontools (and restart if
there's a problem):

skynet enable check
skynet enable notify

To watch the log output:

skynet log check &
skynet log notify &

Now we need to tell boss that there is something listening on these
queues and available for use in processes:

skynet register -n check
skynet register -n notify

The skynet command also supports:
* stop : shutdown the participant and don't respawn.
* reload : stop and start once
* register : can also specify the amqp queue to use if it's not the name

== Launch a process ==

With a participant ready to do some work, we're ready to launch a process:

/usr/share/boss-skynet/skynet_launch

The log output should show

2011-03-29 00:49:21.186558500 checking for /tmp/success
2011-03-29 00:49:21.186849500 Failed to read /tmp/success
2011-03-29 00:49:21.708706500 Nothing to say

Now do:

echo Hi there > /tmp/success

And relaunch:

/usr/share/boss-skynet/skynet_launch

gives
2011-03-29 00:49:31.659986500 checking for /tmp/success
2011-03-29 00:49:31.660440500 Read /tmp/success
2011-03-29 00:49:31.878978500 Send email saying : Hi there
2011-03-29 00:49:31.878980500

Of course a process can be launched from anywhere on the network using
any 'trigger' and participants also run on any machine.

== Debugging ==

= Next steps =

Do something useful with your installation using already available participants and launchers : [[/Release_Infrastructure/BOSS/Participants|Participants]]

Release Infrastructure/BOSS/Installation

2011-04-26T13:55:40Z

Iamer:

= Overview =

Whilst BOSS is deployed in production, this packaged version is still
in beta-testing and is subject to change. In particular the
repositories are not yet finalised (eg installing from :RC).

However BOSS and SkyNET both install and run at a basic level on a
clean Debian 6.0 or OpenSuse 11.4 installation.

= Installation =

== Preparation ==

* Proxy : On OpenSUSE you need to set the proxy in /etc/sysconfig/proxy

== BOSS ==
=== Overview ===
BOSS provides the ruote worker and the main AMQP server.

This component is the main process dispatcher; no work happens
here but all process steps are initiated from here.

Currently BOSS uses the Ruote filesystem storage system and the
worker and viewer need to run on the same system. This will change
when BOSS moves to Redis.

The stable and testing projects are at the community OBS, the state
of the packages can be monitored at :

Testing : https://build.pub.meego.com/project/monitor?project=Project:MINT:Testing

Release Candidate : (No release yet) https://build.pub.meego.com/project/monitor?project=Project:MINT:RC

Stable : (No release yet) https://build.pub.meego.com/project/monitor?project=Project:MINT

=== SUSE ===

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo
zypper ref
zypper in boss

(new version of BOSS on opensuse 11.4)
zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss

This will install Rabbit MQ and the boss worker daemon.

To run boss:

rcboss start / stop

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss

This will install Rabbit MQ and the boss worker daemon.

Then, as usual
/etc/init.d/boss {start|stop|restart|force-reload|log}

Default settings are found in /etc/default/boss

=== Configuration ===

BOSS is preconfigured with username boss and password boss on the boss vhost
(AMQP will lose vhosts soon)

BOSS stores workitem and process data in /var/spool/boss/

== BOSS-Viewer ==

This must be installed on the boss server when using the FS storage.

The default port is 9292, point your browser at http://127.0.0.1:9292/_ruote/ (if you are running on the same machine)

#FIXME: Make sure you go to /_ruote/ otherwise you will get a big scary error.

=== SUSE ===

zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss-viewer

This will install boss-viewer.

To run boss-viewer:

rcboss-viewer start / stop

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss-viewer

This will install the boss viewer.

Then, as usual
/etc/init.d/boss {start|stop|restart|force-reload|log}

== SkyNET ==
=== Overview ===

SkyNET manages long-running BOSS Participants.

It provides:

* daemonisation
* start/stop control
* consistent logging
* user-owned participants

=== SUSE ===

zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss-skynet

If you installed a minimal OpenSUSE you might get a dependency error and as shown here proceed with choice 1 :

Problem: boss-skynet-0.1-5.1.noarch requires /usr/bin/python, but this requirement cannot be provided
uninstallable providers: python-base-2.7-8.4.i586[openSUSE-11.4-11.4-0]
python-base-2.7-8.4.x86_64[openSUSE-11.4-11.4-0]
Solution 1: deinstallation of patterns-openSUSE-minimal_base-11.4-6.9.1.x86_64
Solution 2: do not install boss-skynet-0.1-5.1.noarch
Solution 3: break boss-skynet by ignoring some of its dependencies
Choose from above solutions by number or cancel [1/2/3/c] (c): 1
.
.
.

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
EOF

apt-get install --no-install-recommends boss-skynet

=== Configuration ===
The file /etc/skynet/boss.conf is used by all participants

It specifies the boss instance used by default and the credentials
required. Individual participants can override this.

The /etc/skynet/skynet.conf file specifies where SkyNET stores the
running participants and where daemontools monitors them
(/var/lib/SkyNET/services/ and /var/lib/SkyNET/store/ by default)

== BOSS OBS Plugin ==
The OBS plugin is designed to launch processes when a build/publish event occurs on the OBS

=== Installation ===
The plugin is installed on the OBS backend which runs the schedulers

openSUSE 11.2
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.2/Project:MINT:RC.repo
zypper ref
zypper in boss-obs-plugin

opensuse 11.4
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss-obs-plugin

=== Configuration ===

'''You *MUST* have a participant registered as "obs_event" running before enabling this plugin or ruote will silently fill up with hundreds/thousands/millions of stalled processes.'''

Edit the file:
/usr/lib/obs/server/BSConfig.pm
and define/add:
our $notification_plugin = "notify_boss";
our $BOSS_host="boss"; # hostname for server running BOSS AMQP
our $BOSS_user="boss"; # AMQP username
our $BOSS_passwd="boss"; # AMQP password (cleartext)

The plugin itself is installed in :
/usr/lib/obs/server/plugins/notify_boss.pm

The schedulers will need to be restarted to take effect. Note there is currently an issue that
the plugin will block if the AMQP server is not available - this will halt the OBS.

= Getting started =

Once you have installed BOSS and SkyNET you are ready to create and run some processes.

== Watch BOSS ==
On the BOSS machine:

/etc/init.d/boss log &

You'll see messages about the engine being alive and participant
registration. Further debug output is possible here.

== Setup a participant ==

On the skynet machine we need to setup some code as a participant:

skynet make_participant -n check -p /usr/share/doc/python-boss-skynet/example-check-participant
skynet make_participant -n notify -p /usr/share/doc/python-boss-skynet/example-notify-participant

This creates a daemontools/participant directory structure with a
symbolic link to the code for these participants.

Now we need to ensure they start to run under daemontools (and restart if
there's a problem):

skynet enable check
skynet enable notify

To watch the log output:

skynet log check &
skynet log notify &

Now we need to tell boss that there is something listening on these
queues and available for use in processes:

skynet register -n check
skynet register -n notify

The skynet command also supports:
* stop : shutdown the participant and don't respawn.
* reload : stop and start once
* register : can also specify the amqp queue to use if it's not the name

== Launch a process ==

With a participant ready to do some work, we're ready to launch a process:

/usr/share/boss-skynet/skynet_launch

The log output should show

2011-03-29 00:49:21.186558500 checking for /tmp/success
2011-03-29 00:49:21.186849500 Failed to read /tmp/success
2011-03-29 00:49:21.708706500 Nothing to say

Now do:

echo Hi there > /tmp/success

And relaunch:

/usr/share/boss-skynet/skynet_launch

gives
2011-03-29 00:49:31.659986500 checking for /tmp/success
2011-03-29 00:49:31.660440500 Read /tmp/success
2011-03-29 00:49:31.878978500 Send email saying : Hi there
2011-03-29 00:49:31.878980500

Of course a process can be launched from anywhere on the network using
any 'trigger' and participants also run on any machine.

== Debugging ==

==Next steps==

Do something useful with your installation using already available participants and launchers : [[/BOSS/Participants|Participants]]

Release Infrastructure

2011-04-26T13:53:59Z

Iamer: /* Tools */

== Overview ==
Describe the overall release and build infrastructure.

== Tools ==
* [[/BOSS|Build Orchestration BOSS]]
* [[/REVS|Release reporting REVS]]
* [[/IMG|Image Creation IMG]]
* Build System OBS

* [[/Packaging|Release team packaging information]]
* [[/BOSS_Participant|Participant documentation]]
* [[/BOSS_Launcher|Launcher documentation]]
* [[/BOSS_Process|Process documentation]]
* [[/BOSS_Testing|Testing]]

==New BOSS SkyNET==
* [[/BOSS/Installation|BOSS and Skynet]]
* [[/BOSS/Participants|Participants]]

Release Infrastructure/BOSS/Participants

2011-04-26T13:52:22Z

Iamer:

Release Infrastructure/BOSS/Participants

2011-04-26T13:51:35Z

Iamer: /* OBS process / workflow */

Release Infrastructure/BOSS/Participants

2011-04-26T13:50:54Z

Iamer: /* Notification */

Release Infrastructure/BOSS/Participants

2011-04-26T13:50:13Z

Iamer: /* OBS prechecks */

Release Infrastructure/BOSS/Participants

2011-04-26T13:07:38Z

Iamer: Created page with "List of participants ported / implemented for BOSS SkyNET : == OBS prechecks == GIT : https://meego.gitorious.org/meego-infrastructure-tools/boss-participant-prechecks Package..."

Release Infrastructure/BOSS/Installation

2011-04-13T06:52:14Z

Iamer: /* Overview */

= Overview =

Whilst BOSS is deployed in production, this packaged version is still
in beta-testing and is subject to change. In particular the
repositories are not yet finalised (eg installing from :RC).

However BOSS and SkyNET both install and run at a basic level on a
clean Debian 6.0 or OpenSuse 11.4 installation.

= Installation =

== Preparation ==

* Proxy : On OpenSUSE you need to set the proxy in /etc/sysconfig/proxy

== BOSS ==
=== Overview ===
BOSS provides the ruote worker and the main AMQP server.

This component is the main process dispatcher; no work happens
here but all process steps are initiated from here.

Currently BOSS uses the Ruote filesystem storage system and the
worker and viewer need to run on the same system. This will change
when BOSS moves to Redis.

The stable and testing projects are at the community OBS, the state
of the packages can be monitored at :

Testing : https://build.pub.meego.com/project/monitor?project=Project:MINT:Testing

Stable : (No release yet) https://build.pub.meego.com/project/monitor?project=Project:MINT

=== SUSE ===

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo
zypper ref
zypper in boss

(new version of BOSS on opensuse 11.4)
zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss

This will install Rabbit MQ and the boss worker daemon.

To run boss:

rcboss start / stop

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss

This will install Rabbit MQ and the boss worker daemon.

Then, as usual
/etc/init.d/boss {start|stop|restart|force-reload|log}

Default settings are found in /etc/default/boss

=== Configuration ===

BOSS is preconfigured with username boss and password boss on the boss vhost
(AMQP will lose vhosts soon)

BOSS stores workitem and process data in /var/spool/boss/

== BOSS-Viewer ==

This must be installed on the boss server when using the FS storage.

The default port is 9292, point your browser at http://127.0.0.1:9292/_ruote/ (if you are running on the same machine)

#FIXME: Make sure you go to /_ruote/ otherwise you will get a big scary error.

=== SUSE ===

zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss-viewer

This will install boss-viewer.

To run boss-viewer:

rcboss-viewer start / stop

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss-viewer

This will install the boss viewer.

Then, as usual
/etc/init.d/boss {start|stop|restart|force-reload|log}

== SkyNET ==
=== Overview ===

SkyNET manages long-running BOSS Participants.

It provides:

* daemonisation
* start/stop control
* consistent logging
* user-owned participants

=== SUSE ===

zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss-skynet

If you installed a minimal OpenSUSE you might get a dependency error and as shown here proceed with choice 1 :

Problem: boss-skynet-0.1-5.1.noarch requires /usr/bin/python, but this requirement cannot be provided
uninstallable providers: python-base-2.7-8.4.i586[openSUSE-11.4-11.4-0]
python-base-2.7-8.4.x86_64[openSUSE-11.4-11.4-0]
Solution 1: deinstallation of patterns-openSUSE-minimal_base-11.4-6.9.1.x86_64
Solution 2: do not install boss-skynet-0.1-5.1.noarch
Solution 3: break boss-skynet by ignoring some of its dependencies
Choose from above solutions by number or cancel [1/2/3/c] (c): 1
.
.
.

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
EOF

apt-get install --no-install-recommends boss-skynet

=== Configuration ===
The file /etc/skynet/boss.conf is used by all participants

It specifies the boss instance used by default and the credentials
required. Individual participants can override this.

The /etc/skynet/skynet.conf file specifies where SkyNET stores the
running participants and where daemontools monitors them
(/var/lib/SkyNET/services/ and /var/lib/SkyNET/store/ by default)

== BOSS OBS Plugin ==
The OBS plugin is designed to launch processes when a build/publish event occurs on the OBS

=== Installation ===
The plugin is installed on the OBS backend which runs the schedulers

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ref
zypper in boss-obs-plugin

(new version of BOSS on opensuse 11.4)
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper ref
zypper in boss-obs-plugin

=== Configuration ===

'''You *MUST* have a participant registered as "obs_event" running before enabling this plugin or ruote will silently fill up with hundreds/thousands/millions of stalled processes.'''

Edit the file:
/usr/lib/obs/server/BSConfig.pm
and define/add:
our $notification_plugin = "notify_boss";
our $BOSS_host="boss"; # hostname for server running BOSS AMQP
our $BOSS_user="boss"; # AMQP username
our $BOSS_passwd="boss"; # AMQP password (cleartext)

The plugin itself is installed in :
/usr/lib/obs/server/plugins/notify_boss.pm

The schedulers will need to be restarted to take effect. Note there is currently an issue that
the plugin will block if the AMQP server is not available - this will halt the OBS.

= Getting started =

Once you have installed BOSS and SkyNET you are ready to create and run some processes.

== Watch BOSS ==
On the BOSS machine:

/etc/init.d/boss log &

You'll see messages about the engine being alive and participant
registration. Further debug output is possible here.

== Setup a participant ==

On the skynet machine we need to setup some code as a participant:

skynet make_participant -n check -p /usr/share/doc/python-boss-skynet/example-check-participant
skynet make_participant -n notify -p /usr/share/doc/python-boss-skynet/example-notify-participant

This creates a daemontools/participant directory structure with a
symbolic link to the code for these participants.

Now we need to ensure they start to run under daemontools (and restart if
there's a problem):

skynet enable check
skynet enable notify

To watch the log output:

skynet log check &
skynet log notify &

Now we need to tell boss that there is something listening on these
queues and available for use in processes:

skynet register -n check
skynet register -n notify

The skynet command also supports:
* stop : shutdown the participant and don't respawn.
* reload : stop and start once
* register : can also specify the amqp queue to use if it's not the name

== Launch a process ==

With a participant ready to do some work, we're ready to launch a process:

/usr/share/boss-skynet/skynet_launch

The log output should show

2011-03-29 00:49:21.186558500 checking for /tmp/success
2011-03-29 00:49:21.186849500 Failed to read /tmp/success
2011-03-29 00:49:21.708706500 Nothing to say

Now do:

echo Hi there > /tmp/success

And relaunch:

/usr/share/boss-skynet/skynet_launch

gives
2011-03-29 00:49:31.659986500 checking for /tmp/success
2011-03-29 00:49:31.660440500 Read /tmp/success
2011-03-29 00:49:31.878978500 Send email saying : Hi there
2011-03-29 00:49:31.878980500

Of course a process can be launched from anywhere on the network using
any 'trigger' and participants also run on any machine.

== Debugging ==

Release Infrastructure/BOSS/Installation

2011-04-12T14:30:48Z

Iamer:

= Overview =

Whilst BOSS is deployed in production, this packaged version is still
in beta-testing and is subject to change. In particular the
repositories are not yet finalised (eg installing from :Testing).

However BOSS and SkyNET both install and run at a basic level on a
clean Debian 6.0 or OpenSuse 11.4 installation.

= Installation =

== Preparation ==

* Proxy : On OpenSUSE you need to set the proxy in /etc/sysconfig/proxy

== BOSS ==
=== Overview ===
BOSS provides the ruote worker and the main AMQP server.

This component is the main process dispatcher; no work happens
here but all process steps are initiated from here.

Currently BOSS uses the Ruote filesystem storage system and the
worker and viewer need to run on the same system. This will change
when BOSS moves to Redis.

=== SUSE ===

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo
zypper ref
zypper in boss

(new version of BOSS on opensuse 11.4)
zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss

This will install Rabbit MQ and the boss worker daemon.

To run boss:

rcboss start / stop

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss

This will install Rabbit MQ and the boss worker daemon.

Then, as usual
/etc/init.d/boss {start|stop|restart|force-reload|log}

Default settings are found in /etc/default/boss

=== Configuration ===

BOSS is preconfigured with username boss and password boss on the boss vhost
(AMQP will lose vhosts soon)

BOSS stores workitem and process data in /var/spool/boss/

== BOSS-Viewer ==

This must be installed on the boss server when using the FS storage.

=== SUSE ===

zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss-viewer

This will install boss-viewer.

To run boss-viewer:

rcboss-viewer start / stop

The default port is 9292, point your browser at http://127.0.0.1:9292/_ruote/ (if you are running on the same machine)

#FIXME: Make sure you go to /_ruote/ otherwise you will get a big scary error.

=== Debian 6 ===
Currently n/a

== SkyNET ==
=== Overview ===

SkyNET manages long-running BOSS Participants.

It provides:

* daemonisation
* start/stop control
* consistent logging
* user-owned participants

=== SUSE ===

zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss-skynet

If you installed a minimal OpenSUSE you might get a dependency error and as shown here proceed with choice 1 :

Problem: boss-skynet-0.1-5.1.noarch requires /usr/bin/python, but this requirement cannot be provided
uninstallable providers: python-base-2.7-8.4.i586[openSUSE-11.4-11.4-0]
python-base-2.7-8.4.x86_64[openSUSE-11.4-11.4-0]
Solution 1: deinstallation of patterns-openSUSE-minimal_base-11.4-6.9.1.x86_64
Solution 2: do not install boss-skynet-0.1-5.1.noarch
Solution 3: break boss-skynet by ignoring some of its dependencies
Choose from above solutions by number or cancel [1/2/3/c] (c): 1
.
.
.

=== Debian 6 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
EOF

apt-get install --no-install-recommends boss-skynet

=== Configuration ===
The file /etc/skynet/boss.conf is used by all participants

It specifies the boss instance used by default and the credentials
required. Individual participants can override this.

The /etc/skynet/skynet.conf file specifies where SkyNET stores the
running participants and where daemontools monitors them
(/var/lib/SkyNET/services/ and /var/lib/SkyNET/store/ by default)

== BOSS OBS Plugin ==
The OBS plugin is designed to launch processes when a build/publish event occurs on the OBS

=== Installation ===
The plugin is installed on the OBS backend which runs the schedulers

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ref
zypper in boss-obs-plugin

(new version of BOSS on opensuse 11.4)
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss-obs-plugin

=== Configuration ===

'''You *MUST* have a participant registered as "obs_event" running before enabling this plugin or ruote will silently fill up with hundreds/thousands/millions of stalled processes.'''

Edit the file:
/usr/lib/obs/server/BSConfig.pm
and define/add:
our $notification_plugin = "notify_boss";
our $BOSS_host="boss"; # hostname for server running BOSS AMQP
our $BOSS_user="boss"; # AMQP username
our $BOSS_passwd="boss"; # AMQP password (cleartext)

The plugin itself is installed in :
/usr/lib/obs/server/plugins/notify_boss.pm

The schedulers will need to be restarted to take effect. Note there is currently an issue that
the plugin will block if the AMQP server is not available - this will halt the OBS.

= Getting started =

Once you have installed BOSS and SkyNET you are ready to create and run some processes.

== Watch BOSS ==
On the BOSS machine:

/etc/init.d/boss log &

You'll see messages about the engine being alive and participant
registration. Further debug output is possible here.

== Setup a participant ==

On the skynet machine we need to setup some code as a participant:

skynet make_participant -n check -p /usr/share/doc/python-boss-skynet/example-check-participant
skynet make_participant -n notify -p /usr/share/doc/python-boss-skynet/example-notify-participant

This creates a daemontools/participant directory structure with a
symbolic link to the code for these participants.

Now we need to ensure they start to run under daemontools (and restart if
there's a problem):

skynet enable check
skynet enable notify

To watch the log output:

skynet log check &
skynet log notify &

Now we need to tell boss that there is something listening on these
queues and available for use in processes:

skynet register -n check
skynet register -n notify

The skynet command also supports:
* stop : shutdown the participant and don't respawn.
* reload : stop and start once
* register : can also specify the amqp queue to use if it's not the name

== Launch a process ==

With a participant ready to do some work, we're ready to launch a process:

/usr/share/boss-skynet/skynet_launch

The log output should show

2011-03-29 00:49:21.186558500 checking for /tmp/success
2011-03-29 00:49:21.186849500 Failed to read /tmp/success
2011-03-29 00:49:21.708706500 Nothing to say

Now do:

echo Hi there > /tmp/success

And relaunch:

/usr/share/boss-skynet/skynet_launch

gives
2011-03-29 00:49:31.659986500 checking for /tmp/success
2011-03-29 00:49:31.660440500 Read /tmp/success
2011-03-29 00:49:31.878978500 Send email saying : Hi there
2011-03-29 00:49:31.878980500

Of course a process can be launched from anywhere on the network using
any 'trigger' and participants also run on any machine.

== Debugging ==

Release Infrastructure/BOSS/Installation

2011-04-12T14:12:56Z

Iamer: /* SUSE */ start documenting boss-viewer

= Overview =

Whilst BOSS is deployed in production, this packaged version is still
in beta-testing and is subject to change. In particular the
repositories are not yet finalised (eg installing from :Testing).

However BOSS and SkyNET both install and run at a basic level on a
clean Debian 6.0 or OpenSuse 11.4 installation.

= Installation =

== Preparation ==

* Proxy : On OpenSUSE you need to set the proxy in /etc/sysconfig/proxy

== BOSS ==
=== Overview ===
BOSS provides the ruote worker and the main AMQP server.

This component is the main process dispatcher; no work happens
here but all process steps are initiated from here.

Currently BOSS uses the Ruote filesystem storage system and the
worker and viewer need to run on the same system. This will change
when BOSS moves to Redis.

=== SUSE ===

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo
zypper ref
zypper in boss

(new version of BOSS on opensuse 11.4)
zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss

This will install Rabbit MQ and the boss worker daemon.

To run boss:

rcboss start / stop

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss

This will install Rabbit MQ and the boss worker daemon.

Then, as usual
/etc/init.d/boss {start|stop|restart|force-reload|log}

Default settings are found in /etc/default/boss

=== Configuration ===

BOSS is preconfigured with username boss and password boss on the boss vhost
(AMQP will lose vhosts soon)

BOSS stores workitem and process data in /var/spool/boss/

== BOSS-Viewer ==

This must be installed on the boss server when using the FS storage.

=== SUSE ===

zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss-viewer

This will install boss-viewer.

To run boss-viewer:

rcboss-viewer start / stop

The default port is 9292, point your browser at 127.0.0.1:9292 (if you are running on the same machine)

=== Debian 6 ===
Currently n/a

== SkyNET ==
=== Overview ===

SkyNET manages long-running BOSS Participants.

It provides:

* daemonisation
* start/stop control
* consistent logging
* user-owned participants

=== SUSE ===

zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss-skynet

If you installed a minimal OpenSUSE you might get a dependency error and as shown here proceed with choice 1 :

Problem: boss-skynet-0.1-5.1.noarch requires /usr/bin/python, but this requirement cannot be provided
uninstallable providers: python-base-2.7-8.4.i586[openSUSE-11.4-11.4-0]
python-base-2.7-8.4.x86_64[openSUSE-11.4-11.4-0]
Solution 1: deinstallation of patterns-openSUSE-minimal_base-11.4-6.9.1.x86_64
Solution 2: do not install boss-skynet-0.1-5.1.noarch
Solution 3: break boss-skynet by ignoring some of its dependencies
Choose from above solutions by number or cancel [1/2/3/c] (c): 1
.
.
.

=== Debian 6 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
EOF

apt-get install --no-install-recommends boss-skynet

=== Configuration ===
The file /etc/skynet/boss.conf is used by all participants

It specifies the boss instance used by default and the credentials
required. Individual participants can override this.

The /etc/skynet/skynet.conf file specifies where SkyNET stores the
running participants and where daemontools monitors them
(/var/lib/SkyNET/services/ and /var/lib/SkyNET/store/ by default)

== BOSS OBS Plugin ==
The OBS plugin is designed to launch processes when a build/publish event occurs on the OBS

=== Installation ===
The plugin is installed on the OBS backend which runs the schedulers

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ref
zypper in boss-obs-plugin

(new version of BOSS on opensuse 11.4)
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss-obs-plugin

=== Configuration ===

'''You *MUST* have a participant registered as "obs_event" running before enabling this plugin or ruote will silently fill up with hundreds/thousands/millions of stalled processes.'''

Edit the file:
/usr/lib/obs/server/BSConfig.pm
and define/add:
our $notification_plugin = "notify_boss";
our $BOSS_host="boss"; # hostname for server running BOSS AMQP
our $BOSS_user="boss"; # AMQP username
our $BOSS_passwd="boss"; # AMQP password (cleartext)

The plugin itself is installed in :
/usr/lib/obs/server/plugins/notify_boss.pm

The schedulers will need to be restarted to take effect. Note there is currently an issue that
the plugin will block if the AMQP server is not available - this will halt the OBS.

= Getting started =

Once you have installed BOSS and SkyNET you are ready to create and run some processes.

== Watch BOSS ==
On the BOSS machine:

/etc/init.d/boss log &

You'll see messages about the engine being alive and participant
registration. Further debug output is possible here.

== Setup a participant ==

On the skynet machine we need to setup some code as a participant:

skynet make_participant -n check -p /usr/share/doc/python-boss-skynet/example-check-participant
skynet make_participant -n notify -p /usr/share/doc/python-boss-skynet/example-notify-participant

This creates a daemontools/participant directory structure with a
symbolic link to the code for these participants.

Now we need to ensure they start to run under daemontools (and restart if
there's a problem):

skynet enable check
skynet enable notify

To watch the log output:

skynet log check &
skynet log notify &

Now we need to tell boss that there is something listening on these
queues and available for use in processes:

skynet register -n check
skynet register -n notify

The skynet command also supports:
* stop : shutdown the participant and don't respawn.
* reload : stop and start once
* register : can also specify the amqp queue to use if it's not the name

== Launch a process ==

With a participant ready to do some work, we're ready to launch a process:

/usr/share/boss-skynet/skynet_launch

The log output should show

2011-03-29 00:49:21.186558500 checking for /tmp/success
2011-03-29 00:49:21.186849500 Failed to read /tmp/success
2011-03-29 00:49:21.708706500 Nothing to say

Now do:

echo Hi there > /tmp/success

And relaunch:

/usr/share/boss-skynet/skynet_launch

gives
2011-03-29 00:49:31.659986500 checking for /tmp/success
2011-03-29 00:49:31.660440500 Read /tmp/success
2011-03-29 00:49:31.878978500 Send email saying : Hi there
2011-03-29 00:49:31.878980500

Of course a process can be launched from anywhere on the network using
any 'trigger' and participants also run on any machine.

== Debugging ==

Release Infrastructure/BOSS/Installation

2011-04-05T07:28:58Z

Iamer: add note about proxy

= Overview =

Whilst BOSS is deployed in production, this packaged version is still
in beta-testing and is subject to change. In particular the
repositories are not yet finalized (eg installing from :Testing).

However BOSS and SkyNET both install and run at a basic level on a
clean Debian 6.0 or OpenSuse 11.4 installation.

= Installation =

== Preparation ==

* Proxy : On OpenSUSE you need to set the proxy in /etc/sysconfig/proxy

== BOSS ==
=== Overview ===
BOSS provides the ruote worker and the main AMQP server.

This component is the main process dispatcher; no work happens
here but all process steps are initiated from here.

Currently BOSS uses the Ruote filesystem storage system and the
worker and viewer need to run on the same system. This will change
when BOSS moves to Redis.

=== SUSE ===

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo
zypper ref
zypper in boss

(new version of BOSS on opensuse 11.4)
zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss

This will install Rabbit MQ and the boss worker daemon.

To run boss:

rcboss start / stop

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss

This will install Rabbit MQ and the boss worker daemon.

Then, as usual
/etc/init.d/boss {start|stop|restart|force-reload|log}

Default settings are found in /etc/default/boss

=== Configuration ===

BOSS is preconfigured with username boss and password boss on the boss vhost
(AMQP will lose vhosts soon)

BOSS stores workitem and process data in /var/spool/boss/

== BOSS-Viewer ==

This must be installed on the boss server when using the FS storage.

=== SUSE ===
Currently n/a

=== Debian 6 ===
Currently n/a

== SkyNET ==
=== Overview ===

SkyNET manages long-running BOSS Participants.

It provides:

* daemonisation
* start/stop control
* consistent logging
* user-owned participants

=== SUSE ===

zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss-skynet

If you installed a minimal OpenSUSE you might get a dependency error and as shown here proceed with choice 1 :

Problem: boss-skynet-0.1-5.1.noarch requires /usr/bin/python, but this requirement cannot be provided
uninstallable providers: python-base-2.7-8.4.i586[openSUSE-11.4-11.4-0]
python-base-2.7-8.4.x86_64[openSUSE-11.4-11.4-0]
Solution 1: deinstallation of patterns-openSUSE-minimal_base-11.4-6.9.1.x86_64
Solution 2: do not install boss-skynet-0.1-5.1.noarch
Solution 3: break boss-skynet by ignoring some of its dependencies
Choose from above solutions by number or cancel [1/2/3/c] (c): 1
.
.
.

=== Debian 6 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
EOF

apt-get install --no-install-recommends boss-skynet

=== Configuration ===
The file /etc/skynet/boss.conf is used by all participants

It specifies the boss instance used by default and the credentials
required. Individual participants can override this.

The /etc/skynet/skynet.conf file specifies where SkyNET stores the
running participants and where daemontools monitors them
(/var/lib/SkyNET/services/ and /var/lib/SkyNET/store/ by default)

== BOSS OBS Plugin ==
The OBS plugin is designed to launch processes when a build/publish event occurs on the OBS

=== Installation ===
The plugin is installed on the OBS backend which runs the schedulers

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ref
zypper in boss-obs-plugin

(new version of BOSS on opensuse 11.4)
zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss-obs-plugin

=== Configuration ===

'''You *MUST* have a participant registered as "obs_event" running before enabling this plugin or ruote will silently fill up with hundreds/thousands/millions of stalled processes.'''

Edit the file:
/usr/lib/obs/server/BSConfig.pm
and define/add:
our $notification_plugin = "notify_boss";
our $BOSS_host="boss"; # hostname for server running BOSS AMQP
our $BOSS_user="boss"; # AMQP username
our $BOSS_passwd="boss"; # AMQP password (cleartext)

The plugin itself is installed in :
/usr/lib/obs/server/plugins/notify_boss.pm

The schedulers will need to be restarted to take effect. Note there is currently an issue that
the plugin will block if the AMQP server is not available - this will halt the OBS.

= Getting started =

Once you have installed BOSS and SkyNET you are ready to create and run some processes.

== Watch BOSS ==
On the BOSS machine:

/etc/init.d/boss log &

You'll see messages about the engine being alive and participant
registration. Further debug output is possible here.

== Setup a participant ==

On the skynet machine we need to setup some code as a participant:

skynet make_participant -n check -p /usr/share/doc/python-boss-skynet/example-check-participant
skynet make_participant -n notify -p /usr/share/doc/python-boss-skynet/example-notify-participant

This creates a daemontools/participant directory structure with a
symbolic link to the code for these participants.

Now we need to ensure they start to run under daemontools (and restart if
there's a problem):

skynet enable check
skynet enable notify

To watch the log output:

skynet log check &
skynet log notify &

Now we need to tell boss that there is something listening on these
queues and available for use in processes:

skynet register -n check
skynet register -n notify

The skynet command also supports:
* stop : shutdown the participant and don't respawn.
* reload : stop and start once
* register : can also specify the amqp queue to use if it's not the name

== Launch a process ==

With a participant ready to do some work, we're ready to launch a process:

/usr/share/boss-skynet/skynet_launch

The log output should show

2011-03-29 00:49:21.186558500 checking for /tmp/success
2011-03-29 00:49:21.186849500 Failed to read /tmp/success
2011-03-29 00:49:21.708706500 Nothing to say

Now do:

echo Hi there > /tmp/success

And relaunch:

/usr/share/boss-skynet/skynet_launch

gives
2011-03-29 00:49:31.659986500 checking for /tmp/success
2011-03-29 00:49:31.660440500 Read /tmp/success
2011-03-29 00:49:31.878978500 Send email saying : Hi there
2011-03-29 00:49:31.878980500

Of course a process can be launched from anywhere on the network using
any 'trigger' and participants also run on any machine.

== Debugging ==

Release Infrastructure/BOSS/Installation

2011-04-04T23:56:57Z

Iamer: /* SUSE */ mention possibly confusing dependency error

= Overview =

Whilst BOSS is deployed in production, this packaged version is still
in beta-testing and is subject to change. In particular the
repositories are not yet finalized (eg installing from :Testing).

However BOSS and SkyNET both install and run at a basic level on a
clean Debian 6.0 or OpenSuse 11.4 installation.

= Installation =
== BOSS ==
=== Overview ===
BOSS provides the ruote worker and the main AMQP server.

This component is the main process dispatcher; no work happens
here but all process steps are initiated from here.

Currently BOSS uses the Ruote filesystem storage system and the
worker and viewer need to run on the same system. This will change
when BOSS moves to Redis.

=== SUSE ===

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo
zypper ref
zypper in boss

(new version of BOSS on opensuse 11.4)
zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss

This will install Rabbit MQ and the boss worker daemon.

To run boss:

rcboss start / stop

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss

This will install Rabbit MQ and the boss worker daemon.

Then, as usual
/etc/init.d/boss {start|stop|restart|force-reload|log}

Default settings are found in /etc/default/boss

=== Configuration ===

BOSS is preconfigured with username boss and password boss on the boss vhost
(AMQP will lose vhosts soon)

BOSS stores workitem and process data in /var/spool/boss/

== BOSS-Viewer ==

This must be installed on the boss server when using the FS storage.

=== SUSE ===
Currently n/a

=== Debian 6 ===
Currently n/a

== SkyNET ==
=== Overview ===

SkyNET manages long-running BOSS Participants.

It provides:

* daemonisation
* start/stop control
* consistent logging
* user-owned participants

=== SUSE ===

zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss-skynet

If you installed a minimal OpenSUSE you might get a dependency error and as shown here proceed with choice 1 :

Problem: boss-skynet-0.1-5.1.noarch requires /usr/bin/python, but this requirement cannot be provided
uninstallable providers: python-base-2.7-8.4.i586[openSUSE-11.4-11.4-0]
python-base-2.7-8.4.x86_64[openSUSE-11.4-11.4-0]
Solution 1: deinstallation of patterns-openSUSE-minimal_base-11.4-6.9.1.x86_64
Solution 2: do not install boss-skynet-0.1-5.1.noarch
Solution 3: break boss-skynet by ignoring some of its dependencies
Choose from above solutions by number or cancel [1/2/3/c] (c): 1
.
.
.

=== Debian 6 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
EOF

apt-get install --no-install-recommends boss-skynet

=== Configuration ===
The file /etc/skynet/boss.conf is used by all participants

It specifies the boss instance used by default and the credentials
required. Individual participants can override this.

The /etc/skynet/skynet.conf file specifies where SkyNET stores the
running participants and where daemontools monitors them
(/var/lib/SkyNET/services/ and /var/lib/SkyNET/store/ by default)

== BOSS OBS Plugin ==
The OBS plugin is designed to launch processes when a build/publish event occurs on the OBS

=== Installation ===
The plugin is installed on the OBS backend which runs the schedulers

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ref
zypper in boss-obs-plugin

(new version of BOSS on opensuse 11.4)
zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss-obs-plugin

=== Configuration ===

'''You *MUST* have a participant registered as "obs_event" running before enabling this plugin or ruote will silently fill up with hundreds/thousands/millions of stalled processes.'''

Edit the file:
/usr/lib/obs/server/BSConfig.pm
and define/add:
our $notification_plugin = "notify_boss";
our $BOSS_host="boss"; # hostname for server running BOSS AMQP
our $BOSS_user="boss"; # AMQP username
our $BOSS_passwd="boss"; # AMQP password (cleartext)

The plugin itself is installed in :
/usr/lib/obs/server/plugins/notify_boss.pm

The schedulers will need to be restarted to take effect. Note there is currently an issue that
the plugin will block if the AMQP server is not available - this will halt the OBS.

= Getting started =

Once you have installed BOSS and SkyNET you are ready to create and run some processes.

== Watch BOSS ==
On the BOSS machine:

/etc/init.d/boss log &

You'll see messages about the engine being alive and participant
registration. Further debug output is possible here.

== Setup a participant ==

On the skynet machine we need to setup some code as a participant:

skynet make_participant -n check -p /usr/share/doc/python-boss-skynet/example-check-participant
skynet make_participant -n notify -p /usr/share/doc/python-boss-skynet/example-notify-participant

This creates a daemontools/participant directory structure with a
symbolic link to the code for these participants.

Now we need to ensure they start to run under daemontools (and restart if
there's a problem):

skynet enable check
skynet enable notify

To watch the log output:

skynet log check &
skynet log notify &

Now we need to tell boss that there is something listening on these
queues and available for use in processes:

skynet register -n check
skynet register -n notify

The skynet command also supports:
* stop : shutdown the participant and don't respawn.
* reload : stop and start once
* register : can also specify the amqp queue to use if it's not the name

== Launch a process ==

With a participant ready to do some work, we're ready to launch a process:

/usr/share/boss-skynet/skynet_launch

The log output should show

2011-03-29 00:49:21.186558500 checking for /tmp/success
2011-03-29 00:49:21.186849500 Failed to read /tmp/success
2011-03-29 00:49:21.708706500 Nothing to say

Now do:

echo Hi there > /tmp/success

And relaunch:

/usr/share/boss-skynet/skynet_launch

gives
2011-03-29 00:49:31.659986500 checking for /tmp/success
2011-03-29 00:49:31.660440500 Read /tmp/success
2011-03-29 00:49:31.878978500 Send email saying : Hi there
2011-03-29 00:49:31.878980500

Of course a process can be launched from anywhere on the network using
any 'trigger' and participants also run on any machine.

== Debugging ==

Release Infrastructure/BOSS/Installation

2011-04-04T23:33:27Z

Iamer: Updated OpenSUSE 11.4 instructions.

= Overview =

Whilst BOSS is deployed in production, this packaged version is still
in beta-testing and is subject to change. In particular the
repositories are not yet finalized (eg installing from :Testing).

However BOSS and SkyNET both install and run at a basic level on a
clean Debian 6.0 or OpenSuse 11.4 installation.

= Installation =
== BOSS ==
=== Overview ===
BOSS provides the ruote worker and the main AMQP server.

This component is the main process dispatcher; no work happens
here but all process steps are initiated from here.

Currently BOSS uses the Ruote filesystem storage system and the
worker and viewer need to run on the same system. This will change
when BOSS moves to Redis.

=== SUSE ===

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo
zypper ref
zypper in boss

(new version of BOSS on opensuse 11.4)
zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss

This will install Rabbit MQ and the boss worker daemon.

To run boss:

rcboss start / stop

=== Debian Squeeze/6.0 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss

This will install Rabbit MQ and the boss worker daemon.

Then, as usual
/etc/init.d/boss {start|stop|restart|force-reload|log}

Default settings are found in /etc/default/boss

=== Configuration ===

BOSS is preconfigured with username boss and password boss on the boss vhost
(AMQP will lose vhosts soon)

BOSS stores workitem and process data in /var/spool/boss/

== BOSS-Viewer ==

This must be installed on the boss server when using the FS storage.

=== SUSE ===
Currently n/a

=== Debian 6 ===
Currently n/a

== SkyNET ==
=== Overview ===

SkyNET manages long-running BOSS Participants.

It provides:

* daemonisation
* start/stop control
* consistent logging
* user-owned participants

=== SUSE ===

zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss-skynet

=== Debian 6 ===

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
EOF

apt-get install --no-install-recommends boss-skynet

=== Configuration ===
The file /etc/skynet/boss.conf is used by all participants

It specifies the boss instance used by default and the credentials
required. Individual participants can override this.

The /etc/skynet/skynet.conf file specifies where SkyNET stores the
running participants and where daemontools monitors them
(/var/lib/SkyNET/services/ and /var/lib/SkyNET/store/ by default)

== BOSS OBS Plugin ==
The OBS plugin is designed to launch processes when a build/publish event occurs on the OBS

=== Installation ===
The plugin is installed on the OBS backend which runs the schedulers

(old version of BOSS on openSUSE 11.2)
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ref
zypper in boss-obs-plugin

(new version of BOSS on opensuse 11.4)
zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss-obs-plugin

=== Configuration ===

'''You *MUST* have a participant registered as "obs_event" running before enabling this plugin or ruote will silently fill up with hundreds/thousands/millions of stalled processes.'''

Edit the file:
/usr/lib/obs/server/BSConfig.pm
and define/add:
our $notification_plugin = "notify_boss";
our $BOSS_host="boss"; # hostname for server running BOSS AMQP
our $BOSS_user="boss"; # AMQP username
our $BOSS_passwd="boss"; # AMQP password (cleartext)

The plugin itself is installed in :
/usr/lib/obs/server/plugins/notify_boss.pm

The schedulers will need to be restarted to take effect. Note there is currently an issue that
the plugin will block if the AMQP server is not available - this will halt the OBS.

= Getting started =

Once you have installed BOSS and SkyNET you are ready to create and run some processes.

== Watch BOSS ==
On the BOSS machine:

/etc/init.d/boss log &

You'll see messages about the engine being alive and participant
registration. Further debug output is possible here.

== Setup a participant ==

On the skynet machine we need to setup some code as a participant:

skynet make_participant -n check -p /usr/share/doc/python-boss-skynet/example-check-participant
skynet make_participant -n notify -p /usr/share/doc/python-boss-skynet/example-notify-participant

This creates a daemontools/participant directory structure with a
symbolic link to the code for these participants.

Now we need to ensure they start to run under daemontools (and restart if
there's a problem):

skynet enable check
skynet enable notify

To watch the log output:

skynet log check &
skynet log notify &

Now we need to tell boss that there is something listening on these
queues and available for use in processes:

skynet register -n check
skynet register -n notify

The skynet command also supports:
* stop : shutdown the participant and don't respawn.
* reload : stop and start once
* register : can also specify the amqp queue to use if it's not the name

== Launch a process ==

With a participant ready to do some work, we're ready to launch a process:

/usr/share/boss-skynet/skynet_launch

The log output should show

2011-03-29 00:49:21.186558500 checking for /tmp/success
2011-03-29 00:49:21.186849500 Failed to read /tmp/success
2011-03-29 00:49:21.708706500 Nothing to say

Now do:

echo Hi there > /tmp/success

And relaunch:

/usr/share/boss-skynet/skynet_launch

gives
2011-03-29 00:49:31.659986500 checking for /tmp/success
2011-03-29 00:49:31.660440500 Read /tmp/success
2011-03-29 00:49:31.878978500 Send email saying : Hi there
2011-03-29 00:49:31.878980500

Of course a process can be launched from anywhere on the network using
any 'trigger' and participants also run on any machine.

== Debugging ==

Release Infrastructure/REVS

2010-12-02T10:40:33Z

Iamer: /* REVS */

= REVS =

So you want to get a ''grasp'' on what's going on when you release? Enter REVS – Release Engineering Visibility System (a software previously know as grasp).

REVS...
* provides an authoritative package database handling Products, Platforms, Licenses, Bug/Feature Trackers, Roles, Packages etc
* contains a release data-warehouse section for Releases, Baselines, Changelogs, Bugs, Images etc
* has a flexible group hierarchy allowing data to be presented as it is needed
* provides high speed access to all aspects of a release
* allows management reports on bugs, features, changes, packages

==Use cases (WIP)==
===Basic===
# Import releases of baselines from a repository of packages.
# Compare the delta between baselines showing packages added, removed, unchanged and changed and their versions.
# List bugs that were mentioned in the changelogs of packages that have changed as links to their respective bug trackers.
# Mix baselines into products and be able to visualise and compare them as above.
# List all packages of a certain release, product or baseline and present them in a printable or machine readable format.

===Advanced===
# Define and display all kinds of grouping of packages by : category or arbitrary naming.
# Group and display packages by license and maintainer.
# Associate audit trail of where a package originated (source control url and tag), and how it was accepted into a baseline (Promotion requests)
# Store URLs of repositories and build configuration templates (kickstarts) that can be used to reproduce an image.

It integrates with, but isn't part of the [[BOSS]] workflow system.

Implementation consists of :
* Data model
** Master python models
*** sack : for collections of packages
*** dwh : for the release datawarehouse
*** nyc : License audit support (No You Can't)
* Data IN
** Rest API for updates
** Data management via Django
** Import from changes, rpms etc
** BOSS continuous integration participant
* Data OUT
** Rest API for updates
** Package set comparison reporting via Django

== Download Source Code ==

REVS is currently available in gitorious:

http://meego.gitorious.org/meego-infrastructure-tools/revs

git clone git@gitorious.org:meego-infrastructure-tools/revs.git

Installable packages (deb/rpm) can be found here :

https://build.opensuse.org/package/show?package=revs&project=Maemo%3AMeeGo-Infra

== Installing And Setting Up ==

Installing and setting up REVS is guided in the [http://meego.gitorious.org/meego-infrastructure-tools/revs/blobs/master/README README] file in the source code. It includes the following sections

* Setting REVS Up
** Prerequisites
** Running REVS
* Admin Interface and Trackers
** Admin interface
* Test Data
** Adding Data

When installing on openSuse, the following extra repos are needed :

zypper ar http://download.opensuse.org/repositories/server:/http/openSUSE_11.2/server:http.repo
zypper ar http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_11.2/devel:languages:python.repo

== Testing ==

REVS allows usage of Django test framework to run unit tests on its components. Example is provided by running

<pre>
run_tests.sh
</pre>

in the source root directory. It reports if all tests passed.

== REST ==

REVS has an API for controlling the models via HTTP. Currently Baseline, Package, Platform, Product, Product Release and Tracker models are modifiable and readable using our RESTful API.

Our REST implementation of choice is [http://bitbucket.org/jespern/django-piston/wiki/Home Django Piston], as it allows direct model handling using the Django ORM.

Currently the best way to use the API is using curl or anything that can GET / POST JSON. In curl this is achieved using

-d @file.json

as the command line parameter to get data from a file. See the API section for details and examples for each model.

Also the responses are given in JSON, so one might need eg. [http://packages.debian.org/lenny/edit-json edit-json] to view or edit JSON files.

=== API ===
The API handlers all support either pure url-encoded data or JSON. Each model is capable of both but this document reviews only the JSON input and output.
----
==== Baseline ====
'''JSON format for input and output'''
<pre>
{
"name": "1.1.80.4.20101102.1",
"date": "2010-11-10",
"pkgs": [
{
"packageversion": {
"version": "1.0.4",
"changelogentry_set": [
{
"author": "Anas Nashif ",
"log": "- Updated with latest spectacle\n- Include YAML file in source rpm\n",
"date": "Sat Feb 27 2010",
"version": "1.0.4"
},
{
"author": "Li Peng ",
"log": "- libXres 1.0.4\n",
"date": "Fri Dec 11 2009",
"version": "1.0.4"
}
]
},
"name": "libXres"
},
(many more packages)
}
"contains": [
],
"platform": {
"name": "MeeGo"
}
</pre>
----
==== Package ====

'''JSON format for input and output'''
<pre>
{
"name": "bluez",
"platform": "MeeGo",
"latest_version": "4.76",
"versions": [
{
"version": "4.70",
"package_system": "rpm",
"changelogentries": [
{
"date": "Thu Sep 30 2010",
"author": "Zhu Yanhai ",
"email": "<yanhai.zhu@linux.intel.com>",
"links": [],
"log": "- Add powered.patch to make sure the initial power status aligned\n with connman's requirement.\n"
},
{
"date": "Fri Aug 27 2010",
"author": "Zhu Yanhai ",
"email": "<yanhai.zhu@linux.intel.com>",
"links": [],
"log": "- Upgrade to 4.70\n"
}
]
},
... (many more versions)
}
</pre>
----

==== Platform ====

'''JSON format for input and output'''
<pre>
{
"name": "MeeGo"
}
</pre>
----
==== Product ====
'''JSON format for input and output'''
<pre>
{
"platforms": [
{
"name": "MeeGo"
}
],
"name": "foobar",
"target_hardware": "n900"
}
</pre>
----

==== Product Release ====
'''JSON format for input and output'''
<pre>
{
"release_of": "test",
"baselines": [
{
"name": "1.1.80.3.20101026.1"
},
{
"name": "1.1.80.2.20101019.1"
}
],
"name": "test_weekly_w45"
}
</pre>
----

==== Tracker ====
'''JSON format for input and output'''
<pre>
{
"name": "MeeGoBugzilla",
"key2url": "http://bugs.meego.com/show_bug.cgi?id=%key%",
"url": "http://bugs.meego.com/",
"link_class": "Bug",
"filter2url": "<a ref=\"http://bugs.meego.com/show_bug.cgi?id=\\g<key>\">\\g<1></a>",
"regexp": "(\\BMC#(?P<key>\\ )\\))"
"platforms": [
{
"name": "MeeGo"
}
]
}
</pre>
----

== Importing data to REVS ==

To parse changelogs directly from source rpms, use [http://meego.gitorious.org/meego-infrastructure-tools/revs/blobs/master/bin/parse_release.py parse_release.py].

To feed parse release with .changes files, copy this script and run it inside a MeeGo mirror directory:
<pre>
#!/bin/bash

prjdir=$1

[ -d $prjdir ] || {
echo $prjdir is not a directory
exit 1
}

prj=$(basename $prjdir)
dir=$(dirname $prjdir)

mkdir $prj
cd $prj

get_changes() {
while read filename; do
rpmname=$(rpm -qp --qf %{NAME} $filename)
echo found $rpmname in $filename
rpm -qp --changelog $filename > $rpmname.changes
done
}

for area in core handset
do
find $prjdir/$area -name *src.rpm* | get_changes
done
cd ..
tar cjf $prj.tar.bz $prj
</pre>

And then bunzip and tar -x the files, then run parse release for each baseline and platform, like this script does:
<pre>
import() {
parse_release.py --force \
--platform=${platform} \
--baseline=${base} \
releases/${base}
}
platform=MeeGo
base=1.1.80.2.20101019.1
import
base=1.1.80.3.20101026.1
import
base=1.1.80.4.20101102.1
import
</pre>
== REVS License ==

REVS is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.

Release Infrastructure/IMG

2010-11-30T12:57:26Z

Iamer: /* Worker configuration */

= What is IMG =

IMG (Image Me Give) is a small python client/server application suite, its sole job is to get a kickstart file from a user, via a web interface, command line or BOSS process. Then it runs Moblin-Image-Creator, either in a host or in a VM. IMG also has the capability to run as a BOSS participant and thus be a part of a build process.

[[File:IMG.png]]

= Design =
* Asynchronous
:Fire and forget about it, check back later with the cli client for results
* Queue view with django,
:Show a nice html view of currently running or standby MIC jobs in the queue
* Submit a ks via a web interface
:Upload via a form
* No security
:No https or file filter restrictions :)
* Two ways of communication
: BOSS and pure AMQP messages

== Example workflow ==

The application consists of two parts, django frontend and the actual image creation application. Both of them are connected via AMQP, RabbitMQ is used as the server.

So an example workflow, in AMQP:

# User posts an image creation request with email address and sample kickstart file (or a package overlay), using a form or the command line client.
# Django application receives the request and imports the information on a model object and saves it.
# When the object is saved, a message is sent via a broker to the image creation app, containing the details for the creation.
## Image creation commences
## Virtual machine disk image containing mic2 is engaged (as a kvm overlay image), with ssh access.
### When a package overlay is received, a kickstart file is constructed from a ready-made template and passed to kickstart part
### When a kickstart file is received, it is passed directly to worker
## Worker starts up and connects to the virtual machine using ssh and passes the required arguments to mic2.
## Worker copies image on success from the virtual machine to well-known www-root, if there is an error, only a message about it is sent via AMQP.
## When mic2 has finished, the virtual machine is shut down by the worker.
## When image is created, the virtual machine disk image is deleted


= Worker =

The worker code is responsible of running mic2 either locally or on a virtual machine. The initialisation code takes care of creating the necessary command stubs for mic2 use for either implementations. It also handles the KVM creation command stubs.

The method "build" is where the actual business happens. It runs in two ways, as described earlier.

In KVM mode, it first creates an overlay image from the base image (not distributed) and then starts it using the overlay image as the hard disk. Virtio is used as a speed optimisation method for KVM. When a specific amount of time has passed, it will copy the kickstart and mic2 config file to the guest VM. Then it runs mic2 in the VM with parameters specified in the init method. After mic2 has run, the image is copied from the guest using scp.

In local mode, only mic2 is run on the host machine. It will create the image based on the options in the init method along with the kickstart file.

== KVM Image ==

The KVM image must be created in such a way that it has openssh and mic2, along with having ssh-key based login for root, which uses a certain [http://meego.gitorious.org/meego-infrastructure-tools/imger/blobs/master/ssh/id_rsa.pub ssh public key].

The format of the image is recommended to be qcow2, with preallocation and cluster_size=2M as the qemu-img options. This is because the worker code uses qcow2 to create the overlay, thus saving time if the image is in the same format.

If the host is using openSUSE, please enable the following things for zypper repositories:

http://download.opensuse.org/repositories/Kernel:/openSUSE-11.3/openSUSE_11.2/
http://download.opensuse.org/repositories/Virtualization/openSUSE_11.2/

And add kvm-intel to /etc/sysconfig/kernel.

= BOSS participant =

See [http://meego.gitorious.org/meego-infrastructure-tools/imger/blobs/master/src/img_boss/boss_build_image.py boss_build_image.py] for the source code.

The general design of the client is pretty much the same as the AMQP server, since they both get activated and then call the worker code to create the image, with or without KVM. The worker code is described in the section above.

More information about BOSS [[Infrastructure/BOSS]].

= BOSS Client =

See [http://meego.gitorious.org/meego-infrastructure-tools/imger/blobs/master/src/img_boss/boss_img_client.py boss_img_client.py] for example BOSS client.

== Usage ==

Usage: boss_client.py -n|--name <name> -t|--type <imagetype> -e|--email <author@email> -r|--release <release> -a|--arch <arch >-s|--submit -k <kickstart_file.ks>

boss_client.py Sends a message (poll for result later) to the BOSS, using
<kickstart.ks> as the kickstart file.

Options:
-h, --help show this help message and exit
-s, --submit Submit to BOSS, takes no options
-k KICKSTART, --kickstart=KICKSTART
Kickstart file
-t TYPE, --type=TYPE Image type
-n NAME, --name=NAME Image name
-e EMAIL, --email=EMAIL
Author email
-r RELEASE --release Image release, goes directly to mic2
-a ARCH --arch Architecture to use

= Server =

See [http://meego.gitorious.org/meego-infrastructure-tools/imger/blobs/master/src/img_amqp/build_image.py build_image.py] for example server.

This section documents the server component of IMG.

== Functionality ==

=== Consumer functions ===

*kickstarter_callback:
:Receives messages as described [#Kickstarter here], decodes JSON data to variables and creates a kickstart file with the received list of packages and then submits them to the mic-image-creator [#MoblinImageCreator here].
*mic2_callback:
:Receives messages as described [#MoblinImageCreator here], decodes JSON data to variables and runs moblin-image-creator (NB! this has to exist in /usr/bin/moblin-image-creator, will be changeable in the future) against the supplied configuration file and image type.
:The images are saved to to a configured location, needs to be changed in the python subprocess call in order to get downloadable images (eg. to a webserver media path).
:If the image creation fails (is caught with "CalledProcessError" exception in the code, this happens with nonzero exit status), a message is sent to "error_queue" queue, with the following JSON encoded dictionary:

'error': Contains the error message of the exception received from the python subprocess call.
'id': identification string of the original message sent to "image_queue" queue
'url': Contains the URL from which one can download the log

:If the image creation succeeds, a message is queued to "link_queue" queue, with the following JSON encoded dictionary:

'url': Contains the URL from which one can download the image

'id': identification string of the original message sent to "image_queue" queue

== Exchanges ==

* image_exchange, The exchange for image creationg related queues
* django_result_exchange, Exchange for results, picked up by django

== Queues ==

Queues for exchange "image_exchange":

* image_queue
:Stores messages related to Moblin Image Creator runtime.
* kickstarter_queue
:Stores messages related to kickstarter runtime.

Queues for exchange "django_result_exchange":

* status_queue
:Contains messages in the following format (JSON encoded dictionary):
** 'status': status of the image build, compulsory
** 'id': the identification string of the original message sent to kickstarter_queue, compulsory
** 'url': Contains the URL from which one can download the image
** 'log': specifies the log file to read from
** 'error': any error messages faced

= Client =

For demonstration see [http://meego.gitorious.org/meego-infrastructure-tools/imger/blobs/master/src/img_amqp/img_client.py img_client.py] for example client.

== Usage ==

img_client.py -p|--poll <id> -n|--name <name> -t|--type <imagetype> -e|--email <author@email> -r|--release -a|--arch s|--submit -k <kickstart_file.ks>"

where:
: -p|--poll <message_id>, poll the AMQP server for messages relating to the id, see -a|--async
: -t|--type <imagetype>, image type, can be of: livecd, liveusb, loop, raw, nand, mrstnand, vdi or vmdk
: -n|--name Image name
: -a|--arch Image architecture
: -r|--release Image release number
: -c|--config Configuration file to use

== Documentation ==

This client version uses AMQP as the message bus to communicate with the server. The format for the messages is described in the next part of this document.

==== Messages ====

The general message format is JSON encoded dictionary.

===== Moblin Image Creator =====

To initialize the actual image building, send a JSON formatted message as follows:

{'email':email, 'id':id, 'imagetype':imagetype, 'ksfile':ksfile, 'name': name, 'arch':arch, 'release':release}

where:
* email, email address of the submitter
* id, identification string of the original message (passed on from kickstarter process)
* imagetype, image type of values: livecd, liveusb, loop, raw, nand, mrstnand, vdi or vmdk
* ksfile, the actual kickstar file to upload (in text)
* name, the name for the image
* arch, architecture to use for the image building
* release, release numbering, $VERTICAL-$ARCH-$VARIANT like in mic2

= Django client =

=== Views ===
* submit
:If this view receives a POST request, it constructs a bound form with the data from POST array. If the form is valid it extracts the data, as specified in the forms section.
:Applying the overlay is handled in this form, in the following way.
* Get the overlay text from the form field
* use python strings split() method to split the overlay text in to a list, with a comma delimiter
** construct a kickstart file using this list and append the packages to the kickstart files package list
* queue
:This view is responsible of two things. Firstly it checks wether there are any messages in the "status_queue" message queue, if there are, then it updates the !ImageJob model object (identified by the identification string from the message) to include the ready image URL.
:Secondly, it checks for possible error messages in "status_queue" queue, if there are any, it assigns a special variable to indicate that there has been an error.
:If no messages are received, it simply redirects to a template with all the !ImageJob model objects.
* job
:Job view is responsible of providing the information about the log file. The information about the logfile is formed in the server side, such as the URL to the log file. If there is a URL, it will open the URL and read its contents and return the resulting text to the template. The template renders the log message with a textarea html widget.
*index
:Index only returns a template in which one can click a link about uploading the image.

=== URLs ===
url(r'submit/$', 'meego_img.app.views.submit', name='img-app-submit'),
url(r'queue/$', 'meego_img.app.views.queue', name='img-app-queue'),
url(r'job/(?P<msgid>\S+)$', 'meego_img.app.views.job', name='img-app-job'),
url(r'images/(?P<msgid>\S+)$', 'meego_img.app.views.download',name='img-app-download'),
=== Models ===

IMG currently has only one model, !ImageJob, here is the Django code for it:
# Create your models here.
class ImageJob(models.Model):
email = models.CharField(max_length=40)
filename = models.CharField(max_length=40)
logfile = models.CharField(max_length=50)
task_id = models.CharField(max_length=30)
imagefile = models.CharField(max_length=50)
created = models.DateTimeField(auto_now_add=True)
error = models.CharField(max_length=500)
type = models.CharField(max_length=10)
status = models.CharField(max_length=30)
def delete(self, *args, **kwargs):
if self.logfile:
if os.path.exists(self.logfile):
os.remove(self.logfile)
os.remove(self.logfile.replace("-log", ""))
print "Removed %s"%self.logfile
super(ImageJob, self).delete(*args, **kwargs)

As can be seen in the delete method, this model cleans up all image creation related files, like the kickstarter file and log file.

=== Forms ===
==== UploadFileForm ====

Contains the following fields
* email, email field
* overlay, text input field
* name, text input field
* platform, select field for the platform, parsed from the default template
* release, a text input field for release
* arch, architecture selection field
* imagetype, select field for the image type, with values: livecd, liveusb, loop, raw, nand, mrstnand, vdi, vmdk
* ksfile, the raw kickstart file

= Worker configuration =

The configuration file is to be installed in /etc/imger/img.conf and a sample file is [http://meego.gitorious.org/meego-infrastructure-tools/imger/blobs/master/img.conf here].

== Configuration file ==
base_url: A base URL that has a direct access to the base_dir via HTTP, example http://127.0.0.1
base_dir: A directory to put the finished images, example /var/www/images
num_workers: Number of workers to start, OBSOLETE, USE INITSCRIPT OR MANUALLY RUN MANY PARTICIPANTS
post_creation: Path to a script to run after the image is created
use_kvm: Wether to use a virtual machine, values are "yes" or "no"

mic_opts: Example, mic_opts = --save-kernel, --use_comps, so comma separated options

amqp_host: Host that runs BOSS
amqp_user: Username to tap into the BOSS virtual host
amqp_pwd: Password
amqp_vhost: The actual virtual host to connect to

== Virtual machine ==

Currently, IMGer can run the MIC2 jobs inside a KVM virtual machine, the only prerequisites are that image is located in /usr/share/img/base.img (configurable in the future) path and that the virtual machine has MIC2 installed and configured.

Suse studio is one way to create the images, just make sure that you have a rpm repository which contains mic2, python-xml and qemu-static-arm to make mic2 run properly on the image.

One must also confirm that the image SSH-keys in the source distribution are configured in the virtual machine and both keys exist in /usr/share/img (configurable in the future). During the runtime of IMGer, it is possible that ssh-keys inside the image may change, this is already handled by IMGer by ignoring the host key checking.

= BOSS Workitem definition =

== Input ==

The following defines the input dictionary for the participant:
"kickstart" : a kickstart file, in raw, not a path
"email": email address, not parsed so can be a dummy address
"id": a simple UUID, eg. from pythons uuid library
"type":type, from values: livecd, liveusb, loop, raw, nand, mrstnand, vdi or vmdk
"name":name, a simple string for the image name
"arch":architecture, for image
"release":release
== Output ==

The work item can contain the following dictionary entries:

Status: Status, is either "DONE" or "ERROR" in the final workitem, depending on success/failure
URL: URL to a image/log/kickstart file location on the worker
Error: An error string with details of the error, if any
Image: A direct URL to the finished image
Log: A direct URL to the log file of the mic2 output

Release Infrastructure/BOSS Process

2010-10-13T13:03:46Z

Iamer:

Release Infrastructure/BOSS Process

2010-10-06T15:00:35Z

Iamer: Created page with "== BOSS == BOSS is a workflow orchestrator, a system that can be configured to automate your workflow requirements; it interacts with the OBS and with people and systems around …"

Release Infrastructure/BOSS Launcher

2010-10-06T14:37:54Z

Iamer:

== BOSS ==

BOSS is a workflow orchestrator, a system that can be configured to automate your workflow requirements; it interacts with the OBS and with people and systems around it to apply your workflow steps. For more BOSS related general information, see [[Infrastructure/BOSS]].

=== What is BOSS Launcher ===

BOSS Launcher is the process starter in BOSS architecture. Launcher waits for an event and then start a process by sending the process definition to BOSS.

Launcher action is explained in three simple steps:

* Receive any arbitrary event.
* Process the event and extract needed information from it.
* Select and or launch a process along with the associated work item.

=== Launcher Coding Guidelines ===

BOSS launchers are written in Python language (other bindings will be available later). Following code snippets are from the generic launcher template (see chapter Example Code).
Launchers run continuously as daemons. It is convenient to write an init script to wrap and manage them.

==== Prerequisites ====
You need to install two libraries:
* AIR: An RPC library used by BOSS
* route-amqp-pyclient : The Python client library for Ruoute::AMQP
These libraries are open source and the code is published on gitorious :
* http://meego.gitorious.org/meego-infrastructure-tools/air
* http://meego.gitorious.org/meego-infrastructure-tools/ruote-amqp-pyclient

They are also packaged for OpenSuse ( and Debian soon ) on OBS https://build.opensuse.org/project/show?project=Maemo%3AMeeGo-Infra

==== Import ====
Once you have these libraries installed, you need to import them in your launcher like this :

<pre>
from RuoteAMQP.launcher import Launcher
</pre>

==== Class definition ====

The parent class Launcher defined in launcher.py of the route-amqp-pyclient package takes care of most of the details needed to create a launcher.
You just need to subclass it and use the "launch" member function.

For an explanation of processes check [[/BOSS_Process|this page]]

<pre>
process['definition'] = """
Ruote.process_definition :name => 'notification' do
sequence do
notify
end
end
"""

process['fields'] = {'msg' : 'Template', 'result' : 'Result'}
</pre>

This example shows a loop that runs every 10 seconds and checks if a certain condition is met it launches the process defined above:
<pre>
def mainLoop(self):
print "Template participant running"
while True:
# check condtion
if itHappened:
self.launch(process['definition'], process['fields'])
time.sleep(10)
</pre>

==== Registering BOSS Launcher to server ====

To register your launcher you need to do the following :

<pre>
l = templateLauncher(amqp_host=amqp_host, amqp_user=amqp_user, amqp_pass=amqp_pwd, amqp_vhost=amqp_vhost)
l.mainLoop()
</pre>

=== Launcher Example Code ===

See following launcher template for example implementation with configuration, daemonization and packaging for Debian and opensuse look here :
http://meego.gitorious.org/meego-infrastructure-tools/boss-launcher-template

Release Infrastructure/BOSS Launcher

2010-10-06T14:35:44Z

Iamer: /* Class definition */

== BOSS Launcher ==

BOSS is a workflow orchestrator, a system that can be configured to automate your workflow requirements; it interacts with the OBS and with people and systems around it to apply your workflow steps. For more BOSS related general information, see [[Infrastructure/BOSS]].

=== What is BOSS Launcher ===

BOSS Launcher is the process starter in BOSS architecture. Launcher waits for an event and then start a process by sending the process definition to BOSS.

Launcher action is explained in three simple steps:

* Receive any arbitrary event.
* Process the event and extract needed information from it.
* Select and or launch a process along with the associated work item.

=== Launcher Coding Guidelines ===

BOSS launchers are written in Python language (other bindings will be available later). Following code snippets are from the generic launcher template (see chapter Example Code).
Launchers run continuously as daemons. It is convenient to write an init script to wrap and manage them.

==== Prerequisites ====
You need to install two libraries:
* AIR: An RPC library used by BOSS
* route-amqp-pyclient : The Python client library for Ruoute::AMQP
These libraries are open source and the code is published on gitorious :
* http://meego.gitorious.org/meego-infrastructure-tools/air
* http://meego.gitorious.org/meego-infrastructure-tools/ruote-amqp-pyclient

They are also packaged for OpenSuse ( and Debian soon ) on OBS https://build.opensuse.org/project/show?project=Maemo%3AMeeGo-Infra

==== Import ====
Once you have these libraries installed, you need to import them in your launcher like this :

<pre>
from RuoteAMQP.launcher import Launcher
</pre>

==== Class definition ====

The parent class Launcher defined in launcher.py of the route-amqp-pyclient package takes care of most of the details needed to create a launcher.
You just need to subclass it and use the "launch" member function.

For an explanation of processes check [[/BOSS_Process|this page]]

<pre>
process['definition'] = """
Ruote.process_definition :name => 'notification' do
sequence do
notify
end
end
"""

process['fields'] = {'msg' : 'Template', 'result' : 'Result'}
</pre>

This example shows a loop that runs every 10 seconds and checks if a certain condition is met it launches the process defined above:
<pre>
def mainLoop(self):
print "Template participant running"
while True:
# check condtion
if itHappened:
self.launch(process['definition'], process['fields'])
time.sleep(10)
</pre>

==== Registering BOSS Launcher to server ====

To register your launcher you need to do the following :

<pre>
l = templateLauncher(amqp_host=amqp_host, amqp_user=amqp_user, amqp_pass=amqp_pwd, amqp_vhost=amqp_vhost)
l.mainLoop()
</pre>

=== Launcher Example Code ===

See following launcher template for example implementation with configuration, daemonization and packaging for Debian and opensuse look here :
http://meego.gitorious.org/meego-infrastructure-tools/boss-launcher-template

Release Infrastructure

2010-10-06T14:32:23Z

Iamer: /* Tools */

Release Infrastructure/BOSS Launcher

2010-10-06T14:30:32Z

Iamer: /* Class definition */

== BOSS Launcher ==

BOSS is a workflow orchestrator, a system that can be configured to automate your workflow requirements; it interacts with the OBS and with people and systems around it to apply your workflow steps. For more BOSS related general information, see [[Infrastructure/BOSS]].

=== What is BOSS Launcher ===

BOSS Launcher is the process starter in BOSS architecture. Launcher waits for an event and then start a process by sending the process definition to BOSS.

Launcher action is explained in three simple steps:

* Receive any arbitrary event.
* Process the event and extract needed information from it.
* Select and or launch a process along with the associated work item.

=== Launcher Coding Guidelines ===

BOSS launchers are written in Python language (other bindings will be available later). Following code snippets are from the generic launcher template (see chapter Example Code).
Launchers run continuously as daemons. It is convenient to write an init script to wrap and manage them.

==== Prerequisites ====
You need to install two libraries:
* AIR: An RPC library used by BOSS
* route-amqp-pyclient : The Python client library for Ruoute::AMQP
These libraries are open source and the code is published on gitorious :
* http://meego.gitorious.org/meego-infrastructure-tools/air
* http://meego.gitorious.org/meego-infrastructure-tools/ruote-amqp-pyclient

They are also packaged for OpenSuse ( and Debian soon ) on OBS https://build.opensuse.org/project/show?project=Maemo%3AMeeGo-Infra

==== Import ====
Once you have these libraries installed, you need to import them in your launcher like this :

<pre>
from RuoteAMQP.launcher import Launcher
</pre>

==== Class definition ====

The parent class Launcher defined in launcher.py of the route-amqp-pyclient package takes care of most of the details needed to create a launcher.
You just need to subclass it and use the "launch" member function.

For an explanation of processes check [[/BOSS_Process|this page]]

This example shows a loop that runs every 10 seconds and checks if a certain condition is met it launches a process:
<pre>
def mainLoop(self):
print "Template participant running"
while True:
# check condtion
if itHappened:
self.launch(process['definition'], process['fields'])
time.sleep(10)
</pre>

==== Registering BOSS Launcher to server ====

To register your launcher you need to do the following :

<pre>
l = templateLauncher(amqp_host=amqp_host, amqp_user=amqp_user, amqp_pass=amqp_pwd, amqp_vhost=amqp_vhost)
l.mainLoop()
</pre>

=== Launcher Example Code ===

See following launcher template for example implementation with configuration, daemonization and packaging for Debian and opensuse look here :
http://meego.gitorious.org/meego-infrastructure-tools/boss-launcher-template

Release Infrastructure/BOSS Launcher

2010-10-06T14:28:01Z

Iamer: Created page with "== BOSS Launcher == BOSS is a workflow orchestrator, a system that can be configured to automate your workflow requirements; it interacts with the OBS and with people and system…"

== BOSS Launcher ==

BOSS is a workflow orchestrator, a system that can be configured to automate your workflow requirements; it interacts with the OBS and with people and systems around it to apply your workflow steps. For more BOSS related general information, see [[Infrastructure/BOSS]].

=== What is BOSS Launcher ===

BOSS Launcher is the process starter in BOSS architecture. Launcher waits for an event and then start a process by sending the process definition to BOSS.

Launcher action is explained in three simple steps:

* Receive any arbitrary event.
* Process the event and extract needed information from it.
* Select and or launch a process along with the associated work item.

=== Launcher Coding Guidelines ===

BOSS launchers are written in Python language (other bindings will be available later). Following code snippets are from the generic launcher template (see chapter Example Code).
Launchers run continuously as daemons. It is convenient to write an init script to wrap and manage them.

==== Prerequisites ====
You need to install two libraries:
* AIR: An RPC library used by BOSS
* route-amqp-pyclient : The Python client library for Ruoute::AMQP
These libraries are open source and the code is published on gitorious :
* http://meego.gitorious.org/meego-infrastructure-tools/air
* http://meego.gitorious.org/meego-infrastructure-tools/ruote-amqp-pyclient

They are also packaged for OpenSuse ( and Debian soon ) on OBS https://build.opensuse.org/project/show?project=Maemo%3AMeeGo-Infra

==== Import ====
Once you have these libraries installed, you need to import them in your launcher like this :

<pre>
from RuoteAMQP.launcher import Launcher
</pre>

==== Class definition ====

The parent class Launcher defined in launcher.py of the route-amqp-pyclient package takes care of most of the details needed to create a launcher.
You just need to subclass it and use the "launch" member function.

For an explanation of processes check [BOSS_Process]

This example shows a loop that runs every 10 seconds and checks if a certain condition is met it launches a process:
<pre>
def mainLoop(self):
print "Template participant running"
while True:
# check condtion
if itHappened:
self.launch(process['definition'], process['fields'])
time.sleep(10)
</pre>

==== Registering BOSS Launcher to server ====

To register your launcher you need to do the following :

<pre>
l = templateLauncher(amqp_host=amqp_host, amqp_user=amqp_user, amqp_pass=amqp_pwd, amqp_vhost=amqp_vhost)
l.mainLoop()
</pre>

=== Launcher Example Code ===

See following launcher template for example implementation with configuration, daemonization and packaging for Debian and opensuse look here :
http://meego.gitorious.org/meego-infrastructure-tools/boss-launcher-template

Release Infrastructure/BOSS Participant

2010-10-06T13:38:16Z

Iamer:

== BOSS Participant ==

BOSS is a workflow orchestrator, a system that can be configured to automate your workflow requirements; it interacts with the OBS and with people and systems around it to apply your workflow steps. For more BOSS related general information, see [[Infrastructure/BOSS]].

=== What is BOSS Participant ===

BOSS Participant is the actual actor in BOSS architecture. Participant wraps other systems and services related to software development, quality assurance and integration, allowing BOSS to interact with different parties in automated way.

BOSS Participant action explained in three simple steps:

* Receive a workitem from BOSS server
* Process it, interacting with underlying (read: wrapped) system or service
* Return the filled worksheet back to BOSS server

=== Participant Coding Guidelines ===

BOSS participant is written in Python language (other bindings will be available later). Following code snippets are from Notifier Participant (see chapter Example Code).
Participants run continuously as daemons. It is convenient to write an init script to wrap and manage them.

==== Participant Naming Convention ====
The participant's name should give an indication of the function it does, usually in the form noun_verb.py

==== Prerequisites ====
You need to install two libraries:
* AIR: An RPC library used by BOSS
* route-amqp-pyclient : The Python client library for Ruoute::AMQP
These libraries are open source and the code is published on gitorious :
* http://meego.gitorious.org/meego-infrastructure-tools/air
* http://meego.gitorious.org/meego-infrastructure-tools/ruote-amqp-pyclient

They are also packaged for OpenSuse ( and Debian soon ) on OBS https://build.opensuse.org/project/show?project=Maemo%3AMeeGo-Infra

==== Import ====
Once you have these libraries installed, you need to import them in your participant like this :

<pre>
from RuoteAMQP.workitem import Workitem
from RuoteAMQP.participant import Participant
</pre>

==== Class definition ====

The parent class Participant defined in participant.py of the route-amqp-pyclient package takes care of most of the details needed to create a participant.
You just need to subclass it and define a "consume" member function.

The received work item is stored as an attribute, it contains the information that was passed from previous steps in the workflow. Usually you would read the work item, do some processing, and then store results in the work item.

For participants in a certain workflow to be able to communicate effectively, they need to follow the agreed upon "WID" or work item definition. The work item is just a JSON hash with the information stored in nested hashes and the WID is a namespace definition.

The WID should also be generic enough to allow for many participants storing their results in the work item without collisions. Namespacing is a good idea so for example a generic "id" hash could be troublesome, while "notification.id" is clearer.

That's all there is to it, the parent class will take care of putting the workitem back on the amqp queue.

<pre>
class NotifierParticipant(Participant):
def consume(self):
# look at workitem
print json.dumps(self.workitem.to_h(), indent=4)
#get some data from it
package = self.workitem.lookup("package")
# do some work
# write some results to workitem
self.workitem.set_field("accepted", "yes")
self.workitem.set_result(True)
</pre>

Here is an example of a workitem :

<pre>
{
"fei": {
"engine_id": "engine",
"expid": "0_0_2",
"sub_wfid": null,
"wfid": "20100905-binapoyachi"
},
"fields": {
"__result__": true,
"comment": null,
"desc": "Another trigger!",
"dispatched_at": "2010-09-05 13:24:44.051971 UTC",
"dst_package": "boss-launcher-srhandler",
"dst_project": "Meego",
"email": "example@example.com",
"msg": "Passes quality checks\n",
"name": "new",
"params": {
"forget": false,
"participant_options": {
"forget": false,
"queue": "notify"
},
"ref": "notify"
},
"project": "Meego",
"reason": "",
"rid": 41,
"src_package": "boss-launcher-srhandler",
"src_project": "Meego",
"src_rev": "2",
"status": "ACCEPTED",
"when": "2010-09-05T16:20:53",
"who": "iamer"
},
"participant_name": "notify"
}
</pre>
==== Registering BOSS Participant to server ====

To register your participant you need to have : a queue name, amqp host, and probably usrename and password for amqp authentication.

<pre>
if __name__ == "__main__":
print "Notifier participant running"
p = NotifierParticipant(ruote_queue="notifier", amqp_vhost="ruote-test")
p.register("notifier", {'queue':'notifier'})
p.run()
</pre>

=== Participant Example Code ===

See following participants template for example implementation with configuration, daemonization and packaging for Debian and opensuse look here :
http://meego.gitorious.org/meego-infrastructure-tools/boss-participant-template

Release Infrastructure/BOSS Participant

2010-09-23T12:31:58Z

Iamer:

== BOSS Participant ==

BOSS is a workflow orchestrator, a system that can be configured to automate your workflow requirements; it interacts with the OBS and with people and systems around it to apply your workflow steps. For more BOSS related general information, see [[Infrastructure/BOSS]].

=== What is BOSS Participant ===

BOSS Participant is the actual actor in BOSS architecture. Participant wraps other systems and services related to software development, quality assurance and integration, allowing BOSS to interact with different parties in automated way.

BOSS Participant action explained in three simple steps:

* Receive a workitem from BOSS server
* Process it, interacting with underlying (read: wrapped) system or service
* Return the filled worksheet back to BOSS server

=== Participant Coding Guidelines ===

BOSS participant is written in Python language (other bindings will be available later). Following code snippets are from Notifier Participant (see chapter Example Code).
Participants run continuously as daemons. It is convenient to write an init script to wrap and manage them.

==== Participant Naming Convention ====
The participant's name should give an indication of the function it does, usually in the form noun_verb.py

==== Prerequisites ====
You need to install two libraries:
* AIR: An RPC library used by BOSS
* route-amqp-pyclient : The Python client library for Ruoute::AMQP
These libraries are open source and the code is published on gitorious :
* http://meego.gitorious.org/meego-infrastructure-tools/air
* http://meego.gitorious.org/meego-infrastructure-tools/ruote-amqp-pyclient

They are also packaged for OpenSuse ( and Debian soon ) on OBS https://build.opensuse.org/project/show?project=Maemo%3AMeeGo-Infra

==== Import ====
Once you have these libraries installed, you need to import them in your participant like this :

<pre>
from RuoteAMQP.workitem import Workitem
from RuoteAMQP.participant import Participant
</pre>

==== Class definition ====

The parent class Participant defined in participant.py of the route-amqp-pyclient package takes care of most of the details needed to create a participant.
You just need to subclass it and define a "consume" member function.

The received work item is stored as an attribute, it contains the information that was passed from previous steps in the workflow. Usually you would read the work item, do some processing, and then store results in the work item.

For participants in a certain workflow to be able to communicate effectively, they need to follow the agreed upon "WID" or work item definition. The work item is just a JSON hash with the information stored in nested hashes and the WID is a namespace definition.

The WID should also be generic enough to allow for many participants storing their results in the work item without collisions. Namespacing is a good idea so for example a generic "id" hash could be troublesome, while "notification.id" is clearer.

That's all there is to it, the parent class will take care of putting the workitem back on the amqp queue.

<pre>
class NotifierParticipant(Participant):
def consume(self):
# look at workitem
print json.dumps(self.workitem.to_h(), indent=4)
#get some data from it
package = self.workitem.lookup("package")
# do some work
# write some results to workitem
self.workitem.set_field("accepted", "yes")
self.workitem.set_result(True)
</pre>

Here is an example of a workitem :

<pre>
{
"fei": {
"engine_id": "engine",
"expid": "0_0_2",
"sub_wfid": null,
"wfid": "20100905-binapoyachi"
},
"fields": {
"__result__": true,
"comment": null,
"desc": "Another trigger!",
"dispatched_at": "2010-09-05 13:24:44.051971 UTC",
"dst_package": "boss-launcher-srhandler",
"dst_project": "Meego",
"email": "example@example.com",
"msg": "Passes quality checks\n",
"name": "new",
"params": {
"forget": false,
"participant_options": {
"forget": false,
"queue": "notify"
},
"ref": "notify"
},
"project": "Meego",
"reason": "",
"rid": 41,
"src_package": "boss-launcher-srhandler",
"src_project": "Meego",
"src_rev": "2",
"status": "ACCEPTED",
"when": "2010-09-05T16:20:53",
"who": "iamer"
},
"participant_name": "notify"
}
</pre>
==== Registering BOSS Participant to server ====

To register your participant you need to have : a queue name, amqp host, and probably usrename and password for amqp authentication.

<pre>
if __name__ == "__main__":
print "Notifier participant running"
p = NotifierParticipant(ruote_queue="notifier", amqp_vhost="ruote-test")
p.register("notifier", {'queue':'notifier'})
p.run()
</pre>

=== Participant Example Code ===

See following participants for example implementation:
[[http://gitorious.org/locusf-home/boss-participants/blobs/master/notifier/notifier_participant.py notifier_participant.py]] [[http://gitorious.org/locusf-home/boss-participants/blobs/master/build-ks-participant/build_ks_participant.py build_ks_participant.py]]

For a more advanced template with configuration, daemonization and packaging for Debian and opensuse look here :
[[http://gitorious.org/locusf-home/boss-participants/trees/master/ots ots_participant]]

Release Infrastructure/BOSS/Code

2010-09-17T00:00:08Z

Iamer: /* Server */

The following packages are part of the Release Infrastructure team's codebase
Additional external dependencies will need to be packaged from eggs/gems to support BOSS installation.

== Maemo:MeeGo-Infra ==
=== BOSS ===
BOSS source is managed in gitorious:
* http://meego.gitorious.org/meego-infrastructure-tools/boss

There should be enough information by each package to build a new version : FIXME.

=== MeeGo-IMG ===
Aleksi... FIXME ... link to IMG page?

=== AIR ===
AIR for ruby
http://meego.gitorious.org/meego-infrastructure-tools/air
==== libair-python ====
debian package
==== libair-ruby ====
debian package
==== rubygem-air ====
suse package
Suse: Uses [[Release Infrastructure/Packaging#openSuse|update-gem tool]]
==== python-air?? ====
suse package : no suse package yet.

=== ruote ===
Should be virgin but is likely to be modified slightly and/or tagged
http://github.com/lbt/ruote

Current snapshots:
* 2.1.11.lbt2 b93c6d5f1798a07546c881e909e28a78673ba696

=== libruote-ruby ===
Debian : http://github.com/lbt/pkg-ruote

=== ruote-amqp-pyclient ===

<pre>
git checkout pkg
git merge master
#edit changelogs
git commit -am"Update to packaging"
rm /maemo/obs/opensuse/Maemo:MeeGo-Infra/route-amqp-pyclient/*.gz
git archive $TAG --format=tar --prefix=route-amqp-pyclient-$TAG/ | gzip > /maemo/obs/opensuse/Maemo:MeeGo-Infra/route-amqp-pyclient/route-amqp-pyclient-$TAG.tar.gz
cp *.spec /maemo/obs/opensuse/Maemo:MeeGo-Infra/route-amqp-pyclient/
cd /maemo/obs/opensuse/Maemo:MeeGo-Infra/route-amqp-pyclient/
osc -A suse ar
osc -A suse commit -m"Update to $TAG"
</pre>

=== rubygem-ruote ===
Suse: Uses [[Release Infrastructure/Packaging#openSuse|update-gem tool]]

=== ruote-amqp ===
Source:
http://github.com/lbt/ruote-amqp
=== libruote-amqp-ruby ===
Debian : http://github.com/lbt/pkg-ruote-amqp
=== rubygem-ruote-amqp ===
Suse: Uses [[Release Infrastructure/Packaging#openSuse|update-gem tool]]

=== python-amqlib ===
Local version of python amqp library
http://github.com/lbt/py-amqplib

=== libsetup-ruby ===
Required for Debian 5 support for ruby packaging
=== ruby-pkg-tools ===
Required for Debian 5 support for ruby packaging
=== ruby1.9.1 ===
Required for Debian 5 support for ruby packaging
=== python-support ===

== Maemo:MeeGo-Infra:Req ==
=== ejabberd ===
=== erlang ===
=== hevea ===
=== image-creator ===
=== isomd5sum ===
=== pykickstart ===
=== rabbitmq-server ===
=== rabbitmq-server-1.8.0 ===
=== squashfs-tools ===
=== yum ===

== Maemo:MeeGo-Infra:OBS ==
The team will build and package custom versions of OBS to allow for testing as packages before patches are submitted upstream.

http://gitorious.org/~lbt/opensuse/lbt-build-service

To package:

== Installation ==
W I P
=== Server ===

11.2 type:
<pre>
zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo
zypper ar http://download.opensuse.org/repositories/network:/messaging:/amqp/openSUSE_11.2/network:messaging:amqp.repo
zypper ar http://download.opensuse.org/repositories/devel:/languages:/ruby:/extensions/openSUSE_11.2/devel:languages:ruby:extensions.repo
</pre>

11.3 is similar but some build problems on suse OBS (24 July) mean you need http://download.opensuse.org/repositories/home:/lbt:/branches:/network:/messaging:/amqp/openSUSE_11.3/

Then
zypper ref
zypper in boss

To start BOSS:
rcboss start

Don't forget to follow the instructions in /usr/share/doc/packages/boss/INSTALL

Release Infrastructure/REVS

2010-09-16T22:45:07Z

Iamer: /* Installing And Setting Up */

= REVS =

So you want to get a ''grasp'' on what's going on when you release? Enter REVS – Release Engineering Visibility System (a software previously know as grasp).

REVS...
* is a data-warehouse, it's not a master data source. Updates may be lost.
* provides high speed access to all aspects of a release
* allows management reports on bugs, features, changes, packages

It integrates with, but isn't part of the [[BOSS]] workflow system.

Implementation consists of three areas:
* Data model
** Master python model
** Reporting model (django)
* Data uptake
** Sqlalchemy based; uses the revs object
** Per data-source feeds
* Reporting via Django

== Download Source Code ==

REVS is currently available in gitorious:

http://meego.gitorious.org/meego-infrastructure-tools/revs

git clone git@gitorious.org:meego-infrastructure-tools/revs.git

Installable packages (deb/rpm) can be found here :

https://build.opensuse.org/package/show?package=revs&project=Maemo%3AMeeGo-Infra

== Installing And Setting Up ==

Installing and setting up REVS is guided in the [http://meego.gitorious.org/meego-infrastructure-tools/revs/blobs/master/README README] file in the source code. It includes the following sections

* Setting REVS Up
** Prerequisites
** Running REVS
* Admin Interface and Trackers
** Admin interface
* Test Data
** Adding Data

When installing on openSuse, the following extra repos are needed :

zypper ar http://download.opensuse.org/repositories/server:/http/openSUSE_11.2/server:http.repo
zypper ar http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_11.2/devel:languages:python.repo

== Testing ==

REVS allows usage of Django test framework to run unit tests on its components. Example is provided by running

<pre>
run_tests.sh
</pre>

in the source root directory. It reports if all tests passed.

== REVS License ==

REVS is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.

Release Infrastructure/REVS

2010-09-16T22:44:49Z

Iamer: /* Installing And Setting Up */

Build Infrastructure/Community Builder/Upgrade

2010-09-08T20:49:47Z

Iamer:

# Backup /srv
# Backup /etc/
# Backup /usr/lib/obs
# Backup databases webui_production and api_production
# stop services
# upgrade packages
# merge changed config files
# perform database migrations
# start services.