Meego Wiki
Views

Release Infrastructure/BOSS/Installation

From MeeGo wiki
< Release Infrastructure | BOSS(Difference between revisions)
Jump to: navigation, search
(→Next steps)
(→Participants and SkyNET Installation)
 
(43 intermediate revisions not shown)
Line 2: Line 2:
Whilst BOSS is deployed in production, this packaged version is still
Whilst BOSS is deployed in production, this packaged version is still
-
in beta-testing and is subject to change. In particular the
+
in beta-testing and is subject to continuous change.
-
repositories are not yet finalised (eg installing from :RC).
+
However BOSS and SkyNET both install and run at a basic level on a
However BOSS and SkyNET both install and run at a basic level on a
-
clean Debian 6.0 or OpenSuse 11.4 installation.
+
clean Debian 6.0 installation (OpenSuse 11.4 has some issues - patches welcome)
= Installation =
= Installation =
Line 28: Line 27:
of the packages can be monitored at :
of the packages can be monitored at :
-
  Testing : https://build.pub.meego.com/project/monitor?project=Project:MINT:Testing
+
  Testing : https://build.pub.meego.com/project/monitor?project=Project:MINT:Testing (Current)
-
 
+
-
Release Candidate : (No release yet) https://build.pub.meego.com/project/monitor?project=Project:MINT:RC
+
  Stable : (No release yet) https://build.pub.meego.com/project/monitor?project=Project:MINT
  Stable : (No release yet) https://build.pub.meego.com/project/monitor?project=Project:MINT
Line 42: Line 39:
(new version of BOSS on opensuse 11.4)
(new version of BOSS on opensuse 11.4)
-
  zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
+
  zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo MINT
  zypper ref
  zypper ref
  zypper in boss
  zypper in boss
-
 
This will install Rabbit MQ and the boss worker daemon.
This will install Rabbit MQ and the boss worker daemon.
Line 56: Line 52:
  cat <<EOF > /etc/apt/sources.list.d/MINT.list
  cat <<EOF > /etc/apt/sources.list.d/MINT.list
-
  deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
+
  deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
  EOF
  EOF
Line 87: Line 83:
=== SUSE ===
=== SUSE ===
-
   zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
+
   zypper ar https://build.pub.meego.com/project/monitor?project=Project:MINT:Testing
   zypper ref
   zypper ref
   zypper in boss-viewer
   zypper in boss-viewer
Line 100: Line 96:
  cat <<EOF > /etc/apt/sources.list.d/MINT.list
  cat <<EOF > /etc/apt/sources.list.d/MINT.list
-
  deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
+
  deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
  EOF
  EOF
Line 106: Line 102:
  apt-get install --no-install-recommends boss-viewer
  apt-get install --no-install-recommends boss-viewer
-
This will install the boss viewer.
+
This will install (and start) the boss viewer.
Then, as usual
Then, as usual
-
  /etc/init.d/boss {start|stop|restart|force-reload|log}
+
  /etc/init.d/boss-viewer {start|stop|restart|force-reload|log}
-
== SkyNET ==
 
-
=== Overview ===
 
-
SkyNET manages long-running BOSS Participants.
+
== Participants and SkyNET Installation ==
-
It provides:
+
BOSS needs participants to do any real work. These can be installed on the main BOSS machine or on other machines in the network.
-
* daemonisation
+
Participants are managed by SkyNET (ie start/stop/logging/registration etc)
-
* start/stop control
+
-
* consistent logging
+
-
* user-owned participants
+
-
 
+
-
=== SUSE ===
+
-
zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
+
You may install SkyNET by itself or it will be pulled in when you install any of the boss-participant packages. See http://autodoc.meego.com/mint/boss-standard-workflow/ for details of participants
-
zypper ref
+
-
zypper in boss-skynet
+
-
If you installed a minimal OpenSUSE you might get a dependency error and as shown here proceed with choice 1 :
+
=== SUSE ===
-
   Problem: boss-skynet-0.1-5.1.noarch requires /usr/bin/python, but this requirement cannot be provided
+
   zypper ar https://build.pub.meego.com/project/monitor?project=Project:MINT:Testing
-
    uninstallable providers: python-base-2.7-8.4.i586[openSUSE-11.4-11.4-0]
+
  zypper ref
-
                    python-base-2.7-8.4.x86_64[openSUSE-11.4-11.4-0]
+
  zypper in boss-participant-standard-workflow boss-participant-prechecks
-
  Solution 1: deinstallation of patterns-openSUSE-minimal_base-11.4-6.9.1.x86_64
+
or
-
  Solution 2: do not install boss-skynet-0.1-5.1.noarch
+
  zypper in boss-skynet
-
  Solution 3: break boss-skynet by ignoring some of its dependencies
+
-
  Choose from above solutions by number or cancel [1/2/3/c] (c): 1
+
-
  .
+
-
  .
+
-
  .
+
=== Debian Squeeze/6.0 ===
=== Debian Squeeze/6.0 ===
  cat <<EOF > /etc/apt/sources.list.d/MINT.list
  cat <<EOF > /etc/apt/sources.list.d/MINT.list
-
  deb http://repo.pub.meego.com/Project:/MINT:/RC/Debian_6.0/ /
+
  deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
  EOF
  EOF
 +
apt-get update
 +
apt-get install --no-install-recommends boss-standard-workflow boss-participant-standard-workflow boss-participant-prechecks
 +
or for an empty installation:
  apt-get install --no-install-recommends boss-skynet
  apt-get install --no-install-recommends boss-skynet
-
=== Configuration ===
+
This will install (but not start) the boss participants.
-
The file /etc/skynet/boss.conf is used by all participants
+
-
It specifies the boss instance used by default and the credentials
+
=== SkyNET ===
-
required. Individual participants can override this.
+
To use skynet see:
-
The /etc/skynet/skynet.conf file specifies where SkyNET stores the
+
skynet help
-
running participants and where daemontools monitors them
+
-
(/var/lib/SkyNET/services/ and /var/lib/SkyNET/store/ by default)
+
-
== BOSS OBS Plugin ==
+
To see the participants :
-
The OBS plugin is designed to launch processes when a build/publish event occurs on the OBS
+
-
=== Installation ===
+
skynet list
-
The plugin is installed on the OBS backend which runs the schedulers
+
-
openSUSE 11.2
+
= Getting started and Testing the Deployment =
-
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
+
-
zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.2/Project:MINT:RC.repo
+
-
zypper ref
+
-
zypper in boss-obs-plugin
+
-
opensuse 11.4
+
Once you have installed BOSS and SkyNET you are ready to create and run some processes.
-
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
+
-
zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
+
-
zypper ref
+
-
zypper in boss-obs-plugin
+
-
=== Configuration ===
+
This is an optional step to verify that you installation was successful so far. Experienced users can skip it. In order to
-
 
+
make the test works you need to install package <code>boss-skynet</code> manually. (Using the same command you have installed boss components above, it comes from the same repository.) If you choose to skip this step the package will be installed automatically as a dependency in a later step (this is not true as of Sept. '11 because some dependencies are still missing)
-
'''You *MUST* have a participant registered as "obs_event" running before enabling this plugin or ruote will silently fill up with hundreds/thousands/millions of stalled processes.'''
+
-
 
+
-
Edit the file:
+
-
  /usr/lib/obs/server/BSConfig.pm
+
-
and define/add:
+
-
  our $notification_plugin = "notify_boss";
+
-
  our $BOSS_host="boss";  # hostname for server running BOSS AMQP
+
-
  our $BOSS_user="boss";  # AMQP username
+
-
  our $BOSS_passwd="boss"; # AMQP password (cleartext)
+
-
 
+
-
The plugin itself is installed in :
+
-
  /usr/lib/obs/server/plugins/notify_boss.pm
+
-
 
+
-
The schedulers will need to be restarted to take effect. Note there is currently an issue that
+
-
the plugin will block if the AMQP server is not available - this will halt the OBS.
+
-
 
+
-
= Getting started =
+
-
 
+
-
Once you have installed BOSS and SkyNET you are ready to create and run some processes.
+
== Watch BOSS ==
== Watch BOSS ==
Line 232: Line 185:
queues and available for use in processes:
queues and available for use in processes:
-
  skynet register -n check
+
  skynet register check
-
  skynet register -n notify
+
  skynet register notify
Line 241: Line 194:
* register : can also specify the amqp queue to use if it's not the name
* register : can also specify the amqp queue to use if it's not the name
-
== Launch a process ==
+
== Launch a process instance ==
-
With a participant ready to do some work, we're ready to launch a process:
+
With a participant ready to do some work, we're ready to launch a process instance:
  /usr/share/boss-skynet/skynet_launch
  /usr/share/boss-skynet/skynet_launch
Line 272: Line 225:
== Debugging ==
== Debugging ==
 +
TODO
 +
 +
= BOSS OBS Plugin =
 +
The OBS plugin is designed to launch processes when a build/publish event occurs on the OBS
 +
 +
== Installation ==
 +
The plugin is installed on the OBS backend which runs the schedulers
 +
 +
openSUSE 11.2
 +
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
 +
zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.2/Project:MINT:Testing.repo
 +
zypper ref
 +
zypper in boss-obs-plugin
 +
 +
opensuse 11.4
 +
zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.4/devel:languages:perl.repo
 +
zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
 +
zypper ref
 +
zypper in boss-obs-plugin
 +
 +
== Configuration ==
 +
 +
'''You *MUST* have a participant registered as "obs_event" (provided by the robogrator launcher package https://meego.gitorious.org/meego-infrastructure-tools/boss-launcher-robogrator ) running before enabling this plugin or ruote will silently fill up with hundreds/thousands/millions of stalled processes.'''
 +
 +
Edit the file:
 +
  /usr/lib/obs/server/BSConfig.pm
 +
and define/add:
 +
  our $notification_plugin = "notify_boss";
 +
  our $BOSS_host="boss";  # hostname for server running BOSS AMQP
 +
  our $BOSS_user="boss";  # AMQP username
 +
  our $BOSS_passwd="boss"; # AMQP password (cleartext)
 +
 +
You may also uncomment and set to 1:
 +
our $multiaction_notify_support = 1;
 +
(This allows osc requests containing multiple actions to be handled properly).
 +
 +
The plugin itself is installed in :
 +
  /usr/lib/obs/server/plugins/notify_boss.pm
 +
 +
The schedulers will need to be restarted to take effect. Note there is currently an issue that
 +
the plugin will block if the AMQP server is not available - this will halt the OBS.
= Next steps =
= Next steps =
-
Do something useful with your installation using already available participants and launchers : http://wiki.meego.com/Release_Infrastructure/BOSS/Participants
+
Do something useful with your installation using the [[../Standard workflow|standard workflow]] and the [[../Participants|participants and launchers]]

Latest revision as of 10:19, 23 October 2011

Contents

Overview

Whilst BOSS is deployed in production, this packaged version is still in beta-testing and is subject to continuous change.

However BOSS and SkyNET both install and run at a basic level on a clean Debian 6.0 installation (OpenSuse 11.4 has some issues - patches welcome)

Installation

Preparation

  • Proxy : On OpenSUSE you need to set the proxy in /etc/sysconfig/proxy

BOSS

Overview

BOSS provides the ruote worker and the main AMQP server.

This component is the main process dispatcher; no work happens here but all process steps are initiated from here.

Currently BOSS uses the Ruote filesystem storage system and the worker and viewer need to run on the same system. This will change when BOSS moves to Redis.

The stable and testing projects are at the community OBS, the state of the packages can be monitored at : 

Testing : https://build.pub.meego.com/project/monitor?project=Project:MINT:Testing (Current)

Stable : (No release yet) https://build.pub.meego.com/project/monitor?project=Project:MINT

SUSE

(old version of BOSS on openSUSE 11.2) 

zypper ar http://download.opensuse.org/repositories/Maemo:/MeeGo-Infra/openSUSE_11.2-plus/Maemo:MeeGo-Infra.repo
zypper ref
zypper in boss

(new version of BOSS on opensuse 11.4) 

zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo MINT
zypper ref
zypper in boss

This will install Rabbit MQ and the boss worker daemon.

To run boss: 

 rcboss start / stop

Debian Squeeze/6.0 

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss

This will install Rabbit MQ and the boss worker daemon.

Then, as usual 

/etc/init.d/boss {start|stop|restart|force-reload|log}


Default settings are found in /etc/default/boss

Configuration

BOSS is preconfigured with username boss and password boss on the boss vhost (AMQP will lose vhosts soon)

BOSS stores workitem and process data in /var/spool/boss/

BOSS-Viewer

This must be installed on the boss server when using the FS storage.

The default port is 9292, point your browser at http://127.0.0.1:9292/_ruote/ (if you are running on the same machine)

  1. FIXME: Make sure you go to /_ruote/ otherwise you will get a big scary error.

SUSE 

 zypper ar https://build.pub.meego.com/project/monitor?project=Project:MINT:Testing
 zypper ref
 zypper in boss-viewer

This will install boss-viewer.

To run boss-viewer: 

rcboss-viewer start / stop

Debian Squeeze/6.0 

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss-viewer

This will install (and start) the boss viewer.

Then, as usual 

/etc/init.d/boss-viewer {start|stop|restart|force-reload|log}


Participants and SkyNET Installation

BOSS needs participants to do any real work. These can be installed on the main BOSS machine or on other machines in the network.

Participants are managed by SkyNET (ie start/stop/logging/registration etc)

You may install SkyNET by itself or it will be pulled in when you install any of the boss-participant packages. See http://autodoc.meego.com/mint/boss-standard-workflow/ for details of participants

SUSE 

 zypper ar https://build.pub.meego.com/project/monitor?project=Project:MINT:Testing
 zypper ref
 zypper in boss-participant-standard-workflow boss-participant-prechecks

or 

 zypper in boss-skynet

Debian Squeeze/6.0 

cat <<EOF > /etc/apt/sources.list.d/MINT.list
deb http://repo.pub.meego.com/Project:/MINT:/Testing/Debian_6.0/ /
EOF

apt-get update
apt-get install --no-install-recommends boss-standard-workflow boss-participant-standard-workflow boss-participant-prechecks

or for an empty installation: 

apt-get install --no-install-recommends boss-skynet

This will install (but not start) the boss participants.

SkyNET

To use skynet see: 

skynet help

To see the participants : 

skynet list

Getting started and Testing the Deployment

Once you have installed BOSS and SkyNET you are ready to create and run some processes.

This is an optional step to verify that you installation was successful so far. Experienced users can skip it. In order to make the test works you need to install package boss-skynet manually. (Using the same command you have installed boss components above, it comes from the same repository.) If you choose to skip this step the package will be installed automatically as a dependency in a later step (this is not true as of Sept. '11 because some dependencies are still missing)

Watch BOSS

On the BOSS machine: 

/etc/init.d/boss log &

You'll see messages about the engine being alive and participant registration. Further debug output is possible here.

Setup a participant

On the skynet machine we need to setup some code as a participant: 

skynet make_participant -n check -p /usr/share/doc/python-boss-skynet/example-check-participant
skynet make_participant -n notify -p /usr/share/doc/python-boss-skynet/example-notify-participant

This creates a daemontools/participant directory structure with a symbolic link to the code for these participants.

Now we need to ensure they start to run under daemontools (and restart if there's a problem): 

skynet enable check
skynet enable notify

To watch the log output: 

skynet log check &
skynet log notify &

Now we need to tell boss that there is something listening on these queues and available for use in processes: 

skynet register check
skynet register notify


The skynet command also supports:

  • stop : shutdown the participant and don't respawn.
  • reload : stop and start once
  • register : can also specify the amqp queue to use if it's not the name

Launch a process instance

With a participant ready to do some work, we're ready to launch a process instance: 

/usr/share/boss-skynet/skynet_launch

The log output should show 

2011-03-29 00:49:21.186558500 checking for /tmp/success
2011-03-29 00:49:21.186849500 Failed to read /tmp/success
2011-03-29 00:49:21.708706500 Nothing to say

Now do: 

echo Hi there > /tmp/success

And relaunch: 

/usr/share/boss-skynet/skynet_launch

gives 

2011-03-29 00:49:31.659986500 checking for /tmp/success
2011-03-29 00:49:31.660440500 Read /tmp/success
2011-03-29 00:49:31.878978500 Send email saying : Hi there
2011-03-29 00:49:31.878980500

Of course a process can be launched from anywhere on the network using any 'trigger' and participants also run on any machine.

Debugging

TODO

BOSS OBS Plugin

The OBS plugin is designed to launch processes when a build/publish event occurs on the OBS

Installation

The plugin is installed on the OBS backend which runs the schedulers

openSUSE 11.2 

zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.2/devel:languages:perl.repo
zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.2/Project:MINT:Testing.repo
zypper ref
zypper in boss-obs-plugin

opensuse 11.4 

zypper ar http://download.opensuse.org/repositories/devel:/languages:/perl/openSUSE_11.4/devel:languages:perl.repo
zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo
zypper ref
zypper in boss-obs-plugin

Configuration

You *MUST* have a participant registered as "obs_event" (provided by the robogrator launcher package https://meego.gitorious.org/meego-infrastructure-tools/boss-launcher-robogrator ) running before enabling this plugin or ruote will silently fill up with hundreds/thousands/millions of stalled processes.

Edit the file: 

 /usr/lib/obs/server/BSConfig.pm

and define/add: 

 our $notification_plugin = "notify_boss";
 our $BOSS_host="boss";   # hostname for server running BOSS AMQP
 our $BOSS_user="boss";   # AMQP username
 our $BOSS_passwd="boss"; # AMQP password (cleartext)

You may also uncomment and set to 1: 

our $multiaction_notify_support = 1;

(This allows osc requests containing multiple actions to be handled properly).

The plugin itself is installed in : 

 /usr/lib/obs/server/plugins/notify_boss.pm

The schedulers will need to be restarted to take effect. Note there is currently an issue that the plugin will block if the AMQP server is not available - this will halt the OBS.

Next steps

Do something useful with your installation using the standard workflow and the participants and launchers

Retrieved from "http://wiki.meego.com/Release_Infrastructure/BOSS/Installation"
Personal tools