= MINT =
MINT (MeeGo Integration) is a set of integrated tools, that was created with the goal of creating a CI system around OBS for enabling vendors to build meego products.
It provides a business process automation system, interacting with many different systems such as bugzilla and OBS to facilitate product building.

== Overview ==
Describe the overall release and build infrastructure.
=== Features ===
* Covers all the activities involved in continuous integration
** Error and feature management, through bugzilla integration (www.biugzilla.org)
** On device testing, through integration with OTS (http://wiki.meego.com/Quality/QA-tools/OTS/About)
** Creation of installable/flashable images
** Building through integration with OBS (http://en.opensuse.org//Portal:Build_Service)
** publishing and notifictaions
** Reporting on releases progress
* Distributed architecture means no single point of failure and better resource management
* Massive parallelism, all the components of the system run multiple parallel jobs serving several parallel workflows
* Parallel workflows, means you can run many different workflows servicing multiple products
* Simple workflow description language
* Reliable
** based on mature, production level opensource projects
** Reliable messaging using the AMQP standard, means no events will be lost
** Workflow management based on ruote, an enterprise grade workflow engine
* Robust
** Live migration of workflows, means you don't need to bring your system down for changing your workflow. Simply deploy your workflow and it gets loaded without disruption to currently running jobs
* Extensible. The system is designed to not have any hardcoded funtions, but rather have each operation/function is written in a plugin (participant)

== Tools ==
* [[/BOSS|Build Orchestration BOSS]]
** [[/BOSS_Launcher|Launcher documentation]]
** [[/BOSS_Process|Process documentation]]
** [[/BOSS_Testing|Testing]]
* [[/REVS|Release reporting REVS]]
* [[/IMG|Image Creation IMG]]
* [[/BOSS_Participant|Participant documentation]]
* Build System OBS

== [[/Packaging|Release team packaging information]] ==

== misc. ==
* [[/devroot|devroot]]

== Team ==

==New BOSS SkyNET==
* [[/BOSS/Installation|BOSS and Skynet]]
* [[/BOSS/Participants|Participants]]

Release Infrastructure/BOSS/Roadmap

2011-07-25T10:18:03Z

Release Infrastructure/IMG

2010-07-20T09:04:09Z

Informatiq: Created page with '= What is IMG = IMG (Image Me Give) is a small python client/server application suite, its sole job is to get a POST'd kickstart file from a user and then run Moblin-Image-Creat…'

= What is IMG =

IMG (Image Me Give) is a small python client/server application suite, its sole job is to get a POST'd kickstart file from a user and then run Moblin-Image-Creator. It is also possible to use kickstarter yaml files to create a tailor-made image or using the default selection of yaml templates.

= Design =
* Asynchronous
:Fire and forget about it until an email about completion is sent. (FUTURE)
* Queue of mic jobs
:Queue mic jobs running in threads
* Queue view,
:Show a nice html view of currently running or standby MIC jobs in the queues
* Submit/cancel via .ks
:Upload via a form
* Email response success/fail
:Email on success
* No security
:No https or file filter restrictions :)

== Example workflow ==

The application consists of two parts, django frontend and the actual image creation application. Both of them are connected via a message broker, RabbitMQ is used.

So an example workflow:

# User posts an image creation request with email address and sample kickstart file (or yaml templates for kickstarter), using a form or the command line client.
# Django application receives the request and imports the information on a model object and saves it.
# When the object is saved, a message is sent via a broker to the image creation app, containing the details for the creation.
## Image creation commences
## Virtual machine disk image containing mic2 and kickstarter is engaged (as a copy), with ssh access and a special mount to a location in which the created images are held.
### When a yaml overlay is received, it is passed via kickstarter and then to mic2
### When a kickstart file is received, it is passed directly to mic2
## Server connects to the virtual machine using ssh and passes the required commands
## When mic2 has finished, the virtual machine shuts down.
## When image is created, the virtual machine disk image is deleted

= Overlays =

Overlays are special to IMG, as they provide a possibility to modify an existing YAML templates. Examples include package adding/removal, configuration changes etc.

The draft for overlay files:
gcc,@MeeGo Netbook Desktop,-google-chrome

So comma separated list of packages and groups. Groups have to be prepended with @-symbol.

When applying the overlay to a template like this:

NETBOOK:
PartSize: 1900
BootloaderAppend: "quiet"
BootloaderTimeout: 0
PostScripts:
- cleanup
- flash
Groups:
- X for Netbooks
- Virtual Machine Support
- Printing
- Games
Kernel: kernel-netbook
Repos:
- adobe
- meego-core-1.0
- meego-netbook-1.0
- meego-extra-1.0
PostScripts:
- prelink
ExtraPackages:
- gdb
- flash-plugin
- adobe-release

It becomes:

NETBOOK:
PartSize: 1900
BootloaderAppend: "quiet"
BootloaderTimeout: 0
PostScripts:
- cleanup
- flash
Groups:
- X for Netbooks
- Virtual Machine Support
- Printing
- Games
- MeeGo Netbook Desktop
Kernel: kernel-netbook
Repos:
- adobe
- meego-core-1.0
- meego-netbook-1.0
- meego-extra-1.0
- internal
PostScripts:
- prelink
ExtraPackages:
- gdb
- flash-plugin
- adobe-release
- gcc
- -google-chrome

= Client =

For demonstration see [http://meego.gitorious.org/meego-infrastructure-tools/imger/blobs/master/src/meego_img/client.py client.py] for example client.

== Django client ==

=== Views ===
* submit[[BR]]
:If this views receives a POST request, it constructs a bound form with the data from POST array. If the form is valid it extracts the data, as specified in the forms section.
:Applying the overlay is handled in this form, in the following way.
* Get the overlay text from the form field
* use python strings split() method to split the overlay text in to a list, with a comma delimiter
*parse the template configurations.yaml
**append the overlay list to the platform selected via the form, to the !ExtraPackages element
:After the overlay is applied, the resulting YAML template is passed to the kickstarter via a message, see [#Kickstarter]
* queue
:This view is responsible of two things. Firstly it wether there are any ready images in the "link_queue" message queue, if there are, then update the !ImageJob model object (identified by the identification string from the message) to include the ready image URL.
:Secondly, it checks for possible error messages in "error_queue" queue, if there are any, it assigns a special variable to indicate that there has been an error.
:If no messages are received, it simply redirects to a template with all the !ImageJob model objects.
* job
:Job view is responsible of providing the information about the log file. The information about the logfile is formed in the server side, such as the file name. If there is a filename, it will open the file and read it and return the resulting text to the template. The template renders the log message with a textarea html widget.
*index
:Index only returns a template in which one can click a link about uploading the image.

=== URLs ===
url(r'submit/$', 'meego_img.app.views.submit', name='img-app-submit'),
url(r'queue/$', 'meego_img.app.views.queue', name='img-app-queue'),
#(r'$', 'meego_img.app.views.index'),
url(r'job/(?P<msgid>\S+)$', 'meego_img.app.views.job', name='img-app-job'),
url(r'clear/$', 'meego_img.app.views.clear', name='img-app-clear'),
url(r'images/(?P<msgid>\S+)$', 'meego_img.app.views.download',name='img-app-download'),
=== Models ===

IMG currently has only one model, !ImageJob, here is the Django code for it:
# Create your models here.
class ImageJob(models.Model):
email = models.CharField(max_length=40)
filename = models.CharField(max_length=40)
logfile = models.CharField(max_length=50)
task_id = models.CharField(max_length=30)
imagefile = models.CharField(max_length=50)
created = models.DateTimeField(auto_now_add=True)
error = models.CharField(max_length=500)
type = models.CharField(max_length=10)
status = models.CharField(max_length=30)
def delete(self, *args, **kwargs):
if self.logfile:
if os.path.exists(self.logfile):
os.remove(self.logfile)
os.remove(self.logfile.replace("-log", ""))
print "Removed %s"%self.logfile
super(ImageJob, self).delete(*args, **kwargs)

As can be seen in the delete method, this model cleans up all image creation related files, like the kickstarter file and log file.

=== Forms ===
==== !UploadFileForm ====

Contains the following fields
* email, email field
* overlay, text input field
* platform, select field for the platform, parsed from the default template
* imagetype, select field for the image type, with values: livecd, liveusb, loop, raw, nand, mrstnand, vdi, vmdk
* ksfile, the raw kickstart file

= Server =

See [http://meego.gitorious.org/meego-infrastructure-tools/imger/blobs/master/src/meego_img/image_creator.py image_creator.py] for example server.

This section documents the server component of IMG.

== Functionality ==

=== Consumer functions ===

*kickstarter_callback:
:Receives messages as described [#Kickstarter here], decodes JSON data to variables and runs kickstarter. After kickstarter finishes it sends a message to "image_queue" queue, as specified [#MoblinImageCreator here].
*mic2_callback:
:Receives messages as described [#MoblinImageCreator here], decodes JSON data to variables and runs moblin-image-creator (NB! this has to exist in /usr/bin/moblin-image-creator, will be changeable in the future) against the supplied configuration file and image type.
:The images are saved to to a configured location, needs to be changed in the python subprocess call in order to get downloadable images (eg. to a webserver media path).
:If the image creation fails (is caught with "!CalledProcessError" exception in the code, this happens with nonzero exit status), a message is sent to "error_queue" queue, with the following JSON encoded dictionary:

'error': Contains the error message of the exception received from the python subprocess call.
'id': identification string of the original message sent to "image_queue" queue
'url': Contains the URL from which one can download the log

:Also in the case of an error, the following text is appended at the end of the log file: "IMG FAILED MISERABLY IN CREATING THE IMAGE!\n", along with the same error message as queued to the "error_queue" queue.

:If the image creation succeeds, a message is queued to "link_queue" queue, with the following JSON encoded dictionary:

'url': Contains the URL from which one can download the image

'id': identification string of the original message sent to "image_queue" queue

== Exchanges ==

* image_exchange, The exchange for image creationg related queues
* django_result_exchange, Exchange for results, picked up by django

== Queues ==

Queues for exchange "image_exchange":

* image_queue
:Stores messages related to Moblin Image Creator runtime, see [#MoblinImageCreator Moblin Image Creator] , for the message format
* kickstarter_queue
:Stores messages related to kickstarter runtime, see [#Kickstarter kickstarter], for the message format

Queues for exchange "django_result_exchange":

* link_queue
:Contains messages in the following format (JSON encoded dictionary):
** 'id': the identification string of the original message sent to kickstarter_queue
*** 'url': Contains the URL from which one can download the image
* result_queue
:Contains messages in the following format (JSON encoded dictionary):
** 'logfile': filename of the logfile to read
*** 'id': identification string of the message
* error_queue
:Contains messages in the following format (JSON encoded dictionary):
** 'error': Error message, this message gets recorded in to the log file.
** 'url': Contains the URL from which one can download the log which describes the error.
*** 'id': identification string of the message

= Messages =

The general message format is JSON encoded dictionary.

== Kickstarter ==

To initialize kickstarter, send a JSON formatted message as follows:

{'config':config, 'email':'email', 'imagetype':'imagetype', 'id': id}

where:
* Config, is a yaml formatted configuration for kickstarter
* Email, email address
* Imagetype, image type of values: livecd, liveusb, loop, raw, nand, mrstnand, vdi or vmdk
* Id, a unique id for this job

== Moblin Image Creator ==

To initialize the actual image building, send a JSON formatted message as follows:

{'email':email, 'id':id, 'imagetype':imagetype, 'ksfile':ksfile}

where:
* email, email address of the submitter
* id, identification string of the original message (passed on from kickstarter process)
* imagetype, image type of values: livecd, liveusb, loop, raw, nand, mrstnand, vdi or vmdk
* ksfile, the actual kickstar file to upload (in text)

Release Infrastructure/REVS

2010-07-20T09:03:54Z

2010-07-20T08:39:00Z

Informatiq: moved BOSS/AIR to Infrastructure/BOSS/AIR

#REDIRECT [[Infrastructure/BOSS/AIR]]