Latest revision as of 23:02, 2 November 2011

What is IMG

IMG (Image Me Give) is a small python client/server application suite, its sole job is to get a kickstart file from a user, via a web interface, command line or BOSS process. Then it runs Moblin-Image-Creator, either in a host or in a VM. IMG also has the capability to run as a BOSS participant and thus be a part of a build process.

In order to keep the docs up-to-date they are now integrated into the code and can be found here: http://autodoc.meego.com/mint/imager/

Design

• Asynchronous

Fire and forget about it, check back later with the web client for results

• Queue view with django,

Show a nice html view of currently running or standby MIC jobs in the queue

• Submit a ks via a web interface

Upload via a form

• No security

No https or file filter restrictions :)

• Two ways of communication

BOSS process or Web UI

IMG consists of two parts, django frontend and the actual image creation application. Both of them are connected via AMQP using a RabbitMQ server.

Usage

Example Use

A typical individual user would follow steps like this:

1. User posts an image creation request with email address and sample kickstart file (or a package overlay), using the web application.
2. The IMG django application receives the request and saves the information
3. A message containing the details for the required image is sent via BOSS to the image creation worker.
4. The progress and status is noted on the WebUI
5. When complete a link to the build log and the image is provided.

Behind the scenes

1. The webui launches a simple BOSS process with the selected ks
2. The process asks build_ks to modify the ks file and prepare for image building
3. Then build_image is asked to create an image
   1. The build_image worker that picks up the request commences image creation
   2. A Virtual Machine disk image containing mic2 is started (as a kvm overlay image), with ssh access.
      1. When a package overlay is received, a kickstart file is constructed from a ready-made template and passed to kickstart part
      2. When a kickstart file is received, it is passed directly to worker
4. Worker starts up and connects to the virtual machine using ssh and passes the required arguments to mic2.
   1. The worker copies the image from the virtual machine to either a local or nfs-shared www-root
   2. If there is an error, only a message about it is sent via BOSS.
5. When mic2 has finished, the virtual machine is shut down by the worker and the virtual machine disk image is deleted

Using IMG from BOSS

This is an example workflow in BOSS. The

     # Prepare a ks
     build_ks
     # Register the job with the web ui
     # image.kickstart and image.name must be set here
     request_image
     # Build image
     # image.kickstart, image.name, image.arch etc must be set by
     # this point : see http://autodoc.meego.com/mint/imager/participants/build_image.html
     build_image

TO BE DONE -->

Installation

Installation for IMG can be found at: http://autodoc.meego.com/mint/imager/install.html

Note that IMG performance can be massively improved with caching proxy, please take a look at Setting up IMG PROXY.

Worker / img-core

The worker code is provided by img-core and is responsible for running mic2 either locally or on a virtual machine. The initialisation code takes care of creating the necessary command stubs for mic2 use for either implementations. It also handles the KVM creation command stubs.

In KVM mode, it first creates an overlay image from the base image (not distributed) and then starts it using the overlay image as the hard disk. Virtio is used as a speed optimisation method for KVM. When a specific amount of time has passed, it will copy the kickstart and mic2 config file to the guest VM. Then it runs mic2 in the VM with parameters specified in the init method. After mic2 has run, the image is copied from the guest using scp.

In local mode, only mic2 is run on the host machine. It will create the image based on the options in the init method along with the kickstart file.

A worker needs a queueing mechanism to accept jobs. The primary solution is expected to be the BOSS participant (img-boss) but there is a plain amqp client too (img-amqp).

BOSS participant / img-boss

The general design of the client is pretty much the same as the AMQP server, since they both get activated and then call the worker code to create the image, with or without KVM. The worker code is described in the section above.

More information about BOSS Infrastructure/BOSS.

See boss_build_image.py for the source code.

BOSS Client / img-amqp

See boss_img_client.py for example BOSS client.

Usage

(Is this the img-amqp usage? lbt 27/Dec)

Usage: boss_client.py -n|--name <name> -t|--type <imagetype> -e|--email <author@email> -r|--release <release> -a|--arch <arch >-s|--submit -k <kickstart_file.ks>

boss_client.py Sends a message (poll for result later) to the BOSS, using
<kickstart.ks>  as the kickstart file.

Options:
 -h, --help            show this help message and exit
 -s, --submit          Submit to BOSS, takes no options
 -k KICKSTART, --kickstart=KICKSTART
                       Kickstart file
 -t TYPE, --type=TYPE  Image type
 -n NAME, --name=NAME  Image name
 -e EMAIL, --email=EMAIL
                       Author email
 -r RELEASE --release  Image release, goes directly to mic2
 -a ARCH --arch        Architecture to use

BOSS OTS Client

BOSS OTS Client is a AMQP launcher enabling communication with OTS Framework. BOSS OTS Client is used to send image URL to OTS and commence image testing.

Download from: boss_ots_client.py. Usage boss_ots_client.py --help.

Configuration

See defaultconf section in the script or provide standalone configuration file.

Server

See build_image.py for example server.

This section documents the server component of IMG.

Functionality

Consumer functions

• kickstarter_callback:

Receives messages as described [#Kickstarter here], decodes JSON data to variables and creates a kickstart file with the received list of packages and then submits them to the mic-image-creator [#MoblinImageCreator here].

• mic2_callback:

Receives messages as described [#MoblinImageCreator here], decodes JSON data to variables and runs moblin-image-creator (NB! this has to exist in /usr/bin/moblin-image-creator, will be changeable in the future) against the supplied configuration file and image type.

The images are saved to to a configured location, needs to be changed in the python subprocess call in order to get downloadable images (eg. to a webserver media path).

If the image creation fails (is caught with "CalledProcessError" exception in the code, this happens with nonzero exit status), a message is sent to "error_queue" queue, with the following JSON encoded dictionary:

   'error': Contains the error message of the exception received from the python subprocess call.
   'id': identification string of the original message sent to "image_queue" queue 
   'url': Contains the URL from which one can download the log

If the image creation succeeds, a message is queued to "link_queue" queue, with the following JSON encoded dictionary:

   'url': Contains the URL from which one can download the image

   'id': identification string of the original message sent to "image_queue" queue 

Exchanges

• image_exchange, The exchange for image creationg related queues
• django_result_exchange, Exchange for results, picked up by django

Queues

Queues for exchange "image_exchange":

• image_queue

Stores messages related to Moblin Image Creator runtime.

• kickstarter_queue

Stores messages related to kickstarter runtime.

Queues for exchange "django_result_exchange":

• status_queue

Contains messages in the following format (JSON encoded dictionary):

• • 'status': status of the image build, compulsory
  • 'id': the identification string of the original message sent to kickstarter_queue, compulsory
  • 'url': Contains the URL from which one can download the image
  • 'log': specifies the log file to read from
  • 'error': any error messages faced

Client

For demonstration see img_client.py for example client.

Usage

img_client.py -p|--poll <id> -n|--name <name> -t|--type <imagetype> -e|--email <author@email> -r|--release -a|--arch s|--submit -k <kickstart_file.ks>"

where:

-p|--poll <message_id>, poll the AMQP server for messages relating to the id, see -a|--async

-t|--type <imagetype>, image type, can be of: livecd, liveusb, loop, raw, nand, mrstnand, vdi or vmdk

-n|--name Image name

-a|--arch Image architecture

-r|--release Image release number

-c|--config Configuration file to use

Documentation

This client version uses AMQP as the message bus to communicate with the server. The format for the messages is described in the next part of this document.

Messages

The general message format is JSON encoded dictionary.

Moblin Image Creator

To initialize the actual image building, send a JSON formatted message as follows:

{'email':email, 'id':id, 'imagetype':imagetype, 'ksfile':ksfile, 'name': name, 'arch':arch, 'release':release}

where:

• email, email address of the submitter
• id, identification string of the original message (passed on from kickstarter process)
• imagetype, image type of values: livecd, liveusb, loop, raw, nand, mrstnand, vdi or vmdk
• ksfile, the actual kickstar file to upload (in text)
• name, the name for the image
• arch, architecture to use for the image building
• release, release numbering, $VERTICAL-$ARCH-$VARIANT like in mic2

Django client / img-web

Views

• submit

If this view receives a POST request, it constructs a bound form with the data from POST array. If the form is valid it extracts the data, as specified in the forms section.

Applying the overlay is handled in this form, in the following way.

• Get the overlay text from the form field
• use python strings split() method to split the overlay text in to a list, with a comma delimiter
  • construct a kickstart file using this list and append the packages to the kickstart files package list
• queue

This view is responsible of two things. Firstly it checks wether there are any messages in the "status_queue" message queue, if there are, then it updates the !ImageJob model object (identified by the identification string from the message) to include the ready image URL.

Secondly, it checks for possible error messages in "status_queue" queue, if there are any, it assigns a special variable to indicate that there has been an error.

If no messages are received, it simply redirects to a template with all the !ImageJob model objects.

• job

Job view is responsible of providing the information about the log file. The information about the logfile is formed in the server side, such as the URL to the log file. If there is a URL, it will open the URL and read its contents and return the resulting text to the template. The template renders the log message with a textarea html widget.

• index

Index only returns a template in which one can click a link about uploading the image.

URLs

   url(r'submit/$', 'meego_img.app.views.submit', name='img-app-submit'), 
   url(r'queue/$', 'meego_img.app.views.queue', name='img-app-queue'),    
   url(r'job/(?P<msgid>\S+)$', 'meego_img.app.views.job', name='img-app-job'),       
   url(r'images/(?P<msgid>\S+)$', 'meego_img.app.views.download',name='img-app-download'),

Models

IMG currently has only one model, !ImageJob, here is the Django code for it:

# Create your models here.
class ImageJob(models.Model):    
   email = models.CharField(max_length=40)
   filename = models.CharField(max_length=40)
   logfile = models.CharField(max_length=50)
   task_id = models.CharField(max_length=30)
   imagefile = models.CharField(max_length=50)    
   created = models.DateTimeField(auto_now_add=True)
   error = models.CharField(max_length=500)
   type = models.CharField(max_length=10)
   status = models.CharField(max_length=30)
   def delete(self, *args, **kwargs): 
       if self.logfile:
           if os.path.exists(self.logfile):
               os.remove(self.logfile)
               os.remove(self.logfile.replace("-log", ""))
               print "Removed %s"%self.logfile
       super(ImageJob, self).delete(*args, **kwargs)

As can be seen in the delete method, this model cleans up all image creation related files, like the kickstarter file and log file.

Forms

UploadFileForm

Contains the following fields

• email, email field
• overlay, text input field
• name, text input field
• platform, select field for the platform, parsed from the default template
• release, a text input field for release
• arch, architecture selection field
• imagetype, select field for the image type, with values: livecd, liveusb, loop, raw, nand, mrstnand, vdi, vmdk
• ksfile, the raw kickstart file

Worker configuration

The configuration file is to be installed in /etc/imger/img.conf and a sample file is here.

Configuration file

base_url: A base URL that has a direct access to the base_dir via HTTP, example http://127.0.0.1
base_dir: A directory to put the finished images, example /var/www/images
num_workers: Number of workers to start, OBSOLETE, USE INITSCRIPT OR MANUALLY RUN MANY PARTICIPANTS
post_creation: Path to a script to run after the image is created
use_kvm: Whether to use a virtual machine, values are "yes" or "no"

mic_opts: Example, mic_opts = --save-kernel, --use_comps, so comma separated options

amqp_host: Host that runs BOSS
amqp_user: Username to tap into the BOSS virtual host
amqp_pwd: Password
amqp_vhost: The actual virtual host to connect to

KVM Image

The KVM image must be created in such a way that it has openssh and mic2, along with having ssh-key based login for root, which uses a certain ssh public key.

The format of the image is recommended to be qcow2, with preallocation and cluster_size=2M as the qemu-img options. This is because the worker code uses qcow2 to create the overlay, thus saving time if the image is in the same format.

If the host is using openSUSE, please enable the following things for zypper repositories:

http://download.opensuse.org/repositories/Kernel:/openSUSE-11.3/openSUSE_11.2/ http://download.opensuse.org/repositories/Virtualization/openSUSE_11.2/

And add kvm-intel to /etc/sysconfig/kernel.

Virtual machine

Currently, IMGer can run the MIC2 jobs inside a KVM virtual machine, the only prerequisites are that image is located in /usr/share/img/base.img (configurable in the future) path and that the virtual machine has MIC2 installed and configured.

Suse studio is one way to create the images, just make sure that you have a rpm repository which contains mic2, python-xml and qemu-static-arm to make mic2 run properly on the image.

One must also confirm that the image SSH-keys in the source distribution are configured in the virtual machine and both keys exist in /usr/share/img (configurable in the future). During the runtime of IMGer, it is possible that ssh-keys inside the image may change, this is already handled by IMGer by ignoring the host key checking.

BOSS Integration

One valuable use of IMG is to build an image prior to accepting a package into Trunk. Just verifying that a package doesn't prevent building an image is useful; but beyond this, the image can be sent to automated test system and the results used to determine acceptance.

IMG provides 2 participants that can be used in a BOSS process:

• build_ks : Modifies a template kickstart file to add specific packages or groups
• build_image : Builds an image defined by a kickstart file

build_ks

This participant is used to customise a kickstart file to add specific packages or groups; for instance to add a package to a minimal kickstart for testing purposes.

Configuration

The build_ks participant is configured at the OS level.

This allows the repository server (reposerver) and the kickstart template store (ksstore) to be specified.

Input

The participant documentation defines the input for the participant.

Output

The image.kickstart field is populated with the modified kickstart file.

build_image

The main IMG participant. It takes a kickstart file and some values and returns URLs which point to the image and log.

Configuration

The build_image participant is configured at the OS level. The config file is documented.

It mainly determines where images are stored on the worker and the base url of the webserver that has been setup to serve them.

Input

The following defines the input dictionary for the participant:

"kickstart" : contents of a kickstart file (ie not a path)
"email": email address, not parsed so can be a dummy address
"id": a simple UUID, eg. from pythons uuid library
"type": type, from values: livecd, liveusb, loop, raw, nand, mrstnand, vdi or vmdk
"name": name, a simple string for the image name
"arch": architecture, for image
"release": release

Output

The work item can contain the following dictionary entries:

Status: Status, is either "DONE" or "ERROR" in the final workitem, depending on success/failure
URL: URL to a image/log/kickstart file location on the worker
Error: An error string with details of the error, if any
Image: A direct URL to the finished image
Log: A direct URL to the log file of the mic2 output

Sample Workflow

The following simple workflow can be attached to the REPO_PUBLISHED OBS event like so:

ln -s /srv/BOSS/processes/build_IMG  /srv/BOSS/processes/${PROJECT//:/\/}/REPO_PUBLISHED

Each time $PROJECT is published the process will run.

Note that this is a simple process that does not handle situations like locking the project to ensure changes are not made during the image build process.

Ruote.process_definition :name => 'Project_Trunk_PUB' do
# This process uses:
# img_boss package:
# * build_ks
# * build_image
# optionally from: boss-participant-obsticket
# * obsticket

  sequence do

    set 'archs' => ['i586']
    # The name of the build target (eg 'standard')
    set 'targetrepo' => 'Project_Trunk'
    set 'debug_dump' => 'true'
    set 'image' => {'image_id' => 'latest',
                    'image_type' => 'livecd',
                    'arch' => 'i586',
                    'name' => 'latest',
                    'ksfile' => 'meego.ks'
                    }

    echo "Start Project_PUB  :  ${ev.state} in ${project}"

    # We could lock the Project to ensure that no SR is accepted whilst the image is building
#    with_OBS_ticket do
      build_ks
      build_image
#    end

    # We are not doing acceptance based on the image so drop the lock and test it
#    test_image
  end

# This subprocess shows how a group of actions can be wrapped inside
# other process steps; in this case reserving the use of a build project
  define 'with_OBS_ticket' do
    sequence do
      obsticket :action => 'get', :lock_project => '${test_project}', :debug_dump => 'TRUE'
      apply
      obsticket :action => 'release', :lock_project => '${test_project}', :debug_dump => 'TRUE'
    end
  end

end
<pre>