Revision as of 11:25, 22 July 2010

What is IMG

IMG (Image Me Give) is a small python client/server application suite, its sole job is to get a POST'd kickstart file from a user and then run Moblin-Image-Creator. It is also possible to use kickstarter yaml files to create a tailor-made image or using the default selection of yaml templates.

Design

• Asynchronous

Fire and forget about it, check back later with the cli client for results

• Queue view with django,

Show a nice html view of currently running or standby MIC jobs in the queues

• Submit/cancel via .ks

Upload via a form

• No security

No https or file filter restrictions :)

Example workflow

The application consists of two parts, django frontend and the actual image creation application. Both of them are connected via AMQP, RabbitMQ is used as the server.

So an example workflow:

1. User posts an image creation request with email address and sample kickstart file (or yaml templates for kickstarter), using a form or the command line client.
2. Django application receives the request and imports the information on a model object and saves it.
3. When the object is saved, a message is sent via a broker to the image creation app, containing the details for the creation.
   1. Image creation commences
   2. Virtual machine disk image containing mic2 and kickstarter is engaged (as a kvm overlay image), with ssh access.
      1. When a yaml overlay is received, it is passed via kickstarter and then to worker
      2. When a kickstart file is received, it is passed directly to worker
   3. Worker starts up and connects to the virtual machine using ssh and passes the required arguments to mic2.
   4. Worker copies image on success from the virtual machine to well-known www-root, if there is an error, only a message about it is sent via AMQP.
   5. When mic2 has finished, the virtual machine is shut down by the worker.
   6. When image is created, the virtual machine disk image is deleted

Client

For demonstration see client.py for example client.

Usage

client.py -p|--poll <message_id> -t|--type <imagetype> -a|--async  <kickstart_file.ks>

where:

-p|--poll <message_id>, poll the AMQP server for messages relating to the id, see -a|--async

-t|--type <imagetype>, image type, can be of: livecd, liveusb, loop, raw, nand, mrstnand, vdi or vmdk

-a|--async <kickstart_file.ks>, the kickstart file to submit

BOSS Client

See boss_client.py for example BOSS client.

Usage

boss_client.py <kickstart.ks>

Django client

Views

• submit

If this view receives a POST request, it constructs a bound form with the data from POST array. If the form is valid it extracts the data, as specified in the forms section.

Applying the overlay is handled in this form, in the following way.

• Get the overlay text from the form field
• use python strings split() method to split the overlay text in to a list, with a comma delimiter
• parse the template configurations.yaml
  • append the overlay list to the platform selected via the form, to the !ExtraPackages element

After the overlay is applied, the resulting YAML template is passed to the kickstarter via a AMQP message.

• queue

This view is responsible of two things. Firstly it checks wether there are any messages in the "status_queue" message queue, if there are, then it updates the !ImageJob model object (identified by the identification string from the message) to include the ready image URL.

Secondly, it checks for possible error messages in "status_queue" queue, if there are any, it assigns a special variable to indicate that there has been an error.

If no messages are received, it simply redirects to a template with all the !ImageJob model objects.

• job

Job view is responsible of providing the information about the log file. The information about the logfile is formed in the server side, such as the URL to the log file. If there is a URL, it will open the URL and read its contents and return the resulting text to the template. The template renders the log message with a textarea html widget.

• index

Index only returns a template in which one can click a link about uploading the image.

URLs

   url(r'submit/$', 'meego_img.app.views.submit', name='img-app-submit'), 
   url(r'queue/$', 'meego_img.app.views.queue', name='img-app-queue'),    
   url(r'job/(?P<msgid>\S+)$', 'meego_img.app.views.job', name='img-app-job'),       
   url(r'images/(?P<msgid>\S+)$', 'meego_img.app.views.download',name='img-app-download'),

Models

IMG currently has only one model, !ImageJob, here is the Django code for it:

# Create your models here.
class ImageJob(models.Model):    
   email = models.CharField(max_length=40)
   filename = models.CharField(max_length=40)
   logfile = models.CharField(max_length=50)
   task_id = models.CharField(max_length=30)
   imagefile = models.CharField(max_length=50)    
   created = models.DateTimeField(auto_now_add=True)
   error = models.CharField(max_length=500)
   type = models.CharField(max_length=10)
   status = models.CharField(max_length=30)
   def delete(self, *args, **kwargs): 
       if self.logfile:
           if os.path.exists(self.logfile):
               os.remove(self.logfile)
               os.remove(self.logfile.replace("-log", ""))
               print "Removed %s"%self.logfile
       super(ImageJob, self).delete(*args, **kwargs)

As can be seen in the delete method, this model cleans up all image creation related files, like the kickstarter file and log file.

Forms

 !UploadFileForm

Contains the following fields

• email, email field
• overlay, text input field
• platform, select field for the platform, parsed from the default template
• imagetype, select field for the image type, with values: livecd, liveusb, loop, raw, nand, mrstnand, vdi, vmdk
• ksfile, the raw kickstart file

Server

See image_creator.py for example server.

This section documents the server component of IMG.

Functionality

Consumer functions

• kickstarter_callback:

Receives messages as described [#Kickstarter here], decodes JSON data to variables and runs kickstarter. After kickstarter finishes it sends a message to "image_queue" queue, as specified [#MoblinImageCreator here].

• mic2_callback:

Receives messages as described [#MoblinImageCreator here], decodes JSON data to variables and runs moblin-image-creator (NB! this has to exist in /usr/bin/moblin-image-creator, will be changeable in the future) against the supplied configuration file and image type.

The images are saved to to a configured location, needs to be changed in the python subprocess call in order to get downloadable images (eg. to a webserver media path).

If the image creation fails (is caught with "!CalledProcessError" exception in the code, this happens with nonzero exit status), a message is sent to "error_queue" queue, with the following JSON encoded dictionary:

   'error': Contains the error message of the exception received from the python subprocess call.
   'id': identification string of the original message sent to "image_queue" queue 
   'url': Contains the URL from which one can download the log

Also in the case of an error, the following text is appended at the end of the log file: "IMG FAILED MISERABLY IN CREATING THE IMAGE!\n", along with the same error message as queued to the "error_queue" queue.

If the image creation succeeds, a message is queued to "link_queue" queue, with the following JSON encoded dictionary:

   'url': Contains the URL from which one can download the image

   'id': identification string of the original message sent to "image_queue" queue 

Exchanges

• image_exchange, The exchange for image creationg related queues
• django_result_exchange, Exchange for results, picked up by django

Queues

Queues for exchange "image_exchange":

• image_queue

Stores messages related to Moblin Image Creator runtime.

• kickstarter_queue

Stores messages related to kickstarter runtime.

Queues for exchange "django_result_exchange":

• status_queue

Contains messages in the following format (JSON encoded dictionary):

• • 'status': status of the image build, compulsory
  • 'id': the identification string of the original message sent to kickstarter_queue, compulsory
  • 'url': Contains the URL from which one can download the image
  • 'log': specifies the log file to read from
  • 'error': any error messages faced

BOSS participant

See participant.py for the source code.

More information about BOSS [Infrastructure/BOSS].

Messages

The general message format is JSON encoded dictionary.

Kickstarter

To initialize kickstarter, send a JSON formatted message as follows:

{'config':config, 'email':'email', 'imagetype':'imagetype', 'id': id}

where:

• Config, is a yaml formatted configuration for kickstarter
• Email, email address
• Imagetype, image type of values: livecd, liveusb, loop, raw, nand, mrstnand, vdi or vmdk
• Id, a unique id for this job

Moblin Image Creator

To initialize the actual image building, send a JSON formatted message as follows:

{'email':email, 'id':id, 'imagetype':imagetype, 'ksfile':ksfile}

where:

• email, email address of the submitter
• id, identification string of the original message (passed on from kickstarter process)
• imagetype, image type of values: livecd, liveusb, loop, raw, nand, mrstnand, vdi or vmdk
• ksfile, the actual kickstar file to upload (in text)