Views

Release Infrastructure/IMG Installation for OpenSuse 11.4

From MeeGo wiki

(Difference between revisions)

Revision as of 12:00, 22 September 2011

1 Introduction
2 Installation procedure

Introduction

IMG (Image Me Give) is a small python client/server application suite which manages the supervision of the MIC2 image creation following the submission of a ks file by the user. Before proceeding with the reading of this document the reader is adviced to consult the IMG main page. If successfully completed, the installation procedure below will result in an operational IMG server turning on your local OpenSuse machine. The installation time can be estimated at ~45 minutes.

Remark: Alternatively, one may wish to embed IMG in a VM environment, i.e. to run IMG on OpenSuse within a VM. This can be done using Suse Studio as described in the document IMG installation using the Suse Studio .

Prerequisites

sudo, zypper are available

Resources

1) The relevant IMG resources:

IMG main page

The reader is advised to skim the links in the list above in order to assure the understanding of the principles IMG is based upon.

2) The Linux help resources:

Though the installation procedure is self-sufficient and complete, you are advised to consult these pages if you are not familiar with "zypper" or "lighttpd".

Projects using IMG

The "OBS Light for MeeGo" project will optionally use IMG. For more details see:

OBS Light for MeeGo

Acknowledgment

The first version of this page has been created after a successful OpenSuse IMG installation based on the document IMG Installation for Debian as friendly provided by Nokia. The publisher acknowledges this fact and thanks Nokia for its support.

Installation procedure

Preliminaries

The installation of IMG implies the installation of all IMG dependencies, i.e. of all packages associated with IMG. It may occur that different versions of these dependencies are hosted on different locations and that versions conflicts arise. In this case, the user must assure that he works with the right version of the package in question via:

>zypper -qi package
>zypper info package

With few exceptions, all the commands of the installation procedure below can be entered in the "sudo" mode. Before actually installing IMG, you must first install BOSS & BOSS-SkyNET. The installation of IMG comprises both the actual installation of packages and configuration. The installation of the ensemble of IMG packages is presented first, followed by the configuration part. The installation of IMG packages can be done:

1) either manually by adding all the necessary repositories that contain IMG packages as first proposed for Debian in IMG Installation for Debian or
2) using a script that takes all the packages from a single repository:

http://repo.pub.meego.com/home:/ronan:/OBS_Light:/MINT:/RC2/openSUSE_11.4/home:ronan:OBS_Light:MINT:RC2.repo

which regroups the IMG dependencies.

If the user prefers an installation of IMG packages via a script, he should skip the next section and pass directly to the section "Installation from an OBS Light repository using a script".

Remark: If available, the user is highly encouraged to install and to use YAST which is a time-saving complement to the command line mode presented in the following.

Manual installation

Installation of BOSS and BOSS-SkyNET

The BOSS and BOSS-SkyNET installation procedure as presented in the following is a shortened version of BOSS & BOSS-SkyNET installation adapted to the context of IMG. In particular, "boss-viewer" and "boss-obs-plugin" are irrelevant for IMG and will not be installed. The installation of BOSS and BOSS-SkyNET is currently done from the repository MINT:RC. Add this repository to the list of your repositories:

zypper ar http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/Project:MINT:RC.repo
zypper --no-gpg-checks --gpg-auto-import-keys ref

Install BOSS via:

zypper --non-interactive in boss

Install subsequently BOSS-SkyNET via:

zypper --non-interactive in boss-skynet

Remove now the MINT:RC repository before starting with the IMG installation:

zypper rr http://repo.pub.meego.com/Project:/MINT:/RC/openSUSE_11.4/

Adding the remote repositories

Please launch:

>zypper lr -d

in order to check which repositories have already been added. The repositories below are normally present by default on an OpenSuse machine, but you must add them if this is not the case:

zypper ar http://download.opensuse.org/distribution/11.4/repo/oss/ openSUSE-11.4-distr-repo-oss 
zypper ar http://download.opensuse.org/distribution/11.4/repo/non-oss/ openSUSE-11.4-distr-repo-non-oss
zypper ar http://download.opensuse.org/update/11.4/ openSUSE-11.4-update

Following repositories, if not present, must be now added in order to enable a seamless installation of all IMG dependencies:

zypper ar http://repo.pub.meego.com/Project:/MINT:/Testing/openSUSE_11.4/Project:MINT:Testing.repo   
zypper ar http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_11.4/devel:languages:python.repo
zypper ar http://repo.meego.com/MeeGo/tools/repos/opensuse/11.4/meego-tools.repo
zypper --no-gpg-checks --gpg-auto-import-keys ref

Installation of the IMG-WEB package

Install the packages "img-web" and "img-boss":

zypper --non-interactive in img-web img-boss

Installation of KVM

This section is optional. When working with IMG, the user has 2 possibilities:

create an image directly using the MIC2 installed on your machine
create an image using the MIC2 embeedded in KVM

as described in IMG main page. The two options above correspond to the settings:

use_kvm = no (default) respectively
use_kvm =yes

in the file:

vi /etc/skynet/build_image.conf

In the case that you want to embed MIC2 in KVM, install KVM on your local machine via:

zypper in kvm

Reboot your machine after the KVM installation in order to activate the KVM configuration on your PC.

Remark: Alternatively, KVM can be installed using YAST as described in YAST Installation of KVM for OpenSuse.

Installation of PATCH

The current version of IMG as friendly provided by Nokia in IMG source code (Nokia) can be enhanced by applying patches to some of the source files of IMG. Patching of IMG is fully compatible with OpenSource practices and allows the IMG users community to maximally profit from IMG at the current stage of releases. In order to be able to patch the IMG source and configuration files, we will install the patch package:

zypper --non-interactive in patch

Alternatively, we will also make usage of awk and sed in order to obtain the desired modifications.

Installation from an OBS Light repository using a script

This section presents an alternative way to the manual installation of IMG as described in the previous section above. In the context of the OBS Light project OBS Light for MeeGo a repository:

http://repo.pub.meego.com/home:/ronan:/OBS_Light:/MINT:/RC2/openSUSE_11.4/home:ronan:OBS_Light:MINT:RC2.repo

that contains all the IMG packages has been created. Execute the following script in order to install all the IMG dependencies:

#!/bin/bash
zypper ar http://repo.pub.meego.com/home:/ronan:/OBS_Light:/MINT:/RC2/openSUSE_11.4/home:ronan:OBS_Light:MINT:RC2.repo
zypper ar http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_11.4/devel:languages:python.repo
zypper ar http://repo.meego.com/MeeGo/tools/repos/opensuse/11.4/meego-tools.repo
zypper --no-gpg-checks --gpg-auto-import-keys ref
zypper --non-interactive in boss
zypper --non-interactive in boss-skynet 
zypper --non-interactive in img-web img-boss

Launch of the MySQL server and of BOSS

Following the successful installation of img-web launch the MySQL server:

/etc/init.d/mysql start

The default password for the MySQL user "root" is an empty string. If you want to change the MySQL password for the superuser "root" or for another user, please consult the document Change password for MySQL.

Launch now BOSS by executing:

rcboss start

If you launch the command "rcboss start" within an installation script, it should be followed by a "sleep" command in order to guarantee that the launch of rcboss is fully accomplished before the execution of the next script command:

rcboss start
sleep 10

Remark: You have the possibility to create the log output of boss via:

rcboss log

SkyNET

SkyNET Configuration

SkyNET facilitates the management of BOSS participants and permits the control of each BOSS participant (via: start/stop/reload/status). The configuration is done in:

vi /etc/skynet/skynet.conf

The main value that needs to be set is "amqp_host". Verify that your file "skynet.conf" contains the following lines:

[boss]
amqp_host = 127.0.0.1:5672
amqp_user = boss
amqp_pwd = boss
amqp_vhost = boss

SkyNET launches participants without any system environment. Some installations may need to set proxy information. This can be done in:

vi /etc/skynet/skynet.env

By default, SkyNET participants are installed to:

/var/lib/SkyNET/store

Following a correct BOSS-SkyNET installation you should have 4 participants:

ll /var/lib/SkyNET/store
drwxr-xr-x 4 root root 4096 21 juil. 12:03 build_image
drwxr-xr-x 4 root root 4096 21 juil. 12:03 build_ks
drwxr-xr-x 4 root root 4096 21 juil. 12:03 request_image
drwxr-xr-x 4 root root 4096 21 juil. 12:03 update_image_status

Their setup is presented in the next section.

SkyNET participants

The SkyNET participants are listed below:

1) build_image (worker)
2) build_ks
3) request_image
4) update_image_status

The worker "build_image"

Specifying the configuration in "build_image.conf"

The participant "Build_image" is the actual image building worker. Depending on its configuration it will:

either using sudo (see next section) launch MIC2 or
boot a kvm image and run MIC2 inside that.

Configuration of the image building worker is done in "/etc/skynet/build_image.conf"

The two important configuration options are base_url and base_dir:

base_url sets the URL at which base_dir is served using HTTP
base_dir points to the directory where the images will be saved

Perform the follwong update on the file "build_img.conf":

MYHOST=`/sbin/ifconfig  | grep 'inet adr:'| grep -v '127.0.0.1' | cut -d: -f2 | awk '{ print $1}'`
sed -i "s,^base_url = http://127.0.0.1/images*,base_url = http://$MYHOST/img/images," /etc/skynet/build_image.conf
sed -i "s,^base_dir = /var/www/images*,base_dir = /var/www/img/images," /etc/skynet/build_image.conf

The base_url contains:

either the IP address of your local OpenSuse machine
or that of your OpenSuse embedded in a VM as described in IMG installation using the Suse Studio .

Other users will be thus be able to access remotely your IMG server from their machines.

Using sudo in the non-kvm mode

In order to run the MIC2 tool, the "nobody" user needs sudo rights for mic-image-creator:

chmod 0640 /etc/sudoers
cat >> /etc/sudoers << EOF
nobody ALL=(ALL)NOPASSWD:/usr/bin/mic-image-creator
EOF
chmod 0440 /etc/sudoers

Working in the kvm mode

In the KVM mode, IMG first creates an overlay image from the base image (not distributed), and then starts it using the overlay image as the hard disk. "Virtio" is used as a speed optimization method for KVM. After the elapse of a specified time interval IMG will copy the ks file and the MIC2 configuration file to the guest VM. Subsequently, it runs MIC2 in the VM with parameters specified in the "init" method. Following a successful MIC2 run, the created images are copied from the guest VM via scp.

Start and Register

Once the worker has been configured, it should be started and registered. Enable (start) and register the participant via:

skynet enable build_image
skynet register -n build_image

The participant "build_ks"

This participant validates and manipulates the kickstart file. When extra repositories and packages are added to the ks file, this information is handled by "build_ks". Only one participant should be running. It should be launched on the same machine as "img-web". In order to configure the particpant, edit the file:

/etc/skynet/build_ks.conf

Set "reposerver" to point to the OBS http repository server (the user-facing download server, not the backend bs_reposrv system). Set "ksstore" to point to the location that holds the kickstart files referenced by the webui. Enable (start) and register the participant via:

skynet enable build_ks
skynet register -n build_ks

For more information about participants management see Working with SkyNET.

The participant "request_image"

This participant allows IMG to be used by a normal BOSS process.

Enable (start) and register the participant via:

skynet enable request_image
skynet register -n request_image

The participant "update_image_status"

This participant updates the queue data for the IMG web ui when a worker finishes a job. No configuration is needed. Enable (start) and register the participant via:

skynet enable update_image_status
skynet register -n update_image_status

Launch of SkyNET

Launch SkyNET by entering the command below:

/usr/share/boss-skynet/skynet_launch

Remark: The log files for a participant $NAME can be found in "/var/lib/SkyNET/store/$NAME/log/main/current".

Work with packages groups

If you want to work with packages groups (see: Obstag), apply the following patches:

Update of "views.py"

patch /usr/lib/python2.7/site-packages/img_web/app/views.py << EOF
28c28
< from img_web.app.forms import UploadFileForm, extraReposFormset
---
> from img_web.app.forms import UploadFileForm, extraReposFormset,extraPackagesGroupsFormset
40c40,41
<         formset = extraReposFormset()
---
>         formset = extraReposFormset( prefix='fs1')
>         formPackagesGroupsset = extraPackagesGroupsFormset( prefix='fs2')
42c43
<                                   {'form' : form, 'formset' : formset},
---
>                                   {'form' : form, 'formset' : formset, 'formPackagesGroupsset' :   formPackagesGroupsset},
48c49,50
<         formset = extraReposFormset(request.POST)
---
>         formset = extraReposFormset(request.POST, prefix='fs1')
>         formPackagesGroupsset = extraPackagesGroupsFormset(request.POST, request.FILES, prefix='fs2')
51c53
<                                       {'form': form, 'formset' : formset},
---
>                                       {'form': form, 'formset' : formset, 'formPackagesGroupsset' :  formPackagesGroupsset},
55a58,59
>         data3 = formPackagesGroupsset.cleaned_data
>         
92a97,106
>             for prj in data3:
>                 if 'GPFile' in prj:
>                     if prj['GPFile']:
>                         fileName=prj['GPFile'].name
>                         if not fileName.endswith(".grp-ks"):
>                             print fileName,"is not a valid file"
>                         else:
>                             grpFile=prj['GPFile'].read()
>                             imgjob.kickstart=imgjob.kickstart.replace("%include "+fileName,grpFile)
> 
EOF

Update of "forms.py"

patch /usr/lib/python2.7/site-packages/img_web/app/forms.py << EOF
57a58,67
> class extraPackagesGroupsForm(forms.Form):
>     GPFile = forms.FileField(label="Kickstart file", required=False,
>                              help_text="Extra Packages Groups: Choose a .grp-ks file.")
>     def clean(self):
>         cleaned_data = self.cleaned_data
>         if not cleaned_data['GPFile'] :
>             raise forms.ValidationError("You chose an extra OBS without adding a corresponding  repository.")
>         return cleaned_data
> extraPackagesGroupsFormset = formset_factory(extraPackagesGroupsForm)
> 
EOF

Update of "upload.html"

patch /usr/lib/python2.7/site-packages/img_web/templates/app/upload.html << EOF
11c11,19
<             extraClasses: ['row1', 'row2', 'row3']
---
>             formCssClass: 'dynamic-formset',
>             extraClasses: ['row1', 'row2', 'row3'],
>             prefix: "{{ formset.prefix }}"
> 
>         });
>         \$('#id_extra_packages_groups tbody tr').formset({
>             formCssClass: 'dynamic-formPackagesGroupsset',
>             extraClasses: ['row1', 'row2', 'row3'],
>             prefix: "{{ formPackagesGroupsset.prefix }}"
159c167
<              <tr id="{{ f.prefix }}-row">
---
>              <tr id="{{ f.prefix }}-row" class="form-container">
177a186,207
> <fieldset>
>     <legend>Extra packages groups</legend>
>     <table id="id_extra_packages_groups" border="0" cellpadding="0" cellspacing="5">
>          <thead>
>              <tr>
>                 <th scope="col">Add grp-ks file</th>
>              </tr>
>          </thead>
>          <tbody>
>              {% for f in formPackagesGroupsset.forms %}
>              <tr id="{{ f.prefix }}-row" class="form-container">
> 
>                  <td>{{ f.GPFile }}</td>
> 
>              </tr>
>              {% endfor %}
>          </tbody>
>      </table>
>     {{ formPackagesGroupsset.management_form }}
> 
> </fieldset>
> 
EOF

WEB UI

IMG uses a django web application to provide a UI. This WEB UI needs a MySQL DB to store information about the images.

MySQL Configuration

MySQL is installed by default and during the installation a root account may have been created. Create now a password for the user "img" and grant him the appropriate rights on the MySQL IMG DB:

mysql -u root --password="" -e "create database imgdb CHARACTER SET utf8; GRANT ALL ON imgdb.* TO 'img'@'localhost' IDENTIFIED BY 'img' "

Remark 1: By default, no password for the root user of MySQL will have been created, i.e. the password is an empty string. In such a case you should simply type "Enter" when asked for password after entering the command "mysql -u root -p". If you work on an OpenSUSE host with a default OBS installation or an OBS appliance, the mysql root password is "opensuse".

Django Configuration

The Django interface is configured via the file /etc/imager/img.conf. Take the default configuration values in the file "/etc/imager/img.conf" as provided or adapt them to your needs. Create now locations for the images and templates:

install -d -m 0777 /var/www/img/images
install -d -m 0777 /var/www/img/templates

Setup django:

export DJANGO_SETTINGS_MODULE=img_web.settings

Attention: The export command above must be done by root, sudo is not admitted here!

For a successful synchronization between the MySQL DB and the WEB UI of IMG launch the command:

django-admin.py syncdb --noinput
django-admin.py createsuperuser --noinput --username imager --email imager@imager.org
mysql -u root --password="" -e "use imgdb;UPDATE auth_user SET password='sha1\$a4cbe\$e586fd28fe54781b694eb6e2fe6ce20022843658' WHERE username='imager';"

The login username and a password for the WEB UI of IMG is: "imager"/"imager".

patch /usr/lib/python/site-packages/img_web/templates/registration/login.html << EOF
8a9
> <p>(by default Username:imager Password:imager)</p>
EOF

Attention: You will need these identifiers during the login in the WEB UI of IMG. Note them carefully. Accomplish the synchronization by doing:

django-admin.py migrate
django-admin.py collectstatic --noinput

Configuration of the web server

Creation of the file /lighttpd/vhosts.d/img.conf

Create the file

vi /etc/lighttpd/vhosts.d/img.conf

with the following content:

cat > /etc/lighttpd/vhosts.d/img.conf << EOF
var.namebasedir = "/img"
  
\$HTTP["url"] =~ "^" + namebasedir {
       dir-listing.activate = "enable"
}

url.redirect += (
 "^" + namebasedir + "$" => namebasedir + "/"
)

fastcgi.server += (
   "/img.fcgi" => (
       "main" => (
           "socket" => "/var/run/img_web" + ".socket",
           "check-local" => "disable",
           "allow-x-send-file" => "enable",
       )
   ),
)

url.rewrite-if-not-file += (
   "^(" + namebasedir + "/.*)$" => "/img.fcgi/\$1"
)
EOF

Lighttpd configuration files

Now we will modify the following 2 lighttpd configuration files:

/etc/lighttpd/lighttpd.conf
/etc/lighttpd/modules.conf

Modifications in /etc/lighttpd/lighttpd.conf

Edit the file:

vi /etc/lighttpd/lighttpd.conf

and add the line:

include "/etc/lighttpd/vhosts.d/img.conf"

at the end:

echo 'include "/etc/lighttpd/vhosts.d/img.conf"' >> /etc/lighttpd/lighttpd.conf

For more information on the "lighttpd.conf" file the user is encouraged to consult the Doc on lighttpd.

Modifications in /etc/lighttpd/modules.conf

Edit the file by:

vi /etc/lighttpd/modules.conf

Assure the presence of the following block in the file "modules.conf" by uncommenting the needed server-modules and by putting the "server.modules" list inside the global block:

global{
server.modules = (
"mod_access",
"mod_alias",
"mod_accesslog",
"mod_compress",
"mod_fastcgi",
"mod_rewrite",
)
}

To achieve the necessary modifications you can directly apply the following patch:

patch /etc/lighttpd/modules.conf << EOF
41c41
< 
---
> global{
44,50c44,48
< #  "mod_alias",
< #  "mod_auth",
< #  "mod_evasive",
< #  "mod_redirect",
< #  "mod_rewrite",
< #  "mod_setenv",
< #  "mod_usertrack",
---
>   "mod_alias",
>   "mod_accesslog",
>   "mod_compress",
>   "mod_fastcgi",
>   "mod_rewrite",
51a50
> }
EOF

Working with IMG

Start IMG first time

First you have to start the lighttpd fastcgi server:

service lighttpd start

Start IMG with the following command:

service img-web start

In the case that the server wasn't launched successfully, you need to restart it:

service lighttpd restart; service img-web restart

Starting IMG after the reboot of your machine

In order to automatically activate at each reboot all the services necessary for IMG, execute the following commands:

chkconfig --add mysql
chkconfig --add lighttpd 
sed -i "/^exit 0/ i\service img-web start" /etc/init.d/rc

The IMG server will be automatically launched now for the user "root" after each reboot!

Tests

The locations of the user WI, of the admin WI and of the images WI are listed below:

 http://$IP-address/img/
 http://$IP-address/img/admin/

In order to check that the IMG server has been successfully launched, verify that you can access these pages. If your installation has been successful, you should see on:

 http://$IP-address/img/images/

the correct appearance of IMG., see below (left image):

Correct appearance of IMG	Default (wrong) appearance of IMG

Install another stable release version of "img-web" if this is not the case, i.e. if you obtain a wrong appearance of IMG, e.g. the default appearance of IMG, see above (right image).

Finally, choose the "Create an image" action and try to create an image with a valid ks file.

Remark: The users from other machines should be able to access your operational IMG server remotely via the same urls as presented above.

IMG Guide

The Wiki page:

IMG guide for beginners

explains in detail how to work with IMG by means of WI snapshots for each user's action. It allows a beginner to rapidly acquire the hands-on knowledge of MIC2 image creation jobs' management via IMG. The reader is asked to consult this page in order to get a practical introduction to IMG.

Overview of IMG compilation possibilities for different architectures and image types

IMG has been tested for different architectures and image types. In order to be able to create an image for ARM, you need first to install the packages "qemu" and "qemu-arm-static" packages:

zypper --non-interactive in qemu
zypper --non-interactive in qemu-arm-static

The table below presents the complete results of IMG compilation tests:

IMG test results
i586	ok	ok	ok	ok	ok	ok	ok	ok (1)	ok (2)
	RootFS	Live CD	Live USB	Loop file	Raw disk image	NAND	MRST NAND	VDI file	VMDK file
i686	ok	ok	ok	ok	ok	ok	ok	ok (1)	ok (2)
armv7l	ok	not possible	not possible	ok	ok	not possible	not possible	not relevant	not relevant
armv7hl	ok	not possible	not possible	ok	ok	not possible	not possible	not relevant	not relevant
armv7nhl	not tested	not possible	not possible	not tested	not tested	not possible	not possible	not relevant	not relevant

In order to be able to create images of type VDI and VMDK, the following additional operations must be done beforehand:

1. For the creation of VDI images you need to install the package "virtualbox". Execute the following commands:

zypper ar http://download.opensuse.org/repositories/Virtualization:/VirtualBox_backports/openSUSE_11.4_Update_standard/Virtualization:VirtualBox_backports.repo
zypper --no-gpg-checks --gpg-auto-import-keys ref											
zypper --non-interactive in virtualbox

2. For the creation of VMDK images you need to install the package "virt-utils" in order to enable the command 'qemu-img'. Enter:

zypper --non-interactive in virt-utils

Remark: The image types "Live CD" and "Live USB" for the ARM architecture don't make sense and therefore don't have to be considered. The images of type "NAND" and "MRST NAND" for ARM are presently created from an image of type "RAW" as described in MeeGo on QEMU.

Diagnostics

In order to see the list of participants enter:

skynet list

In order to view a participant log, enter one of the following:

skynet log build_image
skynet log build_ks
skynet log request_image
skynet log update_image_status

The command "skynet log build_image" will provide you with the path to the directory containing the input ks file and a build image log file. In the case of a successful image creation this directory will also contain an iso file. The user is encouraged to consult the document Working with SkyNET for more details.

Troubleshooting

The IMG server cannot be accessed from a remote machine

Description:

 Following the entering of http://$IP-address-of-your-machine/img/admin in the web browser of his remote machine, the user cannot access your operational IMG server.

Resolution:

 Desactivate the firewall on your OpenSuse machine.

You change the hostname of the machine