Contents |
Agile evangelists will tell us that deployment is the first thing we should work on.
So it is perhaps fitting that this is the probably the last thing I'll do on OTS - in the current arrangement anyway. Given where we are (fixing it on at the end) it is not something that I can resolve at this stage, but process is a journey anyway... blah blah
I have seen consistent errors on the introduction of the Django Plugins when trying to run the unittests straight for a git checkout. The workaround is to go back and uninstall the Developer eggs in the Django plugins
The development workflow *should* be simple
1. Download OTS 2. Run tests OTS 3. Start work
OTS is a large project now <#FIXME how many?> LOC. A significant proportion of this is backend.
In fact the front end (Django) work are supposed to be Plugins so they really shouldn't be causing any issues with the backend.
We have <#FIXME how many> unittests. Wouldn't it be great if we harness these on deployment the system?
For getting new developers up an running and installing OTS the first response to any problem should be the same:-
"Do the tests run?"
With all the components of OTS we need some kind of metapackaging to bring it together?
But we have meta packaging with RPMs / Debian don't we?
RPM and Debian are deployment domain tools and not really suited to development.
Installing development software at system level is impolite and limiting.
It would be good if we had a consistent and reliable unittest interface across all components.
We started from a privileged position. I have never worked on a project which has only targeted one OS. Let alone a subset of that OS which has one package management system. Debian was just about OK under Maemo. Meego means we have RPMs too.
We are lucky enough to have good people happy to maintain two package management systems.
If OTS is success grows then the requirement to install on other OS is pretty much an inevitability.
Currently a smaller user base making custom installs on a regular basis. I am aware of at least 6 variants.
| Installation | Webserver | File System |
|---|---|---|
| My Commercial pay per month service (dev) | Nginex | virtualenv |
| My commercial Deployment ppm (deploy) | Lighttpd | virtualenv |
| Corporate setup #1 | Apache mod_python | debian |
| Corporate setup #2 | Apache wsgi | debian |
| Corporate setup #3 | Apache mod_python | rpm |
| Meego | Unicorn | virtual env |
That's fine and quite normal but.. it might be that everyone has their own hack to get it working in their set up... how are these being tracked? Hopefully I won't need to convince test engineers the value of reproducibility!
Some notes on repeatability of buildout cf virtualenv here
I wrote the setup.sh and nose.sh early in the OTS development for personal convenience. They were shared and they have started to evolve. These scripts solve basic and common problems and we have to maintain them rather than investing in these scrappy shell scripts I'd suggest it would be better to invest the time understanding more elegant and powerful solutions.
1. Loose Coupling between Coarse Grained Components 2. Consistent and Easy to set up Development Environment 3. Consistent and Easy to set up Deployment Environments 4. Verification of Deployment set up with unittests 5. A reproducible and improvable process
Lofty aims...
Helped us with our decoupling with Publishers in the journey from 0.1 -> 0.8
There is plenty of propaganda documentation that makes claims that Buildout can solve many of the issues.
Buildout is designed to work with setuptools so far so good. However it is from the Zope / Plone world, concerns about interaction with Django where dispelled when I saw this post in which the Django creator says that
it’s an exceedingly civilized way to develop an app.
As with many of the commentators I have found the documentation patchy and there have been things that have taken a while to workout. Hopefully this wiki will help.
However the overall experience (as you would expect from a third generation tool) is something that lives up to its promises.
So to work...
I have taken the approach of modelling our system on a basic shell with the same layout as OTS. My experience of deployment is the same as others in that you need to start to start with something then refine it over a number of installs before it works cleanly on a number of systems. That isn't achievable for me but hopefully I've done the groundwork that allows this knowledge to be applied. I think it is achievable over the next few releases with OTS.
You can pull the code from here
python setup.py test
I have added this to the top level django plugin to give consistency with the rest of the code. This wiki makes the point that the Perl guys have the ability to run tests on installation - and we can't have them having one-up on us can we ;-). Also makes us PyPi compatible if the need arises.
The Django plugins have hardcoded each app. Django hasn't got a reputation for being the best framework for doing these things (interacting with other large and complex systems) more aimed at the digital media thing -but that is what we are using so what to do?
The settings are in Python of course so we can apply the extension point ideas in theory:
TARGET_NAMESPACE = "foo.bar"
EXTENDED_APPS = [entry_point.name for entry_point in
pkg_resources.iter_entry_points(TARGET_NAMESPACE)]
INSTALLED_APPS = (
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
# Uncomment the next line to enable the admin:
# 'django.contrib.admin',
# Uncomment the next line to enable admin documentation:
# 'django.contrib.admindocs',
INSTALLED_APPS = tuple(list(INSTALLED_APPS) + EXTENDED_APPS)
It also works in practice with the model, but given the fundamental problem (using Django in a way that wasn't intended) you might prefer to put up with the hardcoding and the tight coupling.
Buildout claims to tick the boxes on a lot of the problems above reproducibility, testing and deployment are it's raison d'etre.
Let's run through the workflow set out above for the buildout I've wrapped around the example model code.
First
$python bootstrap
Then
$bin/buildout
If you check the bin directory now you'll see the parts all there.
$/bin/test Creating test database 'default'... Creating table auth_permission <snip> Simple Test .. ---------------------------------------------------------------------- Ran 2 tests in 0.004s
OK Destroying test database 'default'...
So far so good...
Now you can have a sandboxed environment within which you can work straight away
Coding...
$bin/python >>> foo.bar.__file__ '/path/to/your/buildout/foo.bar.baz/foo/bar/__init__.py'
And checking the results...
$bin/django runserver Validating models... 0 errors found Django version 1.2, using settings 'bar.settings' Development server is running at http://127.0.0.1:8000/ Quit the server with CONTROL-C.
Pretty neat hey?
Let's recap with a few lines of typing we have pulled our development environment, run all the tests and can set to work against the system and it's exactly the same system that all the other developers are using
It gets better...
This is took me a while to get my head around and judging from IRC histories in the Zope channel I'm not alone.
It should be pointed out at this point that Buildout is very much designed with PyPI in mind so it is expecting packages to be built to and come from there, naturally the open source aims of OTS are in line with that but we aren't there yet ;-). I've used file paths.
This is how I got it to work.
You might've noticed that there was a source-release in the bin directory. We use this to build the release from *another* buildout.cfg file.
$/bin/buildout-source-release file:///path/to/your/buildout/deploy deploy.cfg
You'll see that this has created a deploy.tgz
What can we do with that?
Unzipping it would be a start
$tar xvzf deploy.tgz $cd deploy
You'll see that there is an install file there. Run it
$python install.py
So can we test our deployment? Naturally
$bin/test
And Django works too!
Buildout allows you to extend the config files and for various configurations. Naturally you'll want to work with your preferred webserver.
OTS has traditionally used Apache and mod python so that is what is in this example.
This professional wiki held my hand through the process. It also illustrates how other web servers can be used to.
I followed this process and the code is incorporated in the example.
Reviewing the Goals:
Achievable but will need review due to the legacy architecture
Tick
Tick
Tick
Tick
The skeleton is very much modelled on OTS so it should be easy to apply this to our setup.
The buildout.cfg is where you should start and there really isn't much more to it than that.
Naturally you'll want to extend it to include rabbitmq and all the other bits and bobs.
The component tests and even the system tests could be incorporated to - just imagine a one click roll out of OTS that runs a self test - now *that* would be nice.