Architecture/planning/evolution-data-server/eds-improvements

Revision as of 09:49, 5 April 2011

This page describes improvements that are desirable or even essential for good performance of Evolution Data Server (EDS) as PIM storage in MeeGo.

All of these changes should be developed for the EDS master branch, reviewed by upstream maintainers and then back-ported to EDS 2.32, the version in MeeGo.

content protection

Only authorized applications (= processes) with the necessary permissions should be able to read/write data. Exact permissions and mechanisms for verifying them to be defined as part of the (revised?) MeeGo architecture.

This can be added as follows:

1. lock down file permissions
2. add permission checks to EDS D-Bus servers

change tracking

EDS is based on the model/view/controller principle: an app opens a database (calendar or addresbook) and requests a view with server-side filtering, then is sent:

1. all data covered by the view
2. "all data sent" signal
3. changes (add/update/remove) while the view exists

Applications modify data with asynchronous requests (controller) and keep showing the old data until the change notification comes in. The exact changes made to the data are determined by the server (for example when meeting invitations are sent).

QtContacts and KCalCore do not have such a strict model. With QtContacts the app simply loads and stores contacts and is responsible for maintaining its own view. Changes are sent for changes made after the database was opened. KCalCore is based on the concept of loading a calendar (partially) into the client, modifying it there, then writing it back. Changes are also sent while the storage is open.

TODO:

1. extend EDS query language with a flag that suppresses sending existing data when the view is opened
2. implement for local contact and calendar backend, reject with error in others
3. contacts: implement special case where view with "requested_fields = UID" only returns UID

One possibility for adding the flag is to (ab)use the existing query language with a match for special fields:

• in the client: query = e_book_query_orv(e_book_query_field_exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP"), <real query>, NULL);
• in the server: "if (query is OR-Query and first clause is exists("X-EDS-QUERY-FLAG-SKIP-DATA-DUMP")) { skip_data_dump = TRUE; query = second clause; }

optional parsing of data in client

libebook/ecal provide different functionality:

1. exchange data as vCard/iCalendar with EDS daemon
2. parse items and wrap them in a C API

QtContacts and KCalCore replace the second part. An obvious performance improvement is to only do the second part if the additional C APIs are really called. The following calls then transfer data without any additional parsing in libebook/libecal:

• e_book_get_contact()/asynchronous read/view + e_vcard_to_string()
• e_contact_new_from_vcard() + e_book_add_contact()/e_book_commit_contact()/asynchronous storing
• e_cal_get_object_list_as_comp()/asynchronous read/view + e_cal_get_component_as_string()
• icalcomponent_new_from_string() + e_cal_modify_object[_with_mod]()/e_cal_create_object()
• e_cal_get_object_list_as_comp()/e_cal_get_object()/asynchronous view + e_cal_get_component_as_string()

TODO:

1. add delayed parsing to libebook, triggered by API call which needs parsed data
2. add delayed parsing to libical, again triggered by API call

PROBLEM:

1. If delayed parsing fails, how can error be reported?
2. Should delayed parsing be done always (e_vcard_to_string()/icalcomponent_new_from_string()) or only if requested? How would that be configured by a libebook/libecal user?

more efficient access to meta data for change tracking + atomic updates

This was discussed a while ago: [[http://www.mail-archive.com/evolution-hackers@gnome.org/msg03029.html [Evolution-hackers] concurrent modifications of items in GUI and EDS database]]

Data synchronization only needs to know which items exist (list of local IDs) and which revision of each item is currently stored. Revisions can be identified with a string which changes for each modification. Restoring an older revision should (but doesn't have to) restore the revision string.

For calendar, UID+RECURRENCE-ID and LAST-MODIFIED can be used. For contacts, UID + REV.

By comparing a stored list against the current one, changes can be detected without relying on journals in the daemon or comparison of time stamps. Journals are hard to maintain (how many of them, how long?). Comparison of time stamps has problems when changes are allowed while a sync runs and assumes that system time is linear (which is not always true).

As an additional use case, local ID + revision string could be used to prevent overwriting stale data in the server, as in HTTP PUT eTag checks.

TODO:

1. extend EDS query language with a flag that limits delivered data to just UID+RECURRENCE-ID+LAST-MODIFIED resp. UID+REV
2. implement this query for local contact and calendar storage, reject with error in others
3. later: implement atomic changes as discussed in the mail thread

quick check for "data changed"

Even quicker than listing items and comparing their revisions strings would be a comparison of a revision string for the whole database. The semantic would be this:

• If the revision string is empty or different, then the content of the database might have changed. A more detailed check is necessary to determine that.
• If it is non-empty and the same as before, nothing has changed. No further check necessary.

The modification time stamp of the .ics or .db files could be used. Probably need a new "query property" API. e_cal_get_static_capability() only returns bool. Same with e_book_get_static_capabilities()/e_book_check_static_capability().

contacts: store PHOTO data as plain files

Transfering PHOTO data over D-Bus as base64 encoded data in each vCard has obvious performance issues. Instead the PHOTO property should be a link to a local file (supported by vCard), with utility code available to libebook users to convert between the different representations.

Goals:

1. Apps should be able to create photo files without ever passing the data to libebook.
2. External PHOTO file and corresponding contact must be kept in sync:

• • If a contact is deleted, the photo also needs to be removed.
  • If a photo file is created and then storing the contact fails, who removes the file?

1. Access to photo data must be limited to processes which have the right permissions.

Proposal:

• For each ~/.local/share/evolution/foo calendar, store photos in <photo path>=~/.local/share/evolution/foo/photos, with same permissions.
• New photo files are created with a UUID as base name and suffix according to the file content.
• If a contact is deleted and has a PHOTO;VALUE=uri:file://<photo path>/*, delete that file.
• For apps which want to create the photo file directly:
  • Apps get a file descriptor and the path from a new libebook API.
  • They own the file until the EDS server acknowledges the successful storing of a contact with a matching PHOTO;VALUE=uri:file property. If that fails, the app must remove the file.
• Add new API calls to libebook which split out resp. include data inline.
  • Must track ownership of file after splitting it out and before storing - might never get stored!

Open:

• Should there be automatic garbage collection of stale photos in case something went wrong and a photo wasn't properly deleted?
• Trust TYPE=JPEG/PNG/... or look into file content to determine type?